This repository provides an automated and comprehensive evaluation framework for benchmarking Post-Quantum Cryptography (PQC) algorithms. It is designed for researchers and developers looking to evaluate the feasibility of integrating PQC into their environments. The framework streamlines the setup and testing of PQC implementations, enabling the collection of computational and networking performance metrics across x86 and ARM systems through a suite of dedicated automation scripts.
PQC implementations are sourced from multiple libraries, including algorithms natively supported in OpenSSL 3.5.0 and those available from the Open Quantum Safe (OQS) project's Liboqs and OQS-Provider libraries. The framework also provides automated mechanisms for testing PQC TLS handshake performance across physical or virtual networks, providing valuable insight into real-world environment testing. Results are outputted as raw CSV files that are automatically processed using the provided Python parsing scripts to provide detailed metrics and averages ready for analysis.
Future versions of the project aim to support additional PQC libraries, further expanding the scope of supported benchmarking.
The project provides automation for:
-
Verifying and installing required system packages and Python PIP dependencies.
-
Compiling and configuring the OpenSSL, OQS, and ARM PMU dependency libraries.
-
Collecting PQC computational performance data, including CPU and memory usage metrics, using the Liboqs library.
-
Gathering networking performance data for PQC schemes integrated into the TLS 1.3 protocol using the PQC support available natively in OpenSSL 3.5.0 and via the OQS-Provider.
-
Coordinated PQC TLS handshake tests run over the loopback interface or across physical networks between a server and client device.
-
Automatic or manual parsing of raw performance data, including calculating averages across multiple test runs.
For details on the project's development and upcoming features, see the project's GitHub Projects page:
PQC-Evaluation-Tools Project Page
- Supported Hardware and Software
- Supported Cryptographic Algorithms
- Installation Instructions
- Automated Testing Tools
- Parsing Test Results
- Additional Documentation
- Licence
- Acknowledgements
The automated testing tool is currently only supported in the following environments:
- x86 Linux Machines using a Debian-based operating system
- ARM Linux devices using a 64-bit Debian-based Operating System
This version of the repository has been fully tested with the following library versions:
-
Liboqs Version: Post-0.13.0 commit †
-
OQS-Provider Version 0.9.0
-
OpenSSL Version 3.5.0
By default, this repository is configured to use the last tested versions of the OQS libraries. This helps ensure that all automation scripts operate reliably with known working versions. The listed OpenSSL version remains fixed at 3.5.0 to maintain compatibility with the OQS-Provider and the project's performance testing tools.
While this setup maximises reliability, users who need access to more recent updates may configure the setup process accordingly. However, please note that the OQS libraries are still in active development, and upstream changes may occasionally break compatibility with this project’s automation scripts. This is detailed further in the Installation Instructions section.
If any such issues arise, please report them to this repository’s GitHub Issues page so they can be addressed promptly. Instructions for modifying the library versions used by the benchmarking suite are provided in the Installation Instructions section.
† For information on the specific commits used for the last tested versions of the dependency libraries, see the Project Dependencies documentation.
For further information on the classical and PQC algorithms this project provides support for, including information on any exclusions, please refer to the following documentation:
Notice: The HQC KEM algorithms are disabled by default in recent versions of both Liboqs and the OQS-Provider, due to their current implementations not conforming to the latest specification, which includes important security fixes. For benchmarking purposes, the setup process includes an optional flag to enable HQC in these libraries, accompanied by a user confirmation prompt and warning. Enabling HQC is done at the user's own discretion, and this project assumes no responsibility for its use. For instructions on enabling HQC, see the Advanced Setup Configuration Guide, and refer to the Disclaimer Document for more information on this issue.
The standard setup process uses the last tested commits of the project's dependency libraries to ensure compatibility with this project's automation tools. The setup script performs system detection, installs all required components, and supports multiple installation modes depending on the desired testing configuration.
While the default configuration prioritises stability, users may optionally configure the setup to pull newer versions of the OQS libraries. This can be useful for testing upstream changes or recent algorithm updates. For more advanced configuration options, see the Optional Setup Flags section.
The following instructions describe the standard setup process, which is the default and recommended option.
Clone the current stable version:
git clone https://github.com/crt26/pqc-evaluation-tools.git
Move into the cloned repository directory and execute the setup script:
cd pqc-evaluation-tools
./setup.sh
You may need to change the permissions of the setup script; if that is the case, this can be done using the following commands:
chmod +x ./setup.sh
When executing the setup script, you will be prompted to select one of the following installation options:
-
Computational Performance Testing Only - Installs components for standalone performance benchmarking (CPU and memory).
-
Full Install - Installs all components for both computational and TLS performance testing.
-
TLS Testing Libraries Only - Installs only the TLS benchmarking components. (Requires Option 1 has already been completed).
The setup script will also build OpenSSL 3.5.0 inside the repository’s lib directory. This version is required to support the OQS libraries and is built separately from the system’s default OpenSSL installation. It will not interfere with system-level binaries.
If the TLS testing libraries are installed (Options 2 or 3), you will be prompted with the following additional setup options:
-
Enable all disabled signature algorithms – Includes all digital signature algorithms in the OQS-Provider library that are disabled by default. This ensures the full range of supported algorithms can be tested in the TLS performance testing †.
-
Enable KEM encoders – Adds support for OpenSSL’s optional KEM encoder functionality. The benchmarking suite does not currently use this feature, but it is available for developers who wish to experiment with it.
Once all the relevant options have been selected, the setup script will download, configure and build each library. It will also tailor the builds for your system architecture by applying appropriate build flags.
† Enabling all signature algorithms may cause the OpenSSL speed tool to fail due to internal limits in its source code. The setup script attempts to patch this automatically, but you can configure this process manually. Please refer to the Advanced Setup Configuration for further details.
A hidden file named .pqc_eval_dir_marker.tmp is created in the project's root directory during setup. Automation scripts use this marker to reliably identify the root path, which is essential for their correct operation.
When running the setup script, it is vital that this is done from the root of the repository so that this file is placed correctly.
Do not delete or rename this file while the project is in a configured state. It will be automatically removed when uninstalling all libraries using the cleaner.sh utility script. If the file is removed manually, it can be regenerated by rerunning the setup script or creating it manually.
To verify the file exists, use:
ls -la
To manually recreate the file, run the following command from the project's root directory:
touch .pqc_eval_dir_marker.tmp
For advanced setup options, including:
- Pulling the latest version of the OQS libraries rather than the default tested versions
- Custom OpenSSL
speed.climits - Enabling HQC algorithms in the OQS Libraries
Please refer to the Advanced Setup Configuration Guide.
The repository provides two categories of automated PQC benchmarking:
-
Computational Performance Testing – Benchmarks the standalone performance of PQC cryptographic operations, gathering data on CPU and memory usage.
-
TLS Performance Testing – Benchmarks PQC, Hybrid-PQC, and classic algorithms integrated into the TLS 1.3 protocol, including handshake and cryptographic operation performance.
The testing tools are located in the scripts/test_scripts directory and are fully automated. The tools support assigning custom machine-IDs to the gathered results to make it easy to compare performance on differing systems.
This tool benchmarks CPU and memory usage for various PQC algorithms supported by the Liboqs library. It produces detailed performance metrics for each tested algorithm.
For detailed usage instructions, please refer to:
Automated Computational Performance Testing Instructions
Notice: Memory profiling for Falcon algorithm variants is currently non-functional on ARM systems due to issues with the scheme and the Valgrind Massif tool. Please see the bug report for details. Testing and parsing remain fully functional for all other algorithms.
This tool benchmarks the performance of PQC, Hybrid-PQC, and classical algorithms when used in the TLS 1.3 protocol. It utilises the PQC implementations natively available in OpenSSL 3.5.0 and those added via the OQS-Provider.
It conducts two types of testing:
-
TLS handshake performance testing – Measures the performance of PQC and Hybrid-PQC algorithms during TLS 1.3 handshakes.
-
Cryptographic operation benchmarking – Measures the CPU performance of individual PQC/Hybrid-PQC digital signature and Key Encapsulation Mechanism (KEM) cryptographic operations when integrated within OpenSSL.
Testing can be performed on a single machine or across two machines connected via a physical/virtual network. While the multi-machine setup involves additional configuration, it is fully supported by the automation tools.
For detailed usage instructions, please refer to:
Automated TLS Performance Testing Instructions
After the testing has been completed, unparsed results and automatically parsed results will be stored in the generated test_data/ directory:
Unparsed Results: test_data/up_results/
Parsed Results: test_data/results/
By default, test results are automatically parsed at the end of each testing script. This generates structured CSV output files based on the type of performance testing conducted.
Parsed results will be stored in the following directories, depending on which testing was performed:
test_data/results/computational_performance/machine_xtest_data/results/tls_performance/machine_x
Where machine_x is the Machine-ID number assigned to the results when executing the testing scripts. If no custom Machine-ID is assigned, the default ID of 1 will be set for the results.
If needed, automatic parsing can be disabled when calling the testing scripts by passing a flag to the testing script. This then facilitates the manual calling of the Python parsing scripts.
For complete details on parsing functionality and a breakdown of the collected performance metrics, refer to the following documentation:
Links below provide access to the various internal project documentation. However, the majority of these documents can be found in the docs directory at the project's root.
| Category | Documentation |
|---|---|
| Testing Tools Usage | - Automated Computational Performance Testing - Automated TLS Performance Testing |
| Setup & Configuration | - Advanced Setup Configuration - Supported Algorithms |
| Project Dependencies & Developer Information | - Project Dependencies - Project Scripts - Repository Structure |
| Performance Results | - Parsing Performance Results Usage Guide - Performance Metrics Guide |
| Other Resources | - Project Disclaimer |
The information provided in the internal documentation is also available through the project's GitHub Wiki:
- Liboqs Webpage
- Liboqs GitHub Page
- OQS-Provider Webpage
- OQS-Provider GitHub Page
- Latest Liboqs Release Notes
- Latest OQS-Provider Release Notes
- OpenSSL(3.5.0) Documentation
- TLS 1.3 RFC 8446
This project is licensed under the MIT License – see the LICENSE file for details.
This project depends on the following third-party software and libraries:
-
Liboqs – Used to provide standalone implementations of post-quantum key encapsulation mechanisms (KEMs) and digital signature algorithms for computational performance testing. This project includes modified versions of the
test_kem_mem.candtest_sig_mem.cfiles in order to collect detailed memory usage metrics during benchmarking with minimal terminal output. These modifications remain under the original MIT License, which is noted at the top of each modified file. -
OQS-Provider – Used to integrate post-quantum algorithms from
Liboqsinto OpenSSL via the provider interface, enabling TLS-based performance testing. Modifications include dynamically altering thegenerate.ymltemplate to optionally enable all signature algorithms that are disabled by default. The provider is built locally and dynamically linked into OpenSSL. It is licensed under the MIT License. -
OpenSSL – Used as the core cryptographic library for TLS testing and benchmarking. This project applies runtime modifications during the build process to increase the hardcoded algorithm limits in
speed.c(MAX_KEM_NUMandMAX_SIG_NUM) to support benchmarking of a broader algorithm set, and to append configuration directives toopenssl.cnfto register and activate theoqsprovider. OpenSSL is licensed under the Apache License 2.0. -
pqax – Used to enable access to the ARM Performance Monitor Unit (PMU) on ARM-based systems such as Raspberry Pi. This allows precise benchmarking of CPU cycles. No modifications are made to the original source code. Pqax is licensed under the Creative Commons Zero v1.0 Universal (CC0) license, placing it in the public domain.