PQC-Evaluation-Tools v0.3.1

Version v0.3.1 Release

Welcome to the version 0.3.1 release of the PQC-Evaluation-Tools project!

Release Overview

Version 0.3.1 is a maintenance release focused on resolving critical bugs introduced by recent changes in upstream dependencies. It addresses issues affecting algorithm detection, TLS handshake stability, and improves exception handling for computational performance testing on ARM-based systems. Additionally, a licence compliance review was completed to ensure all third-party usage remains properly attributed and documented.

Project Features

The project provides automation for:

• Compiling and configuration of the OQS, ARM PMU, and OpenSSL dependency libraries.

• Gathering PQC computational performance data, including CPU and memory usage metrics using the Liboqs library.

• Gathering Networking performance data for the integration of PQC schemes in the TLS 1.3 protocol by utilising the OpenSSL 3.4.1 and OQS-Provider libraries.

• Coordinated testing of PQC TLS handshakes using either the loopback interface or a physical network connection between a server and client device.

• Parsing of the PQC performance data, where data from multiple machines can be parsed, averaged, and then compared against each other.

Change Log

• Fix automated algorithm detection and TLS handshake failures due to UOV signature size limits in #44
• Fix ARM PMU access loss after reboot on some Raspberry Pi devices in #45
• Fix HQC not being enabled by default in Liboqs builds in #47
• Complete licence compliance review for third-party dependencies in #51

Full Changelog: v0.3.0...v0.3.1

Important Notes

• Functionality is limited to Debian-based operating systems.

• If issues still occur with the automated TLS performance test control signalling, information on increasing the signal sleep delay can be seen in the Advanced Testing Customisation section of the TLS Performance Testing Instructions documentation file.

Future Development

For details on the project's development and upcoming features, see the project's GitHub Projects page:

PQC-Evaluation-Tools Project Page

We look forward to your feedback and contributions to this project!

PQC-Evaluation-Tools v0.3.0

Version v0.3.0 Release

Welcome to the version 0.3.0 release of the PQC-Evaluation-Tools project!

Release Overview

The version 0.3.0 release of the PQC-Evaluation-Tools suite delivers key updates focused on compatibility, stability, and usability. It adds support for the latest version of the OQS libraries and has been upgraded to use the latest version of OpenSSL (version 3.4.1).

Automation has been refined with fixes to control signalling, improved handling of TLS benchmarking ports, and more robust detection of port conflicts. The suite now supports older Python versions, and script usability has been enhanced through clearer naming and new help flags.

TLS benchmarking is now more stable, resolving issues caused by control signal timing and newer versions of the OQS-Provider library to break the OpenSSL s_speed tool when using all its supported algorithms. The release also includes an updated generate.yml for OQS-Provider, along with documentation improvements and general codebase clean-up.

These changes make the toolset more flexible and reliable for evaluating PQC performance, particularly in TLS-based deployments.

Project Features

The project provides automation for:

• Compiling and configuration of the OQS, ARM PMU, and OpenSSL dependency libraries.

• Gathering PQC computational performance data, including CPU and memory usage metrics using the Liboqs library.

• Gathering Networking performance data for the integration of PQC schemes in the TLS 1.3 protocol by utilising the OpenSSL 3.4.1 and OQS-Provider libraries.

• Coordinated testing of PQC TLS handshakes using either the loopback interface or a physical network connection between a server and client device.

• Parsing of the PQC performance data, where data from multiple machines can be parsed, averaged, and then compared against each other.

Change Log

• Verify project support for liboqs version 0.12.0 in #9
• Update OQS-Provider Script and Variable Naming in #10
• Upgrade OpenSSL Dependency to Version 3.4.0 in #11
• Fix Control Signalling Issue in Automated TLS Benchmarking Scripts in #13
• Add Support for Older Versions of Python in #17
• Upgrade OpenSSL Dependency to Security Patch Version 3.4.1 in #19
• Make TLS Benchmarking Ports Configurable and Improve Conflict Handling in #20
• Update Modified OQS-Provider generate.yml File in #22
• Fix False Positives in Port Usage Checks & Relocate Default TLS Benchmarking Ports in #24
• Fix TLS Speed Benchmark Failure Caused by Excessive Algorithm Loading in #26
• Add Configurable Control Signal Sleep Time to Automated TLS Testing Scripts in #29
• Add support for help Flags in Project Scripts in #32
• Update and Improve Documentation and Clean Codebase in #33

Full Changelog: v0.2.1...v0.3.0

Important Notes

• Functionality is limited to Debian-based operating systems.

• If issues still occur with the automated TLS performance test control signalling, information on increasing the signal sleep delay can be seen in the Advanced Testing Customisation section of the TLS Performance Testing Instructions documentation file.

Future Development

For details on the project's development and upcoming features, see the project's GitHub Projects page:

PQC-Evaluation-Tools Project Page

We look forward to your feedback and contributions to this project!

v0.2.1

Release v0.2.1

Welcome to the version 0.2.1 release of the pqc-evaluation-tools project!

Release Description

Overview

The version 0.2.0 release updated the pqc-evaluation-tools benchmark suite to utilise the most recent versions of the OQS project's PQC implementations. Including, the latest algorithms supported by Liboqs, the use of the improved OQS-Provider library compared to the previous OQS-OpenSSL library, and support for the testing and parsing of Hybrid-PQC algorithms within TLS. This version also improved the general setup of the suite alongside various optimisations and condensing of automation script files.

This small update in version 0.2.1 included a minor fix that resolved an issue with the Python parsing script where trying to parse both Liboqs and OQS-Provider results caused an error. This bug was sadly overlooked, and is now addressed in this minor release.

Features

The project provides automation for:

• Compiling and configuration of the OQS, ARM PMU, and OpenSSL dependency libraries.

• Gathering PQC computational performance data, including CPU and memory usage metrics using the Liboqs library.

• Gathering Networking performance data for the integration of PQC schemes in the TLS 1.3 protocol by utilising the OpenSSL 3.2.1 and OQS-Provider libraries.

• Coordinated testing of PQC TLS handshakes using either the loopback interface or a physical network connection between a server and client device.

• Parsing of the PQC performance data, where data from multiple machines can be parsed, averaged, and then compared against each other.

Change Log

As version 0.2.1 contains only one minor change to the code, to address a serious bug, the information on the changes in v0.2.0 compared to v0.1.0 are included here as well for reference. Future releases will return to a normal changelog format.

Version 0.2.0

A list of the key changes in this release are as follows:

• Refactored Liboqs auto testing scripts into single script and set Liboqs build to use custom OpenSSL-3.2.1

• Update to APT packages dependencies

• TLS handshake tests refactored for PQC testing using OQS-Provider and OpenSSL 3.2.1

• TLS handshake tests refactored for classic algorithm testing using OQS-Provider and OpenSSL 3.2.1

• TLS speed tests refactored for testing using OQS-Provider and OpenSSL 3.2.1

• Added support for the testing and parsing of Hybrid-PQC algorithms when integrated into TLS

• Optimisation of Python Parsing Scripts

• Added backwards comparability in parsing scripts for older versions of Python

• Added exception handling to account for missing Falcon memory metrics data on ARM devices

• Overall Improvements to Code Formatting and Commenting

• Added support for dynamically getting what algorithms are supported by the OQS libraries

• Added safe-setup functionality to main setup script to allow for a fallback to the last tested OQS libraries

• Resolved Issue where users had to be in the same directory when executing a script in order for it to function

A full list of the changes made can viewed here:

Full Changelog: v0.1.0-alpha...v0.2.0

Version 0.2.1

• Resolved bug where parsing both liboqs and oqs-provider results failed

Known Issues

• Manual copying of generated certificate and private key files for the TLS performance benchmarks required.

• Limited support for system architectures and operating systems.

• Issue with gathering full memory usage metrics for the Falcon Algorithm on ARM devices.

Important Notes

• This is an early version of the project, functionality is limited to debian based operating systems.

We look forward to your feedback and contributions to this project!

v0.2.0

Release v0.2.0

Welcome to the version 0.2.0 release of the pqc-evaluation-tools project!

Release Description

Overview

This release updates the pqc-evaluation-tools benchmark suite to utilise the most recent versions of the OQS project's PQC implementations. Including, the latest algorithms supported by Liboqs, the use of the improved OQS-Provider library compared to the previous OQS-OpenSSL library, and support for the testing and parsing of Hybrid-PQC algorithms within TLS. This version also improves the general setup of the suite alongside various optimisations and condensing of automation script files.

Features

The project provides automation for:

• Compiling and configuration of the OQS, ARM PMU, and OpenSSL dependency libraries.

• Gathering PQC computational performance data, including CPU and memory usage metrics using the Liboqs library.

• Gathering Networking performance data for the integration of PQC schemes in the TLS 1.3 protocol by utilising the OpenSSL 3.2.1 and OQS-Provider libraries.

• Coordinated testing of PQC TLS handshakes using either the loopback interface or a physical network connection between a server and client device.

• Parsing of the PQC performance data, where data from multiple machines can be parsed, averaged, and then compared against each other.

Change Log

A list of the key changes in this release are as follows:

• Refactored Liboqs auto testing scripts into single script and set Liboqs build to use custom OpenSSL-3.2.1

• Update to APT packages dependencies

• TLS handshake tests refactored for PQC testing using OQS-Provider and OpenSSL 3.2.1

• TLS handshake tests refactored for classic algorithm testing using OQS-Provider and OpenSSL 3.2.1

• TLS speed tests refactored for testing using OQS-Provider and OpenSSL 3.2.1

• Added support for the testing and parsing of Hybrid-PQC algorithms when integrated into TLS

• Optimisation of Python Parsing Scripts

• Added backwards comparability in parsing scripts for older versions of Python

• Added exception handling to account for missing Falcon memory metrics data on ARM devices

• Overall Improvements to Code Formatting and Commenting

• Added support for dynamically getting what algorithms are supported by the OQS libraries

• Added safe-setup functionality to main setup script to allow for a fallback to the last tested OQS libraries

• Resolved Issue where users had to be in the same directory when executing a script in order for it to function

A full list of the changes made can viewed here:

Full Changelog: v0.1.0-alpha...v0.2.0

Known Issues

• Manual copying of generated certificate and private key files for the TLS performance benchmarks required.

• Limited support for system architectures and operating systems.

• Issue with gathering full memory usage metrics for the Falcon Algorithm on ARM devices.

Important Notes

• This is an early version of the project, functionality is limited to x86 Linux devices and Raspberry Pis. However, future updates will address this issue.

We look forward to your feedback and contributions to this project!

v0.1.0-alpha

Release v0.1.0-alpha

Welcome to the first pre-release of the PQC Evaluation Tools Project!

Features

• Automated PQC performance benchmarking using Liboqs library.

• Automated PQC integration into TLS performance benchmarking using OQS-OpenSSL library.

• Parsing scripts to transform raw benchmark data into workable CSV files.

• Automated setup and build scripts for both Liboqs and OQS-OpenSSL libraries.

• Supports benchmarking on x86 Linux devices and Raspberry Pi's.

Bug Fixes

This being the first pre-release, there are no bug fixes relative to a previous version.

Known Issues

• Automated graph generation is currently under development and will be included in future releases.

• Manual copying of generated certificate and private key files for the TLS performance benchmarks required.

• Limited support for system architectures and operating systems

• Limited documentation

Important Notes

• This is an early version of the project, functionality is limited to x86 Linux devices and Raspberry Pis. However, future updates will address this issue.

• Activating the automated tools will delete all results currently stored in the up-results directory. To retain previous results, these should be moved to another location prior to re-running the automated test tool.

We look forward to your feedback and contributions to this project!

Full Changelog: https://github.com/crt26/pqc-evaluation-tools/commits/v0.1.0-alpha