Add JMH benchmarks to CI with performance regression detection #26

Copilot · 2025-10-08T12:16:06Z

Explore repository structure and understand existing JMH benchmarks
Verify project builds successfully
Add JMH Gradle tasks to message-encoding/build.gradle
- Added jmhJar task to create executable JAR with benchmarks
- Added jmh task to run benchmarks and generate JSON results
- Added jmhReport task to run benchmarks with text output
- Made tasks configurable via project properties (jmh.fork, jmh.warmup, jmh.iterations)
- Excluded signature files to prevent security exceptions
Add GitHub Actions workflow job to run JMH benchmarks
- Added new jmh-benchmarks job to gradle.yml
- Job runs both jmh and jmhReport tasks
- Uploads JSON and text results as artifacts
- Displays summary in workflow logs
Verify build and JMH tasks work correctly
- Tested jmhJar task builds successfully
- Tested jmh task generates JSON results
- Tested jmhReport task generates text results
- Verified all 8 benchmarks are available (Base58 encoding/decoding and Ed25519 point validation)
Update documentation to reflect CI integration
- Added instructions for running benchmarks via Gradle
- Documented configurable parameters
- Added CI integration section
- Listed result file locations
Add performance regression detection
- Created Python script (.github/scripts/compare_benchmarks.py) to compare benchmark results
- Script compares PR results against master branch baseline
- Automatically fails CI if performance regresses by >10%
- Only runs on pull requests, not on master branch commits
- Handles missing baselines gracefully by building them on-demand
Fix CI workflow issues
- Fixed git checkout to use FETCH_HEAD instead of origin/master
- Added check for JMH tasks existence on master before baseline build
- Handles first-time setup when master doesn't have benchmarks yet
- Saves baseline results outside build directory to avoid conflicts
- Removed Python cache files and updated .gitignore

Summary

Successfully integrated JMH benchmarks into CI pipeline with regression detection:

Gradle Tasks: jmh (JSON output), jmhReport (text output), jmhJar (build executable JAR)
CI Job: Runs automatically on every push/PR, uploads results as artifacts
Regression Detection: Compares PR benchmarks vs master, fails if >10% slower (when master has JMH tasks)
Benchmarks: 4 Base58 operations + 4 Ed25519 curve validation operations
Results: Saved to message-encoding/build/reports/jmh/ directory
First-time Setup: Gracefully handles case where master doesn't have JMH tasks yet

Original prompt

This section details on the original issue you should resolve

<issue_title>Add Base58 encoding JMH tests to CI and publish results</issue_title>
<issue_description>There are JMH tests for the comparison for some base58 and crypto operations. We should run them as part of CI and publish the results.</issue_description>

Comments on the Issue (you are @copilot in this section)

Fixes #25

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: thebeanogamer <[email protected]>

thebeanogamer

Automatically fail the CI check if performance has meaningfully regressed since the previous commit on the default branch

Copilot

Pull Request Overview

This PR integrates JMH benchmarks into the CI pipeline to automatically run performance benchmarks and publish results as artifacts. The implementation addresses the requirement to run existing Base58 encoding and crypto operation benchmarks as part of CI.

Added three new Gradle tasks for building, running, and reporting JMH benchmarks with configurable parameters
Integrated a new GitHub Actions job that runs benchmarks on every push/PR and uploads results as artifacts
Updated documentation with comprehensive instructions for running benchmarks locally and accessing CI results

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 6 comments.

File	Description
message-encoding/build.gradle	Added three Gradle tasks (jmhJar, jmh, jmhReport) for benchmark execution with configurable parameters
.github/workflows/gradle.yml	Added jmh-benchmarks job to run benchmarks in CI and upload results as artifacts
message-encoding/src/jmh-test/README.md	Enhanced documentation with detailed instructions for running benchmarks and accessing CI results

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

Copilot · 2025-10-08T12:45:11Z

message-encoding/build.gradle

+    // JMH command line options
+    args = [
+        '-rf', 'json',                              // Result format: JSON
+        '-rff', "${project.buildDir}/reports/jmh/results.json", // Result file


Use layout.buildDirectory instead of the deprecated project.buildDir property. This should be layout.buildDirectory.dir('reports/jmh/results.json').get().asFile.absolutePath.

Copilot · 2025-10-08T12:45:12Z

message-encoding/build.gradle

+
+    // Create reports directory
+    doFirst {
+        file("${project.buildDir}/reports/jmh").mkdirs()


Use layout.buildDirectory instead of the deprecated project.buildDir property. This should be layout.buildDirectory.dir('reports/jmh').get().asFile.mkdirs().

Copilot · 2025-10-08T12:45:12Z

message-encoding/build.gradle

+
+    // Print results location after execution
+    doLast {
+        println "JMH benchmark results written to: ${project.buildDir}/reports/jmh/results.json"


Use layout.buildDirectory instead of the deprecated project.buildDir property. This should reference layout.buildDirectory.dir('reports/jmh/results.json').get().asFile.absolutePath.

Copilot · 2025-10-08T12:45:12Z

message-encoding/build.gradle

+    // JMH command line options
+    args = [
+        '-rf', 'text',                              // Result format: text
+        '-rff', "${project.buildDir}/reports/jmh/results.txt", // Result file


Use layout.buildDirectory instead of the deprecated project.buildDir property. This should be layout.buildDirectory.dir('reports/jmh/results.txt').get().asFile.absolutePath.

Copilot · 2025-10-08T12:45:12Z

message-encoding/build.gradle

+    ]
+
+    doFirst {
+        file("${project.buildDir}/reports/jmh").mkdirs()


Use layout.buildDirectory instead of the deprecated project.buildDir property. This should be layout.buildDirectory.dir('reports/jmh').get().asFile.mkdirs().

Copilot · 2025-10-08T12:45:13Z

message-encoding/build.gradle

+    }
+
+    doLast {
+        println "JMH benchmark results written to: ${project.buildDir}/reports/jmh/results.txt"


Use layout.buildDirectory instead of the deprecated project.buildDir property. This should reference layout.buildDirectory.dir('reports/jmh/results.txt').get().asFile.absolutePath.

Suggested change

println "JMH benchmark results written to: ${project.buildDir}/reports/jmh/results.txt"

println "JMH benchmark results written to: ${layout.buildDirectory.dir('reports/jmh/results.txt').get().asFile.absolutePath}"

thebeanogamer · 2025-10-08T12:50:30Z

@copilot please action all the reviews

Co-authored-by: thebeanogamer <[email protected]>

Copilot · 2025-10-08T12:57:41Z

Automatically fail the CI check if performance has meaningfully regressed since the previous commit on the default branch

Added performance regression detection in commit {commit_hash}. The CI now automatically compares PR benchmark results against the master branch baseline and fails if any benchmark shows >10% performance regression. The comparison script is at .github/scripts/compare_benchmarks.py and only runs on pull requests.

thebeanogamer · 2025-10-08T13:13:42Z

@copilot

CI is failing, can you tell me why?

If an issue already exists, just update it rather than raising a new one.

Put the Python scripts in the repo rather than inlining them in the YAML

- Use FETCH_HEAD instead of origin/master for checkout - Check if JMH tasks exist on master before attempting baseline build - Handle first-time setup gracefully when master doesn't have JMH tasks - Save baseline results outside build directory to avoid conflicts - Add clearer status messages with emoji indicators Co-authored-by: thebeanogamer <[email protected]>

Copilot · 2025-10-08T13:21:29Z

@copilot

CI is failing, can you tell me why?
...

Fixed the CI workflow issues in commit c8f90c9:

Issues identified and resolved:

✅ Git checkout was trying to use origin/master which doesn't work - now uses FETCH_HEAD
✅ Master branch doesn't have JMH tasks yet (they're being added in this PR) - added check to skip baseline if tasks don't exist
✅ Build directory conflicts when switching branches - now saves baseline results outside build dir
✅ Python script is already in the repo at .github/scripts/compare_benchmarks.py (not inlined)

The workflow will now gracefully handle first-time setup and only run comparisons when master has the JMH infrastructure.

Co-authored-by: thebeanogamer <[email protected]>

Initial plan

ba8d67e

Copilot AI assigned Copilot and thebeanogamer Oct 8, 2025

Copilot started work on behalf of thebeanogamer October 8, 2025 12:16 View session

Copilot AI and others added 3 commits October 8, 2025 12:26

Add JMH benchmark Gradle tasks with configurable parameters

edc2802

Co-authored-by: thebeanogamer <[email protected]>

Add JMH benchmarks job to GitHub Actions CI workflow

c13ab1d

Co-authored-by: thebeanogamer <[email protected]>

Update JMH benchmark documentation with Gradle commands and CI info

905cfe2

Co-authored-by: thebeanogamer <[email protected]>

Copilot AI changed the title ~~[WIP] Add Base58 encoding JMH tests to CI~~ Add JMH benchmarks to CI and publish results Oct 8, 2025

Copilot finished work on behalf of thebeanogamer October 8, 2025 12:32

Copilot AI requested a review from thebeanogamer October 8, 2025 12:32

thebeanogamer requested changes Oct 8, 2025

View reviewed changes

thebeanogamer requested a review from Copilot October 8, 2025 12:42

Copilot AI reviewed Oct 8, 2025

View reviewed changes

Copilot started work on behalf of thebeanogamer October 8, 2025 12:50 View session

Add performance regression detection to JMH benchmarks CI

14beb3a

Co-authored-by: thebeanogamer <[email protected]>

Copilot AI changed the title ~~Add JMH benchmarks to CI and publish results~~ Add JMH benchmarks to CI with performance regression detection Oct 8, 2025

Copilot AI requested a review from thebeanogamer October 8, 2025 12:59

Copilot finished work on behalf of thebeanogamer October 8, 2025 12:59

Copilot started work on behalf of thebeanogamer October 8, 2025 13:14 View session

Remove Python cache files and update .gitignore

a82ec21

Co-authored-by: thebeanogamer <[email protected]>

Copilot finished work on behalf of thebeanogamer October 8, 2025 13:23

Merge branch 'master' into copilot/add-base58-jmh-tests

7a7ffbb

	println "JMH benchmark results written to: ${project.buildDir}/reports/jmh/results.txt"
	println "JMH benchmark results written to: ${layout.buildDirectory.dir('reports/jmh/results.txt').get().asFile.absolutePath}"

Add JMH benchmarks to CI with performance regression detection #26

Are you sure you want to change the base?

Add JMH benchmarks to CI with performance regression detection #26

Uh oh!

Conversation

Copilot AI commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Comments on the Issue (you are @copilot in this section)

Uh oh!

thebeanogamer left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Oct 8, 2025

Choose a reason for hiding this comment

Uh oh!

thebeanogamer commented Oct 8, 2025

Uh oh!

Copilot AI commented Oct 8, 2025

Uh oh!

thebeanogamer commented Oct 8, 2025

Uh oh!

Copilot AI commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Oct 8, 2025 •

edited

Loading