Add Jetpack performance testing CI infrastructure #46287

LiamSarsfield · 2025-12-12T14:09:06Z

Addresses HOG-438: Create Jetpack Performance Tooling for LCP

Proposed changes:

Add performance testing infrastructure under tools/performance/ to measure wp-admin dashboard LCP (Largest
Contentful Paint) with Jetpack connected
Uses Docker for isolated WordPress environment with simulated WordPress.com connection (fake tokens + mocked API
with 200ms latency)
Includes CPU throttling calibration for consistent results across different machines
Posts metrics to CodeVitals for tracking over time

Other information:

Have you written new tests for your changes, if applicable?
Have you checked the E2E test CI results, and verified that your changes do not break them?
Have you tested your changes on WordPress.com, if applicable (if so, you'll see a generated comment below with
a script to run)?

Jetpack product discussion

pc9hqz-3Rb-p2

Does this pull request change what data or activity we track or use?

No

Testing instructions:

Prerequisites: Docker running, Node 18+, Jetpack built (pnpm jetpack build plugins/jetpack)

cd tools/performance
pnpm install
pnpm exec playwright install chromium
pnpm calibrate
pnpm test --skip-codevitals

Expected output: LCP measurement for wp-admin dashboard with Jetpack connected (simulated)

Introduces automated LCP (Largest Contentful Paint) measurement for the wp-admin dashboard with simulated Jetpack WordPress.com connection. Key components: - Docker environment with WordPress + simulated Jetpack connection - CPU throttling calibration for consistent results across CI agents - Playwright-based LCP measurement - CodeVitals integration for metric tracking Metric posted: wp-admin-dashboard-connection-sim-largestContentfulPaint

github-actions · 2025-12-12T14:10:12Z

Thank you for your PR!

When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:

✅ Include a description of your PR changes.
✅ Add a "[Status]" label (In Progress, Needs Review, ...).
✅ Add a "[Type]" label (Bug, Enhancement, Janitorial, Task).
✅ Add testing instructions.
✅ Specify whether this PR includes any changes to data or privacy.
✅ Add changelog entries to affected projects

This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖

Follow this PR Review Process:

Ensure all required checks appearing at the bottom of this PR are passing.
Make sure to test your changes on all platforms that it applies to. You're responsible for the quality of the code you ship.
You can use GitHub's Reviewers functionality to request a review.
When it's reviewed and merged, you will be pinged in Slack to deploy the changes to WordPress.com simple once the build is done.

If you have questions about anything, reach out in #jetpack-developers for guidance!

- Remove outdated 4-scenario comment (only 1 scenario exists) - Remove dead baseline comparison code that never executed - Trim JSDoc to single-line descriptions across all scripts - Update eslint config to allow minimal JSDoc

Changed the Docker startup sequence to prevent race conditions where WordPress containers interfere with WP-CLI's database operations: 1. Start only the db container 2. Wait for MySQL to be ready 3. Run WP-CLI setup (WordPress container NOT running) 4. Start WordPress containers This ensures WP-CLI has exclusive database access during setup, eliminating "table doesn't exist" errors caused by concurrent access. Changes: - run-performance-tests.js: Sequential container startup - setup-wordpress.sh: Simplified (removed HTTP wait logic) - docker-compose.yml: Removed wpcli depends_on wordpress

The import plugin is not available in this context, and the base config handles import resolution. Also ensure jsdoc rules are disabled for these utility scripts.

Dependencies (playwright, dotenv) are in tools/performance/package.json, not the monorepo root, so the import resolver can't find them.

- Log calibration file path and existence - Show throttle rate, target score, calibration time, and samples - Confirm throttling is applied via CDP on first iteration

…or tools/performance

github-actions · 2025-12-14T19:45:47Z

Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.

To test on WoA, go to the Plugins menu on a WoA dev site. Click on the "Upload" button and follow the upgrade flow to be able to upload, install, and activate the Jetpack Beta plugin. Once the plugin is active, go to Jetpack > Jetpack Beta, select your plugin (Jetpack or WordPress.com Site Helper), and enable the add/perf-testing-ci-mvp branch.
To test on Simple, run the following command on your sandbox:

bin/jetpack-downloader test jetpack add/perf-testing-ci-mvp

bin/jetpack-downloader test jetpack-mu-wpcom-plugin add/perf-testing-ci-mvp

Interested in more tips and information?

In your local development environment, use the jetpack rsync command to sync your changes to a WoA dev blog.
Read more about our development workflow here: PCYsg-eg0-p2
Figure out when your changes will be shipped to customers here: PCYsg-eg5-p2

jp-launch-control · 2025-12-14T20:03:10Z

Code Coverage Summary

This PR did not change code coverage!

That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷

Full summary · PHP report · JS report

…wordpress-jetpack-connected service, ensuring it starts only after the service is initiated, which helps prevent race conditions during container startup.

- Updated README.md to include new environment variables: CODEVITALS_URL, GIT_BRANCH, WP_ADMIN_USER, and WP_ADMIN_PASS. - Modified post-to-codevitals.js to streamline metric extraction by removing unused baseMetrics. - Improved run-performance-tests.js to prioritize GIT_COMMIT environment variable for git hash retrieval, ensuring accurate tracking during CI runs.

- Introduced an empty baseMetrics object in the payload to clarify that baseline normalization is not utilized in the performance metrics submission.

- Simplified browser launch to always use headless mode for consistency in performance calibration. - Removed conditional logic for headful mode, ensuring a streamlined execution in both local and CI environments.

LiamSarsfield · 2025-12-15T13:46:45Z

Hey @anomiex👋 would you mind taking a look at this when you get a chance?

This is the performance testing infrastructure I built during HACK week (more details here pc9hqz-3Rb-p2) it measures wp-admin dashboard LCP for Jetpack and posts results to CodeVitals.
It's a big PR, sorry about that. Most of it breaks down into:

scripts/ - Node.js orchestration and Playwright measurement code
docker/ - Docker Compose setup for WordPress + Jetpack environment
docker/mu-plugins/simulate-wpcom-connection.php - mu-plugin that fakes Jetpack connection and mocks WP.com API
responses

The mu-plugin is probably the most relevant bit for review from a Jetpack perspective as it intercepts pre_http_request to return mock responses for various WP.com endpoints.

Happy to walk through any of it if that's easier.

anomiex

Seems ok from a monorepo perspective. I didn't look too closely at the code.

If you're wanting a review of the faked-connection stuff in the mu-plugin, @Automattic/jetpack-vulcan would be the team to ask.

anomiex · 2025-12-15T16:03:12Z

tools/performance/README.md

+
+## CI Usage
+
+The test suite is designed to run in TeamCity. See build configuration for setup.


I wonder if it'd make more sense to run it in Actions rather than TeamCity, on each commit to trunk instead of backfilling weekly.

If you're running it in TeamCity, are you looking at the monorepo, or at https://github.com/Automattic/jetpack-production which has the already-built plugin?

anomiex · 2025-12-15T16:13:40Z

tools/performance/eslint.config.mjs

+			// Dependencies are in tools/performance/package.json, not monorepo root
+			// so the import resolver can't find them. Disable this rule.
+			'import/no-unresolved': 'off',


That doesn't matter, eslint will look in tools/performance/node_modules for files under here.

Also, when I remove this and run eslint, it doesn't seem to complain about anything.

github-actions bot added [Status] In Progress Docs labels Dec 12, 2025

github-actions bot added the [Status] Needs Author Reply We need more details from you. This label will be auto-added until the PR meets all requirements. label Dec 12, 2025

LiamSarsfield added 2 commits December 12, 2025 14:39

Fix eslint import/no-unresolved for standalone dependencies

e0b91db

LiamSarsfield changed the title ~~Add Jetpack performance testing CI infrastructure~~ [Do not merge] Add Jetpack performance testing CI infrastructure Dec 12, 2025

LiamSarsfield added the DO NOT MERGE don't merge it! label Dec 12, 2025

LiamSarsfield added 6 commits December 12, 2025 16:42

Fix ESLint config: remove import/no-unresolved rule

f9dfbd5

The import plugin is not available in this context, and the base config handles import resolution. Also ensure jsdoc rules are disabled for these utility scripts.

Disable import/no-unresolved rule for performance scripts

63dca30

Dependencies (playwright, dotenv) are in tools/performance/package.json, not the monorepo root, so the import resolver can't find them.

Add detailed calibration verification logging

6595660

- Log calibration file path and existence - Show throttle rate, target score, calibration time, and samples - Confirm throttling is applied via CDP on first iteration

Add tools/performance

bbdb6ea

Update pnpm-lock.yaml to include dotenv and playwright dependencies f…

c768b58

…or tools/performance

LiamSarsfield added 5 commits December 14, 2025 20:14

Update docker-compose.yml to change the dependency condition for the …

2492d1e

…wordpress-jetpack-connected service, ensuring it starts only after the service is initiated, which helps prevent race conditions during container startup.

Add baseMetrics placeholder in post-to-codevitals.js

df0cfd2

- Introduced an empty baseMetrics object in the payload to clarify that baseline normalization is not utilized in the performance metrics submission.

Refactor browser launch configuration in calibrate-throttling.js

ce3b20d

- Simplified browser launch to always use headless mode for consistency in performance calibration. - Removed conditional logic for headful mode, ensuring a streamlined execution in both local and CI environments.

Merge remote-tracking branch 'origin/trunk' into add/perf-testing-ci-mvp

55e48ba

LiamSarsfield added [Type] Infrastructure and removed [Status] Needs Author Reply We need more details from you. This label will be auto-added until the PR meets all requirements. labels Dec 15, 2025

LiamSarsfield requested a review from anomiex December 15, 2025 13:46

LiamSarsfield marked this pull request as ready for review December 15, 2025 13:46

LiamSarsfield requested review from dilirity and kraftbj December 15, 2025 13:47

LiamSarsfield removed the DO NOT MERGE don't merge it! label Dec 15, 2025

LiamSarsfield changed the title ~~[Do not merge] Add Jetpack performance testing CI infrastructure~~ Add Jetpack performance testing CI infrastructure Dec 15, 2025

LiamSarsfield added [Status] Needs Review This PR is ready for review. and removed [Status] In Progress labels Dec 15, 2025

anomiex reviewed Dec 15, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add Jetpack performance testing CI infrastructure #46287

Add Jetpack performance testing CI infrastructure #46287

Uh oh!

LiamSarsfield commented Dec 12, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 12, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Dec 14, 2025 •

edited

Loading

Uh oh!

jp-launch-control bot commented Dec 14, 2025 •

edited

Loading

Uh oh!

LiamSarsfield commented Dec 15, 2025

Uh oh!

anomiex left a comment

Uh oh!

anomiex Dec 15, 2025

Uh oh!

anomiex Dec 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		## CI Usage

		The test suite is designed to run in TeamCity. See build configuration for setup.

Add Jetpack performance testing CI infrastructure #46287

Are you sure you want to change the base?

Add Jetpack performance testing CI infrastructure #46287

Uh oh!

Conversation

LiamSarsfield commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed changes:

Other information:

Jetpack product discussion

Does this pull request change what data or activity we track or use?

Testing instructions:

Uh oh!

github-actions bot commented Dec 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jp-launch-control bot commented Dec 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Coverage Summary

Uh oh!

LiamSarsfield commented Dec 15, 2025

Uh oh!

anomiex left a comment

Choose a reason for hiding this comment

Uh oh!

anomiex Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

anomiex Dec 15, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LiamSarsfield commented Dec 12, 2025 •

edited

Loading

github-actions bot commented Dec 12, 2025 •

edited

Loading

github-actions bot commented Dec 14, 2025 •

edited

Loading

jp-launch-control bot commented Dec 14, 2025 •

edited

Loading