[Do not merge] Add Jetpack performance testing infrastructure #46238
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduce LCP baseline testing for wp-admin dashboard across four scenarios: - Baseline WordPress (no Jetpack) - Jetpack disconnected - Jetpack offline mode (JETPACK_DEV_DEBUG) - Jetpack connected (simulated with mocked API + latency) Features: - Docker-based isolated WordPress instances - Playwright LCP measurements with statistical analysis - CodeVitals integration for trend tracking - TeamCity CI setup documentation - Quickstart script for local development Tools: tools/performance/
LiamSarsfield
changed the title
Contributor
|
Thank you for your PR! When contributing to Jetpack, we have a few suggestions that can help us test and review your patch:
This comment will be updated as you work on your PR and make changes. If you think that some of those checks are not needed for your PR, please explain why you think so. Thanks for cooperation 🤖 Follow this PR Review Process:
If you have questions about anything, reach out in #jetpack-developers for guidance! |
github-actions
bot
added
the
[Status] Needs Author Reply
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
…script improvements - Added MYSQL_READY_TIMEOUT_SECONDS and WP_READY_TIMEOUT_SECONDS to .env.example and README for customizable readiness checks. - Updated docker:reset script to skip codevitals posting for faster resets. - Improved quickstart.sh to validate monorepo structure and ensure correct directory context. - Enhanced measure-lcp.js to use waitForURL for navigation stability and added more reliable LCP finalization logic. - Updated post-to-codevitals.js to handle JSON parsing errors gracefully and added comments for clarity. - Refined scenarios.js to ensure environment variable values are trimmed before use. These changes improve the robustness and usability of the performance testing tools.
- Introduced new tools/performance directory to enhance performance testing capabilities. - Added dotenv and playwright dependencies to pnpm-lock.yaml for improved environment setup. - Updated package.json scripts to clarify browser setup command. - Enhanced README.md with new script and configuration details for better usability. - Improved error handling in simulate-wpcom-connection.php for module activation in Docker environments. - Added macOS compatibility in run-performance-tests.js to handle file attribute issues. These changes streamline the performance testing process and improve the overall developer experience.
Contributor
|
Are you an Automattician? Please test your changes on all WordPress.com environments to help mitigate accidental explosions.
Interested in more tips and information?
|
Code Coverage SummaryThis PR did not change code coverage! That could be good or bad, depending on the situation. Everything covered before, and still is? Great! Nothing was covered before? Not so great. 🤷 |
- Modified the page.click method to include a timeout parameter for improved reliability during navigation to the wp-admin dashboard. This change aims to reduce potential race conditions and ensure smoother performance testing.
- Modified Docker commands in run-performance-tests.js to include a project name ('jetpack-perf') for better container management and isolation during performance tests.
This change enhances the clarity and reliability of the Docker setup process.
…rio handling - Updated extractScenarioMetrics function to accept scenario object, allowing for more flexible metric key usage. - Enhanced postToCodeVitals function to skip scenarios not marked for CodeVitals posting and added validation for metrics before posting. - Introduced new metricKey and postToCodeVitals properties in scenarios.js for better configuration of performance metrics. These changes streamline the metric extraction process and improve the robustness of the CodeVitals integration.
- Changed WordPress container ports in docker-compose.yml to dynamic ports for better flexibility. - Added functionality in run-performance-tests.js to discover dynamic ports and set corresponding environment variables for WordPress instances. - Updated scenarios.js to associate Docker services with scenarios for improved clarity in performance testing. These changes streamline the Docker setup process and enhance the performance testing workflow.
- Implemented functionality to automatically detect the site URL from the request, supporting dynamic ports for improved flexibility in the Docker environment.
…tools - Introduced a new script (calibrate-throttling.js) to calibrate CPU throttling for consistent performance testing across different machines. - Updated package.json to include a new script command for calibration. - Enhanced measure-lcp.js to utilize the calibration data for CPU throttling during LCP measurements. - Added shared statistics utility functions (stats.js) for calculating median, mean, and standard deviation, improving statistical analysis in performance tests. - Updated .gitignore to exclude the calibration.json file.
- Updated the browser launch configuration to use headful mode locally and headless mode in CI environments, improving calibration accuracy. - Enhanced logging to indicate the mode in which the browser is launched, providing better visibility during performance testing.
- Added error handling to the WordPress core installation process to retry installation after dropping and recreating the database if the initial attempt fails due to corrupted tables. - Improved user feedback during installation with clear messages indicating the status of the installation and any necessary database repairs.
… handling - Enhanced quickstart.sh to enable pnpm via corepack instead of global installation. - Updated README.md to reflect changes in CodeVitals metrics, focusing on the Jetpack connected (simulated) scenario. - Modified docker-compose.yml comments to clarify dynamic port assignment for WordPress instances. - Removed hardcoded localhost URLs from setup-wordpress.sh and improved messaging regarding dynamic ports. - Added functionality in run-performance-tests.js to update WordPress database options with discovered dynamic URLs. - Updated scenarios.js to include wpPath for each scenario, facilitating dynamic URL updates.
- Removed GIT_BASE_HASH references from .env.example, post-to-codevitals.js, and run-performance-tests.js to simplify git information handling. - Updated getGitInfo function to eliminate baseHash, focusing on hash and branch only, reflecting the main branch tracking for performance tests. - Adjusted console logging to omit base hash details for cleaner output during performance testing.
- Added diagnostic messages to verify the existing WordPress installation state, including user and table listings. - Implemented checks to ensure the database is empty after recreation and added a delay for readiness. - Included logic to create the admin user if it is missing after installation, improving the robustness of the setup process.
- Added a check to create the database if it does not exist, improving the robustness of the installation process. - Included error handling to notify users if the database creation fails, enhancing feedback during setup.
- Added detailed diagnostic messages to track database state before and after installation attempts, improving visibility during setup. - Implemented a loop to ensure the database is empty before retrying installation, addressing potential race conditions with WordPress containers. - Enhanced error handling for the WordPress installation process, including checks for critical tables and improved feedback on installation status.
…up script - Added a loop to verify database accessibility before proceeding with installation, preventing race conditions. - Included detailed feedback on the database connection attempts, enhancing error handling and user visibility during setup.
…setup - Eliminated the wait_for_wordpress function that checked HTTP readiness of WordPress containers, which caused race conditions with WP-CLI during database setup. - Updated comments to clarify the new approach of allowing WP-CLI to handle database initialization directly, avoiding simultaneous table creation attempts.
- Updated the WordPress Docker setup script to ensure that WP-CLI runs before starting WordPress containers, eliminating race conditions during database operations. - Enhanced the script to start only the database container initially, followed by a WP-CLI setup, and then the WordPress containers. - Improved comments and logging for clarity on the setup steps and their order, ensuring a smoother installation process.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses HOG-438: Create Jetpack Performance Tooling for LCP
Proposed changes:
tools/performance/to measure wp-admin dashboard LCP (LargestContentful Paint) with Jetpack connected
with 200ms latency)
Other information:
a script to run)?
Jetpack product discussion
N/A - internal tooling
Does this pull request change what data or activity we track or use?
No
Testing instructions:
Prerequisites: Docker running, Node 18+, Jetpack built (
pnpm jetpack build plugins/jetpack)