Skip to content

Conversation

@rtb-12
Copy link
Contributor

@rtb-12 rtb-12 commented Dec 30, 2025

[CI/CD] Add Victoria Metrics scraping to fuzzy load test workflow

Description

This PR adds Victoria Metrics (vmagent) integration to the fuzzy load test workflow, enabling automatic metrics collection from merod processes during long-running performance tests. The implementation extracts duplicated vmagent management logic into reusable scripts, significantly reducing code duplication and improving maintainability.

Key Changes:

  • New Scripts: Created three reusable scripts for vmagent lifecycle management:
    • scripts/setup-vmagent.sh: Downloads, extracts, and configures vmagent binary
    • scripts/run-vmagent.sh: Starts vmagent with dynamic process discovery and config reload
    • scripts/cleanup-vmagent.sh: Handles graceful shutdown and cleanup
  • Workflow Refactoring: Refactored .github/workflows/fuzzy-load-test.yml to use scripts,
  • Error Handling: Added vmagent startup verification and improved error handling

Metrics Collection:

  • Automatically discovers merod processes listening on ports 2420-2499
  • Dynamically updates scrape configuration every 30 seconds as processes start/stop
  • Sends metrics to VictoriaMetrics with proper labeling (commit SHA, branch, workflow run ID, test case, etc.)

Test plan

Documentation update


Note

Adds VictoriaMetrics vmagent-based metrics collection to the fuzzy load tests with reusable scripts and workflow integration.

  • New scripts: scripts/setup-vmagent.sh, scripts/run-vmagent.sh, scripts/cleanup-vmagent.sh to download/setup vmagent, generate static-port scrape configs with periodic reloads, and perform graceful cleanup
  • Workflow updates: .github/workflows/fuzzy-load-test.yml now sets up vmagent, runs tests with metrics collection for both kv-store and kv-store-with-handlers, captures start/end times and exit codes, and uploads vmagent logs/config as artifacts
  • Temporary testing config: enable pull_request trigger and reduce default duration to 5 minutes (noted TODO to revert)
  • Reliability: stricter error handling and explicit cleanup to ensure vmagent shutdown and final metrics flush

Written by Cursor Bugbot for commit 04c3710. This will update automatically on new commits. Configure here.

- Updated fuzzy load test workflows to use short commit hashes for naming.
- Improved logging in run-vmagent.sh to display process IDs for better tracking.
- Enhanced setup-vmagent.sh to safely extract the vmagent binary and verify its presence, including a fallback mechanism for different archive structures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants