[Security Solution] Parallelize slow Rule Management tests #234930
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nikitaindik
added
release_note:skip
6 tasks
nikitaindik
force-pushed
the
make-slow-rm-tests-run-in-parallel
branch
from
968bdf0 to
5ed1d9c
Compare
Contributor
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
Contributor
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
maximpn
reviewed
Sep 16, 2025
Contributor
maximpn
left a comment
maximpn
left a comment
There was a problem hiding this comment.
@nikitaindik Thanks for splitting our tests into smaller groups to make them running faster 🙏
To be honest I have concerns with merging common_fields and type_specific_fields dirs into a single fields directory. Common/type specific fields separation simplifies navigation and mirrors folders structure in API integration tests. We can keep the previous folder's structure and use the same approach you implemented in this PR by either providing the hardcoded number of groups or scan for folders starting with config_. There are Node.js libs to scan folders structure so it's not necessary to implement it from scratch.
Nit - It'd be nice to see tests run time comparison in before/after fashion.
...tion_tests__/rules_upgrade/upgrade_rule_after_preview/fields/base.jest.integration.config.js
Outdated
Show resolved
Hide resolved
| */ | ||
| function getTestsForConfig(configNumber) { | ||
| const groupIndex = configNumber - 1; | ||
| const configTestFiles = allTestFiles.filter((file, index) => index % totalGroups === groupIndex); |
Contributor
There was a problem hiding this comment.
The implementation could be simplified by using Lodash's chunk utility. A test group may pick a chunk by number. It should work the same way as the current implementation but it's gonna be shorter.
Contributor
Author
There was a problem hiding this comment.
I tried doing it with chunk, but I think it turned out even a bit more complicated.
const configTestFiles = chunk(testFiles, Math.ceil(testFiles.length / totalGroups))[groupIndex];
Also one minor downside of the chunk approach is that if we later add more groups, it won't balance them nicely without writing additional code.
Like, let's say if you have 13 files and 4 groups, it'll distribute them like: [4, 4, 4, 1] and not [4, 3, 3, 3].
Contributor
Author
|
buildkite test this |
Contributor
Author
|
@maximpn I've refactored according to what we discussed. Please take another look. |
maximpn
approved these changes
Sep 17, 2025
Contributor
maximpn
left a comment
maximpn
left a comment
There was a problem hiding this comment.
@nikitaindik Thanks for addressing my comments 🙏
I have some last thoughts though since field test files were moved under fields folder. It should be considered as an optional comment. We could do one of the following
- As field test files are under
fieldsfoldercommon_fieldsandtype_specific_fieldscould be renamed tocommonandtype_specific - It's still possible to preserve the previous folders structure by including
diff_view_options.test.tsandrule_upgrade_button.test.tsinto dynamic groups.
Contributor
Author
|
@maximpn Thanks for giving it another pass! 👍 I've renamed the dirs to get rid of the |
Contributor
⏳ Build in-progress, with failures
Failed CI StepsHistory
cc @nikitaindik |
Contributor
|
Starting backport for target branches: 8.19, 9.1 |
Contributor
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
nikitaindik
added a commit
to nikitaindik/kibana
that referenced
this pull request
Sep 18, 2025
…34930) **Resolves: elastic#232644 This PR splits long-running API integration and Jest configs into smaller configs to reduce overall runtime. We can’t rely on Buildkite’s `parallelism` parameter here, since it’s only works for Cypress steps. ## Changes - **API Integration tests** - Split the prebuilt rules config into 3 smaller configs. - **Rule Upgrade Field Tests (Jest)** - Merged `common_fields` and `type_specific_fields` dirs into a single `fields` directory. - Replaced 2 configs with 4 smaller ones. - Implemented round-robin logic to evenly distribute tests across the 4 configs. - **Alternative**: manually distribute tests into numbered directories. - *Pros*: simpler setup. - *Cons*: requires deciding where each new test should live when adding new fields. - Reviewers, tell me if adding this distribution logic is an overkill in your opinion. (cherry picked from commit 928d4b2) # Conflicts: # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/index.ts
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
nikitaindik
added a commit
to nikitaindik/kibana
that referenced
this pull request
Sep 18, 2025
…34930) **Resolves: elastic#232644 This PR splits long-running API integration and Jest configs into smaller configs to reduce overall runtime. We can’t rely on Buildkite’s `parallelism` parameter here, since it’s only works for Cypress steps. ## Changes - **API Integration tests** - Split the prebuilt rules config into 3 smaller configs. - **Rule Upgrade Field Tests (Jest)** - Merged `common_fields` and `type_specific_fields` dirs into a single `fields` directory. - Replaced 2 configs with 4 smaller ones. - Implemented round-robin logic to evenly distribute tests across the 4 configs. - **Alternative**: manually distribute tests into numbered directories. - *Pros*: simpler setup. - *Cons*: requires deciding where each new test should live when adding new fields. - Reviewers, tell me if adding this distribution logic is an overkill in your opinion. (cherry picked from commit 928d4b2) # Conflicts: # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/prebuilt_rules/customization_enabled/index.ts
nikitaindik
added a commit
that referenced
this pull request
Sep 18, 2025
…4930) (#235565) # Backport This will backport the following commits from `main` to `9.1`: - [[Security Solution] Parallelize slow Rule Management tests (#234930)](#234930) <!--- Backport version: 10.0.2 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Nikita Indik","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-09-18T13:27:31Z","message":"[Security Solution] Parallelize slow Rule Management tests (#234930)\n\n**Resolves: https://github.com/elastic/kibana/issues/232644**\n\nThis PR splits long-running API integration and Jest configs into\nsmaller configs to reduce overall runtime.\nWe can’t rely on Buildkite’s `parallelism` parameter here, since it’s\nonly works for Cypress steps.\n\n## Changes\n- **API Integration tests**\n - Split the prebuilt rules config into 3 smaller configs.\n\n- **Rule Upgrade Field Tests (Jest)**\n- Merged `common_fields` and `type_specific_fields` dirs into a single\n`fields` directory.\n - Replaced 2 configs with 4 smaller ones.\n- Implemented round-robin logic to evenly distribute tests across the 4\nconfigs.\n- **Alternative**: manually distribute tests into numbered directories.\n - *Pros*: simpler setup. \n- *Cons*: requires deciding where each new test should live when adding\nnew fields.\n- Reviewers, tell me if adding this distribution logic is an overkill in\nyour opinion.","sha":"928d4b224ffaadc6f171482f6b95e14e76da912d","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","backport:version","v9.2.0","v8.19.5","v9.1.5"],"title":"[Security Solution] Parallelize slow Rule Management tests","number":234930,"url":"https://github.com/elastic/kibana/pull/234930","mergeCommit":{"message":"[Security Solution] Parallelize slow Rule Management tests (#234930)\n\n**Resolves: https://github.com/elastic/kibana/issues/232644**\n\nThis PR splits long-running API integration and Jest configs into\nsmaller configs to reduce overall runtime.\nWe can’t rely on Buildkite’s `parallelism` parameter here, since it’s\nonly works for Cypress steps.\n\n## Changes\n- **API Integration tests**\n - Split the prebuilt rules config into 3 smaller configs.\n\n- **Rule Upgrade Field Tests (Jest)**\n- Merged `common_fields` and `type_specific_fields` dirs into a single\n`fields` directory.\n - Replaced 2 configs with 4 smaller ones.\n- Implemented round-robin logic to evenly distribute tests across the 4\nconfigs.\n- **Alternative**: manually distribute tests into numbered directories.\n - *Pros*: simpler setup. \n- *Cons*: requires deciding where each new test should live when adding\nnew fields.\n- Reviewers, tell me if adding this distribution logic is an overkill in\nyour opinion.","sha":"928d4b224ffaadc6f171482f6b95e14e76da912d"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.1"],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/234930","number":234930,"mergeCommit":{"message":"[Security Solution] Parallelize slow Rule Management tests (#234930)\n\n**Resolves: https://github.com/elastic/kibana/issues/232644**\n\nThis PR splits long-running API integration and Jest configs into\nsmaller configs to reduce overall runtime.\nWe can’t rely on Buildkite’s `parallelism` parameter here, since it’s\nonly works for Cypress steps.\n\n## Changes\n- **API Integration tests**\n - Split the prebuilt rules config into 3 smaller configs.\n\n- **Rule Upgrade Field Tests (Jest)**\n- Merged `common_fields` and `type_specific_fields` dirs into a single\n`fields` directory.\n - Replaced 2 configs with 4 smaller ones.\n- Implemented round-robin logic to evenly distribute tests across the 4\nconfigs.\n- **Alternative**: manually distribute tests into numbered directories.\n - *Pros*: simpler setup. \n- *Cons*: requires deciding where each new test should live when adding\nnew fields.\n- Reviewers, tell me if adding this distribution logic is an overkill in\nyour opinion.","sha":"928d4b224ffaadc6f171482f6b95e14e76da912d"}},{"branch":"8.19","label":"v8.19.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
nikitaindik
added a commit
that referenced
this pull request
Sep 18, 2025
…34930) (#235567) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] Parallelize slow Rule Management tests (#234930)](#234930) <!--- Backport version: 10.0.2 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Nikita Indik","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-09-18T13:27:31Z","message":"[Security Solution] Parallelize slow Rule Management tests (#234930)\n\n**Resolves: https://github.com/elastic/kibana/issues/232644**\n\nThis PR splits long-running API integration and Jest configs into\nsmaller configs to reduce overall runtime.\nWe can’t rely on Buildkite’s `parallelism` parameter here, since it’s\nonly works for Cypress steps.\n\n## Changes\n- **API Integration tests**\n - Split the prebuilt rules config into 3 smaller configs.\n\n- **Rule Upgrade Field Tests (Jest)**\n- Merged `common_fields` and `type_specific_fields` dirs into a single\n`fields` directory.\n - Replaced 2 configs with 4 smaller ones.\n- Implemented round-robin logic to evenly distribute tests across the 4\nconfigs.\n- **Alternative**: manually distribute tests into numbered directories.\n - *Pros*: simpler setup. \n- *Cons*: requires deciding where each new test should live when adding\nnew fields.\n- Reviewers, tell me if adding this distribution logic is an overkill in\nyour opinion.","sha":"928d4b224ffaadc6f171482f6b95e14e76da912d","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","backport:version","v9.2.0","v8.19.5","v9.1.5"],"title":"[Security Solution] Parallelize slow Rule Management tests","number":234930,"url":"https://github.com/elastic/kibana/pull/234930","mergeCommit":{"message":"[Security Solution] Parallelize slow Rule Management tests (#234930)\n\n**Resolves: https://github.com/elastic/kibana/issues/232644**\n\nThis PR splits long-running API integration and Jest configs into\nsmaller configs to reduce overall runtime.\nWe can’t rely on Buildkite’s `parallelism` parameter here, since it’s\nonly works for Cypress steps.\n\n## Changes\n- **API Integration tests**\n - Split the prebuilt rules config into 3 smaller configs.\n\n- **Rule Upgrade Field Tests (Jest)**\n- Merged `common_fields` and `type_specific_fields` dirs into a single\n`fields` directory.\n - Replaced 2 configs with 4 smaller ones.\n- Implemented round-robin logic to evenly distribute tests across the 4\nconfigs.\n- **Alternative**: manually distribute tests into numbered directories.\n - *Pros*: simpler setup. \n- *Cons*: requires deciding where each new test should live when adding\nnew fields.\n- Reviewers, tell me if adding this distribution logic is an overkill in\nyour opinion.","sha":"928d4b224ffaadc6f171482f6b95e14e76da912d"}},"sourceBranch":"main","suggestedTargetBranches":["8.19","9.1"],"targetPullRequestStates":[{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/234930","number":234930,"mergeCommit":{"message":"[Security Solution] Parallelize slow Rule Management tests (#234930)\n\n**Resolves: https://github.com/elastic/kibana/issues/232644**\n\nThis PR splits long-running API integration and Jest configs into\nsmaller configs to reduce overall runtime.\nWe can’t rely on Buildkite’s `parallelism` parameter here, since it’s\nonly works for Cypress steps.\n\n## Changes\n- **API Integration tests**\n - Split the prebuilt rules config into 3 smaller configs.\n\n- **Rule Upgrade Field Tests (Jest)**\n- Merged `common_fields` and `type_specific_fields` dirs into a single\n`fields` directory.\n - Replaced 2 configs with 4 smaller ones.\n- Implemented round-robin logic to evenly distribute tests across the 4\nconfigs.\n- **Alternative**: manually distribute tests into numbered directories.\n - *Pros*: simpler setup. \n- *Cons*: requires deciding where each new test should live when adding\nnew fields.\n- Reviewers, tell me if adding this distribution logic is an overkill in\nyour opinion.","sha":"928d4b224ffaadc6f171482f6b95e14e76da912d"}},{"branch":"8.19","label":"v8.19.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT-->
nikitaindik
added a commit
that referenced
this pull request
Sep 19, 2025
…tests (#235729) Pipeline with the fix passing in MKI: https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/3397 ## Summary This PR is a follow up to PR #234930 and fixes `ERROR Unable to find config file`. This error occurs when running Rule Management API integration tests in MKI pipelines ([example](https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/3393#01996093-11c2-4107-84e3-7f0b61565cc7)). ## What causes the error PR #234930 split one of the FTR config files into 3 smaller files, but didn't split the corresponding npm script in [package.json](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/test/security_solution_api_integration/package.json). ## Changes - Added npm scripts to run new test configs in periodic and quality gate pipelines - Updated pipeline YMLs to use these new scripts - Removed unused npm scripts to run configs locally. We normally test locally by running `./x-pack/scripts/functional_tests_server --config=<something>`.
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Sep 24, 2025
…34930) **Resolves: elastic#232644 This PR splits long-running API integration and Jest configs into smaller configs to reduce overall runtime. We can’t rely on Buildkite’s `parallelism` parameter here, since it’s only works for Cypress steps. ## Changes - **API Integration tests** - Split the prebuilt rules config into 3 smaller configs. - **Rule Upgrade Field Tests (Jest)** - Merged `common_fields` and `type_specific_fields` dirs into a single `fields` directory. - Replaced 2 configs with 4 smaller ones. - Implemented round-robin logic to evenly distribute tests across the 4 configs. - **Alternative**: manually distribute tests into numbered directories. - *Pros*: simpler setup. - *Cons*: requires deciding where each new test should live when adding new fields. - Reviewers, tell me if adding this distribution logic is an overkill in your opinion.
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Sep 24, 2025
…tests (elastic#235729) Pipeline with the fix passing in MKI: https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/3397 ## Summary This PR is a follow up to PR elastic#234930 and fixes `ERROR Unable to find config file`. This error occurs when running Rule Management API integration tests in MKI pipelines ([example](https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/3393#01996093-11c2-4107-84e3-7f0b61565cc7)). ## What causes the error PR elastic#234930 split one of the FTR config files into 3 smaller files, but didn't split the corresponding npm script in [package.json](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/test/security_solution_api_integration/package.json). ## Changes - Added npm scripts to run new test configs in periodic and quality gate pipelines - Updated pipeline YMLs to use these new scripts - Removed unused npm scripts to run configs locally. We normally test locally by running `./x-pack/scripts/functional_tests_server --config=<something>`.
niros1
pushed a commit
that referenced
this pull request
Sep 30, 2025
**Resolves: #232644 This PR splits long-running API integration and Jest configs into smaller configs to reduce overall runtime. We can’t rely on Buildkite’s `parallelism` parameter here, since it’s only works for Cypress steps. ## Changes - **API Integration tests** - Split the prebuilt rules config into 3 smaller configs. - **Rule Upgrade Field Tests (Jest)** - Merged `common_fields` and `type_specific_fields` dirs into a single `fields` directory. - Replaced 2 configs with 4 smaller ones. - Implemented round-robin logic to evenly distribute tests across the 4 configs. - **Alternative**: manually distribute tests into numbered directories. - *Pros*: simpler setup. - *Cons*: requires deciding where each new test should live when adding new fields. - Reviewers, tell me if adding this distribution logic is an overkill in your opinion.
niros1
pushed a commit
that referenced
this pull request
Sep 30, 2025
…tests (#235729) Pipeline with the fix passing in MKI: https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/3397 ## Summary This PR is a follow up to PR #234930 and fixes `ERROR Unable to find config file`. This error occurs when running Rule Management API integration tests in MKI pipelines ([example](https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/3393#01996093-11c2-4107-84e3-7f0b61565cc7)). ## What causes the error PR #234930 split one of the FTR config files into 3 smaller files, but didn't split the corresponding npm script in [package.json](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/test/security_solution_api_integration/package.json). ## Changes - Added npm scripts to run new test configs in periodic and quality gate pipelines - Updated pipeline YMLs to use these new scripts - Removed unused npm scripts to run configs locally. We normally test locally by running `./x-pack/scripts/functional_tests_server --config=<something>`.
rylnd
pushed a commit
to rylnd/kibana
that referenced
this pull request
Oct 17, 2025
…34930) **Resolves: elastic#232644 This PR splits long-running API integration and Jest configs into smaller configs to reduce overall runtime. We can’t rely on Buildkite’s `parallelism` parameter here, since it’s only works for Cypress steps. ## Changes - **API Integration tests** - Split the prebuilt rules config into 3 smaller configs. - **Rule Upgrade Field Tests (Jest)** - Merged `common_fields` and `type_specific_fields` dirs into a single `fields` directory. - Replaced 2 configs with 4 smaller ones. - Implemented round-robin logic to evenly distribute tests across the 4 configs. - **Alternative**: manually distribute tests into numbered directories. - *Pros*: simpler setup. - *Cons*: requires deciding where each new test should live when adding new fields. - Reviewers, tell me if adding this distribution logic is an overkill in your opinion.
rylnd
pushed a commit
to rylnd/kibana
that referenced
this pull request
Oct 17, 2025
…tests (elastic#235729) Pipeline with the fix passing in MKI: https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/3397 ## Summary This PR is a follow up to PR elastic#234930 and fixes `ERROR Unable to find config file`. This error occurs when running Rule Management API integration tests in MKI pipelines ([example](https://buildkite.com/elastic/kibana-serverless-security-solution-quality-gate-rule-management/builds/3393#01996093-11c2-4107-84e3-7f0b61565cc7)). ## What causes the error PR elastic#234930 split one of the FTR config files into 3 smaller files, but didn't split the corresponding npm script in [package.json](https://github.com/elastic/kibana/blob/main/x-pack/solutions/security/test/security_solution_api_integration/package.json). ## Changes - Added npm scripts to run new test configs in periodic and quality gate pipelines - Updated pipeline YMLs to use these new scripts - Removed unused npm scripts to run configs locally. We normally test locally by running `./x-pack/scripts/functional_tests_server --config=<something>`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves: #232644
This PR splits long-running API integration and Jest configs into smaller configs to reduce overall runtime.
We can’t rely on Buildkite’s
parallelismparameter here, since it’s only works for Cypress steps.Changes
API Integration tests
Rule Upgrade Field Tests (Jest)
diffable_rule_fields:common_fields->commontype_specific_fields->type_specific