[Security Solution] Get rid of esArchiver in import custom rules integration tests #229492
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maximpn
added
impact:high
maximpn
force-pushed
the
refactor-import-rules-integration-tests
branch
2 times, most recently
from
e3a46db to
3e2f234
Compare
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8884[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_import_export/trial_license_complete_tier/configs/serverless.config.ts: 200/200 tests passed. |
Contributor
Flaky Test Runner Stats🎉 All tests passed! - kibana-flaky-test-suite-runner#8883[✅] x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_import_export/trial_license_complete_tier/configs/ess.config.ts: 200/200 tests passed. |
Contributor
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
Contributor
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
Contributor
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
dmlemeshko
approved these changes
Jul 28, 2025
Member
dmlemeshko
left a comment
dmlemeshko
left a comment
There was a problem hiding this comment.
LGTM, please check my comment regarding directory deprecation (ETA: end of August)
| // @ts-ignore not ts yet | ||
| import { EsSupertestWithoutAuthProvider } from './es_supertest_without_auth'; | ||
| import { SecuritySolutionApiProvider } from './security_solution_api.gen'; | ||
| import { SecuritySolutionApiProvider as SecuritySolutionExceptionsApiProvider } from './security_solution_exceptions_api.gen'; |
Member
There was a problem hiding this comment.
we are about to deprecate x-pack/test and I already moved most of the services out of x-pack/test/api_integration/services path.
Maybe you can create a follow-up PR and move all the Security Solution FTR services and utils (x-pack/test/common/utils/security_solution) directly into x-pack/test/security_solution_api_integration dir, this way moving the whole directory to x-pack/solutions/security/test will be less challenging with imports update?
maximpn
added
the
release_note:skip
73 tasks
maximpn
added a commit
that referenced
this pull request
Aug 4, 2025
…es integration tests (#229492) (#230287) # Backport This will backport the following commits from `main` to `8.18`: - [[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)](#229492) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-07-31T15:16:08Z","message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","impact:high","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Rule Import/Export","backport:version","v8.18.0","v9.1.0","v8.19.0","v9.2.0"],"title":"[Security Solution] Get rid of esArchiver in import custom rules integration tests","number":229492,"url":"https://github.com/elastic/kibana/pull/229492","mergeCommit":{"message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c"}},"sourceBranch":"main","suggestedTargetBranches":["8.18","8.19"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/230282","number":230282,"state":"OPEN"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/230279","number":230279,"state":"OPEN"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/229492","number":229492,"mergeCommit":{"message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c"}}]}] BACKPORT-->
maximpn
added a commit
that referenced
this pull request
Aug 4, 2025
…s integration tests (#229492) (#230282) # Backport This will backport the following commits from `main` to `9.0`: - [[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)](#229492) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-07-31T15:16:08Z","message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","impact:high","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Rule Import/Export","backport:version","v8.18.0","v9.1.0","v8.19.0","v9.2.0"],"title":"[Security Solution] Get rid of esArchiver in import custom rules integration tests","number":229492,"url":"https://github.com/elastic/kibana/pull/229492","mergeCommit":{"message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","9.1","8.19"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/229492","number":229492,"mergeCommit":{"message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c"}}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <[email protected]>
This was referenced Aug 5, 2025
maximpn
added a commit
that referenced
this pull request
Aug 5, 2025
…s integration tests (#229492) (#230279) # Backport This will backport the following commits from `main` to `9.1`: - [[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)](#229492) <!--- Backport version: 10.0.1 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-07-31T15:16:08Z","message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","impact:high","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Rule Import/Export","backport:version","v8.18.0","v9.1.0","v8.19.0","v9.2.0"],"title":"[Security Solution] Get rid of esArchiver in import custom rules integration tests","number":229492,"url":"https://github.com/elastic/kibana/pull/229492","mergeCommit":{"message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c"}},"sourceBranch":"main","suggestedTargetBranches":["9.0","8.18","9.1","8.19"],"targetPullRequestStates":[{"branch":"9.0","label":"v9.0.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.18","label":"v8.18.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/229492","number":229492,"mergeCommit":{"message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c"}}]}] BACKPORT--> --------- Co-authored-by: Nikita Indik <[email protected]>
kibanamachine
added
v9.1.1
and removed
backport missing
delanni
pushed a commit
to delanni/kibana
that referenced
this pull request
Aug 5, 2025
…gration tests (elastic#229492) **Resolves: elastic#227454 **Resolves: elastic#224511 **Resolves: elastic#220971 **Resolves: elastic#220536 **Resolves: elastic#217903 **Resolves: elastic#205777 ## Summary This PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`. ## Details Investigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness. On top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections. ## Flaky test runner - ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883) - ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884) ## How to test the fix? - Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless. - run import custom rules integration tests in the `main` branch and observe test failures - run import custom rules integration tests in this PR and observe all tests have passed
Contributor
|
Hey @maximpn, do we have a backport to 8.19? I could only find these in the ticket's timeline:
|
Contributor
Author
|
@banderror we don't have |
Contributor
|
@maximpn @dmlemeshko Hey guys, any updates? We need to backport it to 8.19! |
Contributor
Author
Hey @banderror, I helped @dmlemeshko with addressing multiple issues in the manually created backport. In particular I reapplied my PR merged into main to auto-generated API integration test clients. At the moment there are still some tricky typing issues blocking the CI. @dmlemeshko plans to find a solution. |
This was referenced Sep 5, 2025
maximpn
added a commit
to maximpn/kibana
that referenced
this pull request
Sep 10, 2025
…gration tests (elastic#229492) **Resolves: elastic#227454 **Resolves: elastic#224511 **Resolves: elastic#220971 **Resolves: elastic#220536 **Resolves: elastic#217903 **Resolves: elastic#205777 ## Summary This PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`. ## Details Investigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness. On top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections. ## Flaky test runner - ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883) - ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884) ## How to test the fix? - Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless. - run import custom rules integration tests in the `main` branch and observe test failures - run import custom rules integration tests in this PR and observe all tests have passed (cherry picked from commit 6403fd4) # Conflicts: # src/platform/packages/shared/kbn-es-archiver/src/fixtures/override_saved_objects_index/exception_list.json # x-pack/solutions/security/plugins/cloud_defend/public/pages/index.ts # x-pack/solutions/security/test/common/utils/spaces.ts # x-pack/solutions/security/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_import_export/trial_license_complete_tier/import_rules.ts # x-pack/solutions/security/test/security_solution_api_integration/test_suites/lists_and_exception_lists/utils.ts # x-pack/solutions/security/test/security_solution_api_integration/tsconfig.json # x-pack/test/api_integration/services/index.ts # x-pack/test_serverless/shared/services/deployment_agnostic_services.ts
Contributor
Author
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
maximpn
added a commit
to maximpn/kibana
that referenced
this pull request
Sep 11, 2025
…gration tests (elastic#229492) **Resolves: elastic#227454 **Resolves: elastic#224511 **Resolves: elastic#220971 **Resolves: elastic#220536 **Resolves: elastic#217903 **Resolves: elastic#205777 This PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`. Investigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness. On top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections. - ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883) - ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884) - Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless. - run import custom rules integration tests in the `main` branch and observe test failures - run import custom rules integration tests in this PR and observe all tests have passed (cherry picked from commit 6403fd4)
maximpn
added a commit
that referenced
this pull request
Sep 11, 2025
…es integration tests (#229492) (#234561) # Backport This will backport the following commits from `main` to `8.19`: - [[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)](#229492) <!--- Backport version: 10.0.2 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-07-31T15:16:08Z","message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c","branchLabelMapping":{"^v9.2.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","impact:high","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Rule Import/Export","backport:version","v8.19.0","v9.2.0","v9.0.5","v8.18.5","v9.1.2"],"title":"[Security Solution] Get rid of esArchiver in import custom rules integration tests","number":229492,"url":"https://github.com/elastic/kibana/pull/229492","mergeCommit":{"message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c"}},"sourceBranch":"main","suggestedTargetBranches":["8.19"],"targetPullRequestStates":[{"branch":"8.19","label":"v8.19.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.2.0","branchLabelMappingKey":"^v9.2.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/229492","number":229492,"mergeCommit":{"message":"[Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492)\n\n**Resolves: https://github.com/elastic/kibana/issues/227454**\n**Resolves: https://github.com/elastic/kibana/issues/224511**\n**Resolves: https://github.com/elastic/kibana/issues/220971**\n**Resolves: https://github.com/elastic/kibana/issues/220536**\n**Resolves: https://github.com/elastic/kibana/issues/217903**\n**Resolves: https://github.com/elastic/kibana/issues/205777**\n\n## Summary\n\nThis PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of `esArchiver`.\n\n## Details\n\nInvestigation of the flakes specified in the tickets above revealed that the culprit is `esArchiver`. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.\n\nOn top of just getting rid of `esArchiver` this PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed since `esArchiver` performs connectors migration so importing rules tests interact with the migrated connections.\n\n## Flaky test runner\n\n- ✅ [ESS 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8883)\n- ✅ [Serverless 200 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/8884)\n\n## How to test the fix?\n\n- Add `--xpack.fleet.registryUrl=http://non-existent` to [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/ess/config.base.ts) for ESS or [config.base.ts](https://github.com/elastic/kibana/blob/main/x-pack/test/security_solution_api_integration/config/serverless/config.base.ts) for Serverless.\n- run import custom rules integration tests in the `main` branch and observe test failures\n- run import custom rules integration tests in this PR and observe all tests have passed","sha":"6403fd42ee060736f849af276b63183db247ea7c"}},{"branch":"9.0","label":"v9.0.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/230282","number":230282,"state":"MERGED","mergeCommit":{"sha":"c1c72bbe352a0e09417ff5b2a0edb369ce64ae7d","message":"[9.0] [Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492) (#230282)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.0`:\n- [[Security Solution] Get rid of esArchiver in import custom rules\nintegration tests\n(#229492)](https://github.com/elastic/kibana/pull/229492)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\n---------\n\nCo-authored-by: kibanamachine <[email protected]>"}},{"branch":"8.18","label":"v8.18.5","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/230287","number":230287,"state":"MERGED","mergeCommit":{"sha":"bdc2a70f4785d3220c78e8d0bc42f0f6e82e9c08","message":"[8.18] [Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492) (#230287)\n\n# Backport\n\nThis will backport the following commits from `main` to `8.18`:\n- [[Security Solution] Get rid of esArchiver in import custom rules\nintegration tests\n(#229492)](https://github.com/elastic/kibana/pull/229492)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n"}},{"branch":"9.1","label":"v9.1.2","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/230279","number":230279,"state":"MERGED","mergeCommit":{"sha":"79ac090d20ae5a69043b4ce6886dfeddb1fe5c07","message":"[9.1] [Security Solution] Get rid of esArchiver in import custom rules integration tests (#229492) (#230279)\n\n# Backport\n\nThis will backport the following commits from `main` to `9.1`:\n- [[Security Solution] Get rid of esArchiver in import custom rules\nintegration tests\n(#229492)](https://github.com/elastic/kibana/pull/229492)\n\n\n\n### Questions ?\nPlease refer to the [Backport tool\ndocumentation](https://github.com/sorenlouv/backport)\n\n\n\n---------\n\nCo-authored-by: Nikita Indik <[email protected]>"}}]}] BACKPORT-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
8 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves: #227454
Resolves: #224511
Resolves: #220971
Resolves: #220536
Resolves: #217903
Resolves: #205777
Summary
This PR fixes import custom rules flaky integration tests. This is done by refactoring the tests and getting rid of
esArchiver.Details
Investigation of the flakes specified in the tickets above revealed that the culprit is
esArchiver. It removes the target indexes before loading the documents and applies migrations. In fact it turns out it removes just installed mock prebuilt rules package forcing installing a real prebuilt rules package from the production EPR. Where usage of production EPR has been known to cause flakiness.On top of just getting rid of
esArchiverthis PR refactors the importing custom rules integrations tests. The main goal is to make the tests structure clear and transparent. Previously flaky tests were moved to the different sections. And some of the tests testing Kibana v7.14 connectors were removed sinceesArchiverperforms connectors migration so importing rules tests interact with the migrated connections.Flaky test runner
How to test the fix?
--xpack.fleet.registryUrl=http://non-existentto config.base.ts for ESS or config.base.ts for Serverless.mainbranch and observe test failures