[9.2] [Security Solution] Adds customized_fields and has_base_version fields to rule_source object schema (#234793)
#237537
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ields to `rule_source` object schema (elastic#234793) **Resolves: elastic/security-team#12507 (internal) ## Summary Adds two new fields to the existing `rule_source` object in our rule schema as described in elastic#230856. Also updates and adds test coverage for the new field logic. The new fields are: - `customized_fields`: an array of objects containing rule field names that have been modified from the base version of the prebuilt rule. - Defaults to empty array if prebuilt rule is not customized or if base version did not exist during diff calculation. - `has_base_version`: a boolean field that specifies if the base version of a prebuilt rule was able to be fetched and used during the customization calculation. This PR also adds related telemetry fields as described in elastic#230856. This includes a `customizations` object field which contains a slimmed down version of `customized_fields` and has a `num_functional_fields` number field that is created in the telemetry task pipeline by comparing the customized fields array to a constant list of field names that we are defining as "functional". This source of truth list can be found in the `x-pack/solutions/security/plugins/security_solution/common/detection_engine/constants.ts` file ### Examples ```json { "rule_source": { "type": "external", "is_customized": true, /* New fields */ "customized_fields": [ { "field_name": "tags", }, { "field_name": "query", } ], "has_base_version": true } } ``` ```json "customizations": { "customized_fields": ["tags", "query"], "num_functional_fields": 2, } ``` ## How to test telemetry Link to internal staging with example data: ([internal staging](https://analytics-staging.sde.elastic.dev/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:60000),time:(from:'2025-09-26T15:59:24.512Z',to:'2025-09-26T16:08:58.435Z'))&_a=(columns:!(),dataSource:(dataViewId:'4ca97040-d095-11ec-95a5-011050c1180f',type:dataView),filters:!(),interval:auto,query:(language:kuery,query:'customizations.num_functional_fields%20%3E%200'),sort:!(!('@timestamp',desc)),viewMode:documents))) 1. Set the prebuilt rule task type to something shorter than `1hr` in this file: `x-pack/solutions/security/plugins/security_solution/server/lib/telemetry/tasks/prebuilt_rule_alerts.ts` 2. Add the following to `kibana.dev.yml`: ``` telemetry.enabled: true telemetry.optIn: true // (Optional for checking to see if its working) logging: root: appenders: [default] level: warn loggers: - name: plugins.securitySolution level: debug - name: plugins.ruleRegistry - name: plugins.taskManager ``` 3. Start up both Elasticsearch and kibana (Has to be done _after_ updating task interval as task objects are stored in ES) 4. Install prebuilt rules 5. Modify prebuilt rules with different field customizations and enable those rules 6. Generate alerts that match these rules (resolver script generator, dev tools, query modification, etc.) 7. View the alerts getting sent to the internal staging telemetry cluster (https://analytics-staging.sde.elastic.dev) in the `detections_alert_telemetry_elastic*` index 8. Use the new `customizations` field to filter out/in customized rule alerts ## Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [x] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [x] [Rule customization tests](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/9317) --------- Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Georgii Gorbachev <[email protected]> (cherry picked from commit aeb873a)
4 tasks
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]Async chunks
cc @dplumlee |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Backport
This will backport the following commits from
mainto9.2:customized_fieldsandhas_base_versionfields torule_sourceobject schema (#234793)Questions ?
Please refer to the Backport tool documentation