[streams][significant events] add severity_score #244764

klacabane · 2025-12-01T13:20:11Z

Summary

Add a severity_score to the queries generated by AI

elasticmachine · 2025-12-01T13:24:32Z

Pinging @elastic/streams-program-team (Team:streams-program)

ruflin · 2025-12-01T13:38:31Z

x-pack/platform/packages/shared/kbn-streams-ai/src/significant_events/system_prompt.text

 ### Example 1: Calling `add_queries` with a single query

-`>>> ACTION: add_queries(queries=[{"title": "View all errors", "kql": "error.message:*", "category": "error"}])`
+`>>> ACTION: add_queries(queries=[{"title": "View all errors", "kql": "error.message:*", "category": "error", "severity_score": 60}])`


Should we go from 0.0 - 1.0 or from 0 to 100?

we discussed offline and will go with 0-100 to align with anomaly detection scoring

++, lets also document this somewhere in the code.

…atus --include-path /api/alerting/rule/ --include-path /api/alerting/rules --include-path /api/actions --include-path /api/security/role --include-path /api/spaces --include-path /api/streams --include-path /api/fleet --include-path /api/saved_objects/_import --include-path /api/saved_objects/_export --include-path /api/maintenance_window --include-path /api/agent_builder --update

ruflin · 2025-12-01T13:40:09Z

x-pack/platform/packages/shared/kbn-streams-ai/src/significant_events/system_prompt.text

+| `operational` | 30 | -10 expected lifecycle events |
+| `configuration` | 25 | +10 security-related changes |
+
+**Score ranges:** 80-100 critical, 60-79 high, 40-59 medium, 0-39 low


@shahzad31 If we can overload a prompt, means we can overload this scoring in case we need / want to adjust it.

ruflin · 2025-12-01T13:40:59Z

x-pack/platform/plugins/shared/streams/server/lib/streams/assets/asset_client.ts

  QueryLink,
 } from '../../../../common/assets';
-import { QUERY_KQL_BODY, QUERY_FEATURE_FILTER, QUERY_FEATURE_NAME, QUERY_TITLE } from './fields';
+import {


@cesco-f I remember you had a PR to refactor this? Just stumbled over it as it is kind of odd to that we modify the asset client for sig events. But not urgent.

ruflin · 2025-12-01T13:46:27Z

What happens to existing sig events queries that don't have the score yet? Is it just 0?

klacabane · 2025-12-01T15:06:02Z

What happens to existing sig events queries that don't have the score yet? Is it just 0?

@ruflin the score is currently optional so it won't be defined for existing queries as it sounds difficult to provide a correct backfill default, but if we want to I'd be more inclined to set something in the 50-70 range than 0

ruflin · 2025-12-01T15:09:24Z

Having the score optional is great. Lets keep it that way.

Copilot

Pull request overview

This PR adds a severity_score field to AI-generated queries in the streams significant events system. The severity score is a numeric value (0-100) that indicates the criticality of events matched by each query.

Key changes:

Added severity_score as an optional numeric field to query schemas and storage
Updated AI prompt system to include severity scoring guidelines with category-based baselines
Modified data flow to propagate severity scores through the generation and storage pipeline

Reviewed changes

Copilot reviewed 14 out of 14 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
`x-pack/platform/plugins/shared/streams/server/lib/streams/assets/fields.ts`	Defines new `QUERY_SEVERITY_SCORE` field constant
`x-pack/platform/plugins/shared/streams/server/lib/streams/assets/storage_settings.ts`	Adds severity_score as a long type field in storage schema
`x-pack/platform/plugins/shared/streams/server/lib/streams/assets/asset_client.ts`	Updates storage/retrieval logic to handle severity_score in query links
`x-pack/platform/plugins/shared/streams/server/lib/significant_events/read_significant_events_from_alerts_indices.ts`	Propagates severity_score to significant events returned from alerts
`x-pack/platform/plugins/shared/streams/server/lib/significant_events/generate_significant_events.ts`	Includes severity_score when generating event definitions
`x-pack/platform/packages/shared/kbn-streams-schema/src/queries/index.ts`	Adds optional severity_score field to query schema and validation
`x-pack/platform/packages/shared/kbn-streams-schema/src/api/significant_events/index.ts`	Makes severity_score required in generated event query interface
`x-pack/platform/packages/shared/kbn-streams-ai/src/significant_events/system_prompt.text`	Documents severity scoring system with category baselines and modifiers
`x-pack/platform/packages/shared/kbn-streams-ai/src/significant_events/prompt.ts`	Adds severity_score as required numeric field (0-100) in AI tool schema
`x-pack/platform/packages/shared/kbn-streams-ai/src/significant_events/generate_significant_events.ts`	Includes severity_score in Query interface
`oas_docs/output/kibana.yaml`	Updates OpenAPI spec with severity_score field
`oas_docs/output/kibana.serverless.yaml`	Updates serverless OpenAPI spec with severity_score field
`oas_docs/bundle.serverless.json`	Updates serverless API bundle with severity_score
`oas_docs/bundle.json`	Updates API bundle with severity_score

Copilot · 2025-12-01T15:15:03Z

x-pack/platform/packages/shared/kbn-streams-schema/src/api/significant_events/index.ts

    filter: Condition;
    type: string;
  };
+  severity_score: number;


The severity_score field is required in GeneratedSignificantEventQuery but optional in the schema definitions (streamQueryKqlSchema and upsertStreamQueryRequestSchema). This inconsistency could cause runtime errors when queries without severity_score are processed. Consider making this field optional here as well, or ensure it's always provided during generation.

Suggested change

severity_score: number;

severity_score?: number;

Copilot · 2025-12-01T15:15:04Z

x-pack/platform/packages/shared/kbn-streams-ai/src/significant_events/prompt.ts

+                  },
                },
-                required: ['kql', 'title', 'category'],
+                required: ['kql', 'title', 'category', 'severity_score'],


The AI prompt marks severity_score as required in the tool schema, but the data schema in kbn-streams-schema defines it as optional. This mismatch could cause issues if non-AI code paths create queries without severity scores. Align the requirement status across the codebase or add validation to handle missing values.

Suggested change

required: ['kql', 'title', 'category', 'severity_score'],

required: ['kql', 'title', 'category'],

This is on purpose. existing queries do not have a score and we chose not to backfill that value

miltonhultgren

LGTM, tested locally and see the API respond as expected, UI not updated yet.

Also, more insight will likely come as we start looking at the evaluations for these prompts!

elasticmachine · 2025-12-01T16:28:48Z

💚 Build Succeeded

Buildkite Build
Commit: b5bfa98

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id	before	after	diff
`@kbn/streams-schema`	178	180	+2

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`streamsApp`	1.0MB	1.0MB	+66.0B

Unknown metric groups

API count

id	before	after	diff
`@kbn/streams-schema`	178	180	+2

## Summary Add a `severity_score` to the queries generated by AI --------- Co-authored-by: kibanamachine <[email protected]>

add severity_score

90cb0ea

klacabane added release_note:skip Skip the PR/issue when compiling release notes backport:skip This PR does not require backporting Team:streams-program Team Label for Streams program v9.3.0 Feature:SignificantEvents labels Dec 1, 2025

Merge branch 'main' into sig-events-scoring_1

d42914c

klacabane marked this pull request as ready for review December 1, 2025 13:24

klacabane requested a review from a team as a code owner December 1, 2025 13:24

ruflin reviewed Dec 1, 2025

View reviewed changes

Changes from make api-docs

b5bfa98

elastic-vault-github-plugin-prod bot requested a review from a team as a code owner December 1, 2025 14:00

ruflin requested a review from Copilot December 1, 2025 15:14

Copilot AI reviewed Dec 1, 2025

View reviewed changes

miltonhultgren approved these changes Dec 1, 2025

View reviewed changes

klacabane added 2 commits December 1, 2025 19:17

comment score range

c74e433

Merge branch 'main' into sig-events-scoring_1

5904545

mdbirnstiehl approved these changes Dec 1, 2025

View reviewed changes

klacabane merged commit 5723088 into elastic:main Dec 2, 2025
12 checks passed

	required: ['kql', 'title', 'category', 'severity_score'],
	required: ['kql', 'title', 'category'],

[streams][significant events] add severity_score #244764

[streams][significant events] add severity_score #244764

Uh oh!

Conversation

klacabane commented Dec 1, 2025

Summary

Uh oh!

elasticmachine commented Dec 1, 2025

Uh oh!

ruflin Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

klacabane Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

ruflin Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

ruflin Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

ruflin Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

ruflin commented Dec 1, 2025

Uh oh!

klacabane commented Dec 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ruflin commented Dec 1, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

klacabane Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

miltonhultgren left a comment

Choose a reason for hiding this comment

Uh oh!

elasticmachine commented Dec 1, 2025

💚 Build Succeeded

Metrics [docs]

Public APIs missing comments

Async chunks

API count

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

klacabane commented Dec 1, 2025 •

edited

Loading