Skip to content

[streams][significant events] add severity_score #244764

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Dec 2, 2025
Merged

[streams][significant events] add severity_score #244764

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
90cb0ea
add severity_score
klacabane Dec 1, 2025
d42914c
Merge branch 'main' into sig-events-scoring_1
klacabane Dec 1, 2025
7d4df47
Changes from node scripts/capture_oas_snapshot --include-path /api/…
kibanamachine Dec 1, 2025
b5bfa98
Changes from make api-docs
kibanamachine Dec 1, 2025
c74e433
comment score range
klacabane Dec 1, 2025
5904545
Merge branch 'main' into sig-events-scoring_1
klacabane Dec 1, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions oas_docs/bundle.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53918,6 +53918,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -58955,6 +58958,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -63545,6 +63551,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -68278,6 +68287,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -72870,6 +72882,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -82988,6 +83003,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -83447,6 +83465,9 @@
],
"type": "object"
},
"severity_score": {
"type": "number"
},
"title": {
"minLength": 1,
"type": "string"
Expand Down
21 changes: 21 additions & 0 deletions oas_docs/bundle.serverless.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52997,6 +52997,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -58034,6 +58037,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -62624,6 +62630,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -67357,6 +67366,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -71949,6 +71961,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -82067,6 +82082,9 @@
"query"
],
"type": "object"
},
"severity_score": {
"type": "number"
}
},
"required": [
Expand Down Expand Up @@ -82526,6 +82544,9 @@
],
"type": "object"
},
"severity_score": {
"type": "number"
},
"title": {
"minLength": 1,
"type": "string"
Expand Down
14 changes: 14 additions & 0 deletions oas_docs/output/kibana.serverless.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56049,6 +56049,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -58644,6 +58646,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -61013,6 +61017,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -63458,6 +63464,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -65827,6 +65835,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -71192,6 +71202,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
required:
Expand Down Expand Up @@ -71452,6 +71464,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
title:
minLength: 1
type: string
Expand Down
14 changes: 14 additions & 0 deletions oas_docs/output/kibana.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60423,6 +60423,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -63018,6 +63020,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -65387,6 +65391,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -67832,6 +67838,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -70201,6 +70209,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
type: array
Expand Down Expand Up @@ -75566,6 +75576,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
required:
- kql
required:
Expand Down Expand Up @@ -75826,6 +75838,8 @@ paths:
type: string
required:
- query
severity_score:
type: number
title:
minLength: 1
type: string
Expand Down
2 changes: 2 additions & 0 deletions ...form/packages/shared/kbn-streams-ai/src/significant_events/generate_significant_events.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,9 @@ interface Query {
kql: string;
title: string;
category: SignificantEventType;
severity_score: number;
}

/**
* Generate significant event definitions, based on:
* - the description of the feature (or stream if feature is undefined)
Expand Down
7 changes: 6 additions & 1 deletion x-pack/platform/packages/shared/kbn-streams-ai/src/significant_events/prompt.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,13 @@ export const GenerateSignificantEventsPrompt = createPrompt({
SIGNIFICANT_EVENT_TYPE_SECURITY,
],
},
severity_score: {
type: 'number',
minimum: 0,
maximum: 100,
},
},
required: ['kql', 'title', 'category'],
required: ['kql', 'title', 'category', 'severity_score'],
Copy link

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The AI prompt marks severity_score as required in the tool schema, but the data schema in kbn-streams-schema defines it as optional. This mismatch could cause issues if non-AI code paths create queries without severity scores. Align the requirement status across the codebase or add validation to handle missing values.

Suggested change
required: ['kql', 'title', 'category', 'severity_score'],
required: ['kql', 'title', 'category'],

Copilot uses AI. Check for mistakes.
Copy link
Contributor Author

@klacabane klacabane Dec 2, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is on purpose. existing queries do not have a score and we chose not to backfill that value

},
},
},
Expand Down
34 changes: 25 additions & 9 deletions x-pack/platform/packages/shared/kbn-streams-ai/src/significant_events/system_prompt.text
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Your primary goal is to analyze the provided context about a user's system—inc

| Tool | Function | Notes |
| :--- | :--- | :--- |
| `add_queries` | Submits one or more KQL queries for the user. | Payload is a list of objects, each with `title`, `kql`, and `category`. |
| `add_queries` | Submits one or more KQL queries for the user. | Payload is a list of objects, each with `title`, `kql`, `category`, and `severity_score`. |
| `reason()` | **Begin a Reasoning Monologue** | Outputs your private thoughts. Must use sentinel tags (`<<<BEGIN_INTERNAL>>>`...`<<<END_INTERNAL>>>`). |
| `complete()` | Declare readiness to answer | Ends the loop and triggers the **Definitive Output**. |

Expand Down Expand Up @@ -54,15 +54,15 @@ PLAN> Describe your next action in natural language. If you are ready to answer,

### Example 1: Calling `add_queries` with a single query

`>>> ACTION: add_queries(queries=[{"title": "View all errors", "kql": "error.message:*", "category": "error"}])`
`>>> ACTION: add_queries(queries=[{"title": "View all errors", "kql": "error.message:*", "category": "error", "severity_score": 60}])`
Copy link
Contributor

@ruflin ruflin Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we go from 0.0 - 1.0 or from 0 to 100?

Copy link
Contributor Author

@klacabane klacabane Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we discussed offline and will go with 0-100 to align with anomaly detection scoring

Copy link
Contributor

@ruflin ruflin Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

++, lets also document this somewhere in the code.


### Example 2: Calling `add_queries` with multiple queries

```
>>> ACTION: add_queries(queries=[
{"title": "Application startup", "kql": "message:\"Started Application\"", "category": "operational"},
{"title": "Failed login attempts", "kql": "body.text:\"Failed password for\"", "category": "security"},
{"title": "Out of memory errors", "kql": "body.text:\"java.lang.OutOfMemoryError\"", "category": "error"}
{"title": "Application startup", "kql": "message:\"Started Application\"", "category": "operational", "severity_score": 25},
{"title": "Failed login attempts", "kql": "body.text:\"Failed password for\"", "category": "security", "severity_score": 75},
{"title": "Out of memory errors", "kql": "body.text:\"java.lang.OutOfMemoryError\"", "category": "error", "severity_score": 85}
])
```

Expand Down Expand Up @@ -113,10 +113,10 @@ This is the final, user-facing response that follows the `complete()` call from
**Scenario:** You attempt to add a query with invalid KQL syntax.

**Initial Flawed Action:**
`>>> ACTION: add_queries(queries=[{"title": "Invalid KQL syntax", "kql": "body.text:value AND", "category": "error"}])`
`>>> ACTION: add_queries(queries=[{"title": "Invalid KQL syntax", "kql": "body.text:value AND", "category": "error", "severity_score": 60}])`

**Tool Response (simulated):**
`Failed to add 1 of 1 queries. Invalid: [{"title": "Invalid KQL syntax", "kql": "body.text:value AND", "category": "error", "error": "KQL syntax error: trailing boolean operator"}]`
`Failed to add 1 of 1 queries. Invalid: [{"title": "Invalid KQL syntax", "kql": "body.text:value AND", "category": "error", "severity_score": 60, "error": "KQL syntax error: trailing boolean operator"}]`

**Reasoning Monologue for Repair:**
`<<<BEGIN_INTERNAL>>>`
Expand All @@ -127,7 +127,7 @@ This is the final, user-facing response that follows the `complete()` call from
`<<<END_INTERNAL>>>`

**Corrected Action:**
`>>> ACTION: add_queries(queries=[{"title": "Corrected KQL", "kql": "body.text:value", "category": "error"}])`
`>>> ACTION: add_queries(queries=[{"title": "Corrected KQL", "kql": "body.text:value", "category": "error", "severity_score": 60}])`

---

Expand Down Expand Up @@ -155,7 +155,23 @@ This is the final, user-facing response that follows the `complete()` call from

---

## 8. Tips & hints
## 8. Severity Scoring

Assign a `severity_score` (0-100) based on category baseline + modifiers:

| Category | Base | Modifiers |
|----------|------|-----------|
| `security` | 70 | +15 privilege escalation, +10 repeated failures |
| `error` | 60 | +25 crash/OOM/deadlock, +10 data integrity risk |
| `resource_health` | 50 | +15 exhaustion, +10 degradation warnings |
| `operational` | 30 | -10 expected lifecycle events |
| `configuration` | 25 | +10 security-related changes |

**Score ranges:** 80-100 critical, 60-79 high, 40-59 medium, 0-39 low
Copy link
Contributor

@ruflin ruflin Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shahzad31 If we can overload a prompt, means we can overload this scoring in case we need / want to adjust it.


---

## 9. Tips & hints

* **Focus on Actionable Insights:** Generate queries that a user would find genuinely helpful for debugging, monitoring, or security. Avoid trivial queries (e.g., `message:*`).
* **Categorize Correctly:** Use the provided categories (`operational`, `configuration`, `resource_health`, `error`, `security`). If a query fits multiple, choose the most specific one.
Expand Down
1 change: 1 addition & 0 deletions x-pack/platform/packages/shared/kbn-streams-schema/src/api/significant_events/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ interface GeneratedSignificantEventQuery {
filter: Condition;
type: string;
};
severity_score: number;
Copy link

Copilot AI Dec 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The severity_score field is required in GeneratedSignificantEventQuery but optional in the schema definitions (streamQueryKqlSchema and upsertStreamQueryRequestSchema). This inconsistency could cause runtime errors when queries without severity_score are processed. Consider making this field optional here as well, or ensure it's always provided during generation.

Suggested change
severity_score: number;
severity_score?: number;

Copilot uses AI. Check for mistakes.
}

type SignificantEventsGenerateResponse = Observable<
Expand Down
3 changes: 3 additions & 0 deletions x-pack/platform/packages/shared/kbn-streams-schema/src/queries/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ export interface StreamQueryKql extends StreamQueryBase {
kql: {
query: string;
};
severity_score?: number;
}

export type StreamQuery = StreamQueryKql;
Expand All @@ -47,6 +48,7 @@ export const streamQueryKqlSchema: z.Schema<StreamQueryKql> = z.intersection(
kql: z.object({
query: z.string(),
}),
severity_score: z.number().optional(),
})
);

Expand All @@ -67,6 +69,7 @@ export const upsertStreamQueryRequestSchema = z.object({
kql: z.object({
query: z.string(),
}),
severity_score: z.number().optional(),
});

export const isStreamQueryKql = createIsNarrowSchema(streamQuerySchema, streamQueryKqlSchema);
Loading