-
Notifications
You must be signed in to change notification settings - Fork 8.5k
[Contextual Security] align graph visualization with ECS entity namespace fields for actor and target identification #243711
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
bfa5d96 to
68ea2bb
Compare
🔍 Preview links for changed docs |
e3e3d27 to
da26c57
Compare
a9ea81c to
9432c33
Compare
9432c33 to
5bb24da
Compare
9c1516f to
f81df85
Compare
|
Project deployments require a Github label, please add one or more of |
|
PR Cloud deployment started at: https://buildkite.com/elastic/kibana-deploy-cloud-from-pr/builds/602 |
|
Pinging @elastic/contextual-security-apps (Team:Cloud Security) |
- Add NON_ENRICHED_ENTITY_TYPE constant - Update entity group/type fallback to use 'Entities' - Add actorEntitySubType and targetEntitySubType to STATS aggregation - Add new label logic after STATS based on entity count Verification: - ESLint: PASS - Type check: PASS - Unit tests: PASS
Tests verify ESQL query string contains: - 'Entities' fallback for entity groups and types - Label logic after STATS based on entity count - actorEntitySubType and targetEntitySubType in STATS aggregation Verification: - ESLint: PASS - Type check: PASS - Unit tests: PASS (10 tests passing)
… items for single and grouped nodes
…n storybook tests
…and targetsDocData object fixed and added unit tests to fetchGraph and parseRecords, updated also graph api integration tests
enhanced the graph visualization API to handle multiple target
fir graph visualization unit tests
f81df85 to
931ec5b
Compare
PhilippeOberti
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM for the 4 files impacting the @elastic/security-threat-hunting-investigations team. I did not look at any of the other 25k lines added by this PR 😅 (which seems that a lot tbh)
albertoblaz
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Requesting changes because of the regression spotted eliminating COALESCE from the query
...-security-posture/graph/src/components/graph_investigation/use_entity_node_expand_popover.ts
Outdated
Show resolved
Hide resolved
...ity-posture/graph/src/components/graph_investigation/use_entity_node_expand_popover.test.tsx
Outdated
Show resolved
Hide resolved
...ity-posture/graph/src/components/graph_investigation/use_entity_node_expand_popover.test.tsx
Outdated
Show resolved
Hide resolved
...ity-posture/graph/src/components/graph_investigation/use_entity_node_expand_popover.test.tsx
Outdated
Show resolved
Hide resolved
| actorEntityName IS NOT NULL OR actorEntityType IS NOT NULL OR actorEntitySubType IS NOT NULL, | ||
| CONCAT(",\\"entity\\":", "{", | ||
| "\\"name\\":\\"", actorEntityName, "\\"", | ||
| ",\\"type\\":\\"", actorEntityType, "\\"", | ||
| ",\\"sub_type\\":\\"", actorEntitySubType, "\\"", | ||
| CASE( | ||
| actorHostIp IS NOT NULL, | ||
| CONCAT(",\\"host\\":", "{", "\\"ip\\":\\"", TO_STRING(actorHostIp), "\\"", "}"), | ||
| "" | ||
| ), | ||
| "}", | ||
| "}") | ||
| | EVAL targetDocData = CONCAT("{", | ||
| "\\"id\\":\\"", COALESCE(target.entity.id, ""), "\\"", | ||
| ",\\"type\\":\\"", "${DOCUMENT_TYPE_ENTITY}", "\\"", | ||
| ",\\"entity\\":", "{", | ||
| "\\"name\\":\\"", COALESCE(targetEntityName, ""), "\\"", | ||
| ",\\"type\\":\\"", COALESCE(targetEntityType, ""), "\\"", | ||
| ",\\"sub_type\\":\\"", COALESCE(targetEntitySubType, ""), "\\"", | ||
| CASE ( | ||
| "}"), | ||
| "" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a regression.
If we find an non-null actorEntityType and actorEntitySubType but actorEntityName is NULL, CONCAT will fail because it doesn't support NULL values so the whole JSON won't be build.
We need to restore COALESCE(<field>, "undefined") on each line instead (@kfirpeled pointed out to better fallback to "undefined" than "")
x-pack/solutions/security/plugins/cloud_security_posture/server/routes/graph/fetch_graph.ts
Show resolved
Hide resolved
| // Create minimal actor and target data with entityFieldNamespace even without enrichment | ||
| | EVAL actorDocData = CONCAT("{", | ||
| "\\"id\\":\\"", actorEntityId, "\\"", | ||
| ",\\"type\\":\\"", "${DOCUMENT_TYPE_ENTITY}", "\\"", | ||
| ",\\"entityFieldNamespace\\":\\"", actorEntityFieldHint, "\\"", | ||
| "}") | ||
| | EVAL targetDocData = CONCAT("{", | ||
| "\\"id\\":\\"", COALESCE(targetEntityId, ""), "\\"", | ||
| ",\\"type\\":\\"", "${DOCUMENT_TYPE_ENTITY}", "\\"", | ||
| ",\\"entityFieldNamespace\\":\\"", targetEntityFieldHint, "\\"", | ||
| "}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see this is duplicated in both enriched and non-enriched logic branches and could be simplified.
I would:
- Set
actorEntityFieldto the JSON object in the enriched branch - Set
actorEntityFieldto""in the non-enriched branch - Move
actorDocDataandtargetDocDatabelow, after the conditional (where we evaluatesourceIps), and evaluate it as you do at lines 274-284, so we concat theactorEntityFieldwhatever it is:
| EVAL actorDocData = CONCAT("{",
"\\"id\\":\\"", actorEntityId, "\\"",
",\\"type\\":\\"", "${DOCUMENT_TYPE_ENTITY}", "\\"",
",\\"entityFieldNamespace\\":\\"", actorEntityFieldHint, "\\"",
actorEntityField,
"}")
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed duplication, thanks.
| // Target: Use node ID from ES|QL or UUID for unknown | ||
| const targetId = | ||
| targetIdsCount === 0 | ||
| ? `unknown-${uuidv4()}` // Multiple unknown target nodes possible - differentiate via UUID | ||
| : targetNodeId!; // Use node ID from ES|QL (we know it's not null here) | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could you restore this file? I think it was simpler before since it avoids an extra conditional
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
so many conflicts in this file and at the end this is the same code.
| entity.target.id, | ||
| MV_DEDUPE(MV_APPEND(targetEntityId, entity.target.id)) | ||
| ) | ||
| ) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very unfortunate MV_APPEND doesn't work with NULL :( This could be simplified so much
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep, a good suggestion maybe for esql team to expand the capability of this function.
| }) | ||
| ), | ||
| entity: schema.maybe(entitySchema), | ||
| entityFieldNamespace: schema.maybe(schema.string()), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: I'd name this field entityNamespace or entityTypeNamespace since it denotes the category/basket the entity type belongs to. Having the "field" term in the name feels too generic to me
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I had a feeling this name would cause a debate, i wanted to add the 'field' to indicate that is some kind of a hint that means that the entity field in the document resides inside a namespace, e.g user/host/service etc.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Naming is hard 😅 not a strong opinion, just a nit. Maybe @kfirpeled has a better thought
| targetHostIp = VALUES(targetHostIp), | ||
| targetsDocData = VALUES(targetDocData) | ||
| targetsDocData = VALUES(targetDocData), | ||
| targetEntityFieldHint = VALUES(targetEntityFieldHint) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should remove both actorEntityFieldHint and targetEntityFieldHint from this STATS block because we don't use them elsewhere, they're only as part of the JSON docs in actorsDocData and targetsDocData above. That would allow us to have a cleaner record interface.
So I'd advocate to remove them from here as well as in types.ts and all test cases in parse_records.test.ts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
you are correct, these are leftovers because i moved the field to documentsData
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Public APIs missing comments
Async chunks
Unknown metric groupsAPI count
History
|
Summary
This PR transitions the graph visualization feature to exclusively use ECS (Elastic Common Schema) entity fields for actor and target identification, replacing legacy
actor.entity.idandtarget.entity.idfields. The graph visualization now requires Elastic stack v9.3.0 for proper functionality with logs and alerts containing the new ECS schema fields.This PR focuses on updating the Kibana graph query backend to support querying the new ECS schema fields:
user.entity.id,host.entity.id,service.entity.id,entity.id,user.target.entity.id,host.target.entity.id,service.target.entity.id,entity.target.id.Related issue.
Another enhancement introduced is enabling proper ECS-aware filtering for node actions. Based on a hint which is added to each node's documentsDat field while keeping root-level node fields unchanged. This allows the frontend to dynamically construct the correct ECS field names (e.g., user.entity.id, host.entity.id, service.entity.id) when users interact with graph nodes.
Key Changes - Query Builder
1. Actor vs Target Field Handling
2. ECS Entity Field Support
New Actor Entity Fields (prioritized order):
user.entity.idhost.entity.idservice.entity.identity.idNew Target Entity Fields (collected from ALL fields):
user.target.entity.idhost.target.entity.idservice.target.entity.identity.target.idNote: Unlike actor fields (which use COALESCE for prioritization), target fields use MV_APPEND to collect ALL values from all target fields, enabling comprehensive relationship mapping.
3. Backend Query Logic (
fetch_graph.ts)Updated ES|QL queries to use COALESCE-based fallback mechanism:
The query uses MV_APPEND instead of COALESCE for target fields to:
targetEntityFieldHintNamespace Tracking:
// Track which field each actor ID came from
WHERE Clause: Filters documents that have at least one actor entity field:
DSL Filter: Checks for target entity field existence (when
showUnknownTargetis false):4. Frontend Changes (
use_graph_preview.ts)Graph preview hook now:
5. Test Updates
entityStoreHelpersTest Data Archives:
logs_gcp_audit/data.json: events containing BOTH legacy and new ECS fields.logs_gcp_audit_ecs_only/data.json: events containing ONLY new ECS fields.security_alerts/data.json: alerts containing ONLY legacy ECS fields.security_alerts_ecs_and_legacy_mappings/data.json: alerts containing BOTH legacy and new ECS fields.security_alerts_ecs/data.json: alerts containing ONLY new ECS fields.logs_gcp_audit/data.json: events containing BOTH legacy and new ECS fields.security_alerts_modified_mappings/data.json: alerts containing BOTH legacy and new ECS fields.security_alerts_ecs_only_mappings/data.json: alerts containing ONLY new ECS fields.Backward Compatibility Approach
IMPORTANT: While we maintain backward compatibility at the data integration level by continuing to populate legacy fields (
actor.entity.id,target.entity.id) in logs and alerts, the graph visualization feature exclusively uses the new ECS schema fields starting from version 9.3.0.Data Layer vs Visualization Layer
| Layer | Legacy Fields | New ECS Fields | Support |
|-------|--------------|----------------|---------|
| Data Ingestion | ✅ Populated | ✅ Populated | Backward compatible |
| Graph Visualization | ❌ Not queried | ✅ Required | v9.3.0+ only |
Key Changes - filtering
API Enhancement:
entityFieldNamespacein Node MetadataBackend (
parse_records.ts):entityFieldNamespacetoNodeDocumentDataModelin the schema (v1.ts)entityFieldNamespacefor both enriched and non-enriched entities'user','host','service', or'entity'Frontend (
use_entity_node_expand_popover.ts):entityFieldNamespace:getSourceNamespaceFromNode(): Extracts namespace from node'sdocumentsData[0]getActorFieldFromNamespace(): Maps namespace to actor field (e.g.,'user'→'user.entity.id')getTargetFieldFromNamespace(): Maps namespace to target field (e.g.,'user'→'user.target.entity.id')entityFieldNamespaceWhy This Matters:
actor.entity.id,target.entity.id)Test Coverage Summary:
entityFieldNamespaceis returned whether entity is enriched or notuser,service,host,entity) in the same graphUser Experience
How to test
alexreal1314:14512-aws-cloudtrail-esc-schemaof integrations repo with updated mappings.data_stream.dataset: "aws.cloudtrail"Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
release_note:breakinglabel should be applied in these situations.release_note:*label is applied per the guidelinesbackport:*labels.Video Recording - with ecs schema fields
Screen.Recording.2025-11-23.at.13.08.41.mov
Video Recording - without ecs schema fields - single target field
Screen.Recording.2025-11-17.at.17.24.18.mov
Video Recording - without ecs schema fields - multi target fields
Screen.Recording.2025-11-30.at.16.19.41.mov
Screenshots
single events not enriched:
single event not enriched - disabled 'show entity details' option:
single event enriched - enabled 'show entity details' option:
single actor with a single target field with multiple values - not enriched and entity store disabled:
single actor with two target fields - each target field has two values - not enriched and entity store disabled:
single actor with two target fields - each target field has two values - each target is enriched with entity data:
grouped node:
grouped events:
single entity node - not enriched:
single event:
grouped entity node - enriched: