Skip to content

Conversation

@alexreal1314
Copy link
Contributor

@alexreal1314 alexreal1314 commented Nov 20, 2025

Summary

This PR transitions the graph visualization feature to exclusively use ECS (Elastic Common Schema) entity fields for actor and target identification, replacing legacy actor.entity.id and target.entity.id fields. The graph visualization now requires Elastic stack v9.3.0 for proper functionality with logs and alerts containing the new ECS schema fields.

This PR focuses on updating the Kibana graph query backend to support querying the new ECS schema fields: user.entity.id, host.entity.id, service.entity.id, entity.id, user.target.entity.id, host.target.entity.id, service.target.entity.id, entity.target.id.
Related issue.

Another enhancement introduced is enabling proper ECS-aware filtering for node actions. Based on a hint which is added to each node's documentsDat field while keeping root-level node fields unchanged. This allows the frontend to dynamically construct the correct ECS field names (e.g., user.entity.id, host.entity.id, service.entity.id) when users interact with graph nodes.

Key Changes - Query Builder

1. Actor vs Target Field Handling

Aspect Actor Fields Target Fields
Strategy COALESCE (prioritization) MV_APPEND (collect all)
Result Single actor per event Multiple targets per event
Row Expansion MV_EXPAND actorEntityId MV_EXPAND targetEntityId
Namespace Tracking actorEntityFieldHint targetEntityFieldHint

2. ECS Entity Field Support

New Actor Entity Fields (prioritized order):

  • user.entity.id
  • host.entity.id
  • service.entity.id
  • entity.id

New Target Entity Fields (collected from ALL fields):

  • user.target.entity.id
  • host.target.entity.id
  • service.target.entity.id
  • entity.target.id

Note: Unlike actor fields (which use COALESCE for prioritization), target fields use MV_APPEND to collect ALL values from all target fields, enabling comprehensive relationship mapping.

3. Backend Query Logic (fetch_graph.ts)

Updated ES|QL queries to use COALESCE-based fallback mechanism:

// Actor entity ID resolution with field precedence
EVAL actorEntityId = COALESCE(
  user.entity.id,
  host.entity.id,
  service.entity.id,
  entity.id
)

// Target entity ID resolution - collect ALL values from all fields
EVAL targetEntityId = MV_APPEND(
  MV_APPEND(
    MV_APPEND(
      user.target.entity.id,
      host.target.entity.id
    ),
    service.target.entity.id
  ),
  entity.target.id

...

| MV_EXPAND actorEntityId
| MV_EXPAND targetEntityId

The query uses MV_APPEND instead of COALESCE for target fields to:

  • ✅ Collect ALL target entity IDs from ALL target fields
  • ✅ Support multiple target entities per event (e.g., one service acting on multiple hosts)
  • ✅ Create separate graph edges for each actor-target relationship
  • ✅ Track which ECS field each target ID originated from via targetEntityFieldHint

Namespace Tracking:

// Track which field each actor ID came from

| EVAL actorEntityFieldHint = CASE(
    user.entity.id IS NOT NULL, "user",
    host.entity.id IS NOT NULL, "host",
    service.entity.id IS NOT NULL, "service",
    entity.id IS NOT NULL, "entity",
    ""
 )

// Track which field each target ID came from
EVAL targetEntityFieldHint = CASE(
  MV_CONTAINS(user.target.entity.id, targetEntityId), "user",
  MV_CONTAINS(host.target.entity.id, targetEntityId), "host",
  MV_CONTAINS(service.target.entity.id, targetEntityId), "service",
  MV_CONTAINS(entity.target.id, targetEntityId), "entity",
  ""
)

WHERE Clause: Filters documents that have at least one actor entity field:

WHERE event.action IS NOT NULL AND (
  user.entity.id IS NOT NULL OR 
  host.entity.id IS NOT NULL OR 
  service.entity.id IS NOT NULL OR 
  entity.id IS NOT NULL
)

DSL Filter: Checks for target entity field existence (when showUnknownTarget is false):

bool: {
  should: [
    { exists: { field: 'user.target.entity.id' } },
    { exists: { field: 'host.target.entity.id' } },
    { exists: { field: 'service.target.entity.id' } },
    { exists: { field: 'entity.target.id' } }
  ],
  minimum_should_match: 1
}

4. Frontend Changes (use_graph_preview.ts)

Graph preview hook now:

  • Checks for new ECS entity fields exclusively
  • Requires BOTH actor AND target fields for graph representation
  • Removed legacy field detection logic
const hasGraphRepresentation = 
  Boolean(timestamp) && 
  Boolean(action?.length) && 
  eventIds.length > 0 && 
  actorIds.length > 0 &&    // At least one actor ECS field
  targetIds.length > 0;      // At least one target ECS field

5. Test Updates

  • Updated all test queries to use new ECS fields
  • Refactored entity store setup logic into reusable entityStoreHelpers
  • Created test section: "Only ECS fields for actor and target" to validate the logic against logs and alerts once legacy fields are removed (deprecated currently).

Test Data Archives:

  • Api integration tests:
    • logs_gcp_audit/data.json: events containing BOTH legacy and new ECS fields.
    • logs_gcp_audit_ecs_only/data.json: events containing ONLY new ECS fields.
    • security_alerts/data.json: alerts containing ONLY legacy ECS fields.
    • security_alerts_ecs_and_legacy_mappings/data.json: alerts containing BOTH legacy and new ECS fields.
    • security_alerts_ecs/data.json: alerts containing ONLY new ECS fields.
  • Updated mappings to include all new ECS entity field definitions
  • FTR tests:
    • logs_gcp_audit/data.json: events containing BOTH legacy and new ECS fields.
    • security_alerts_modified_mappings/data.json: alerts containing BOTH legacy and new ECS fields.
    • security_alerts_ecs_only_mappings/data.json: alerts containing ONLY new ECS fields.

Backward Compatibility Approach

IMPORTANT: While we maintain backward compatibility at the data integration level by continuing to populate legacy fields (actor.entity.id, target.entity.id) in logs and alerts, the graph visualization feature exclusively uses the new ECS schema fields starting from version 9.3.0.

Data Layer vs Visualization Layer

| Layer | Legacy Fields | New ECS Fields | Support |

|-------|--------------|----------------|---------|

| Data Ingestion | ✅ Populated | ✅ Populated | Backward compatible |

| Graph Visualization | ❌ Not queried | ✅ Required | v9.3.0+ only |

Key Changes - filtering

API Enhancement: entityFieldNamespace in Node Metadata

Backend (parse_records.ts):

  • Added entityFieldNamespace to NodeDocumentDataModel in the schema (v1.ts)
  • Populated entityFieldNamespace for both enriched and non-enriched entities
  • The field stores the ECS namespace type: 'user', 'host', 'service', or 'entity'

Frontend (use_entity_node_expand_popover.ts):

  • Added helper functions to extract and use entityFieldNamespace:
    • getSourceNamespaceFromNode(): Extracts namespace from node's documentsData[0]
    • getActorFieldFromNamespace(): Maps namespace to actor field (e.g., 'user''user.entity.id')
    • getTargetFieldFromNamespace(): Maps namespace to target field (e.g., 'user''user.target.entity.id')
  • Updated filter creation logic to use dynamic field names based on entityFieldNamespace

Why This Matters:

  • Before: Node actions used hardcoded legacy fields (actor.entity.id, target.entity.id)
  • After: Node actions dynamically use correct ECS fields based on entity type
  • Result: Filters like "Show actions by entity" now work correctly with the new ECS schema

Test Coverage Summary:

  • Happy flows: Checking that entityFieldNamespace is returned whether entity is enriched or not
  • Multi-namespace test: Different entity types (user, service, host, entity) in the same graph
  • Query filtering test: Filtering by new ECS fields when entity is not enriched
  • Enriched entity test: With entity metadata from Asset Inventory and multi-values in target

User Experience

  • ✅ Node action filters now work correctly with ECS schema
  • ✅ "Show actions by entity" uses the correct field name based on entity type
  • ✅ "Show actions on entity" creates proper bidirectional filters
  • ✅ "Explore related entities" includes all relevant ECS fields

How to test

  1. Create a hosted ENV from this PR.
  2. Install Cloud Asset Discovery (skip agent installation)
  3. Re-index Asset Inventory data with this query:
POST _reindex
{
  "conflicts": "proceed",
  "source": {
    "remote": { 
      "host": "${ES_REMOTE_HOST}",
      "username": "${ES_REMOTE_USER}",
      "password": "${ES_REMOTE_PASS}"
    },
    "index": ".entities.v1.latest.security_generic_default",
    "query": {
      "bool": {
        "must": [
          {
            "range": {
              "@timestamp": {
                "gte": "now-1y"
              }
            }
          }
        ]
      }
    }
  },
  "dest": {
    "op_type": "create",
    "index": ".entities.v1.latest.security_generic_default"
  },
  "script": {
    "source": """
      ctx._source.doc_id = ctx._id;
      ctx._source.doc_index = ctx._index;

      if (ctx._source.asset != null) {
        if (ctx._source.asset.containsKey('category')) {
          ctx._source['entity.category'] = ctx._source.asset.category;
        }
        if (ctx._source.asset.containsKey('name')) {
          ctx._source['entity.name'] = ctx._source.asset.name;
        }
        if (ctx._source.asset.containsKey('type')) {
          ctx._source['entity.type'] = ctx._source.asset.type;
        }
        if (ctx._source.asset.containsKey('sub_type')) {
          ctx._source['entity.sub_type'] = ctx._source.asset.sub_type;
        }
        if (ctx._source.asset.containsKey('sub_category')) {
          ctx._source['entity.sub_category'] = ctx._source.asset.sub_category;
        }
      }
    """
  }
}
  1. Clone the following branch - alexreal1314:14512-aws-cloudtrail-esc-schema of integrations repo with updated mappings.
  2. Set env variables to point to this PR deployment - installation guide and install the ingration.
  3. Open aws-cloudtrail integration, make sure the version is the same as in the cloned integrations repo.
  4. Install the integration.
  5. Documents should contain one of user/host/service.entity.id fields and one of user/host/service/target.entity.
  6. Either rules with query data_stream.dataset: "aws.cloudtrail"
  7. Navigate to alerts/explore section and open of the rows flyout which contain graph data.
  8. Graph should work as expected without applying any filters

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
  • Documentation was added for features that require explanation or tutorials
  • Unit or functional tests were updated or added to match the most common scenarios
  • If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
  • This was checked for breaking HTTP API changes, and any breaking changes have been approved by the breaking-change committee. The release_note:breaking label should be applied in these situations.
  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

Video Recording - with ecs schema fields

Screen.Recording.2025-11-23.at.13.08.41.mov

Video Recording - without ecs schema fields - single target field

Screen.Recording.2025-11-17.at.17.24.18.mov

Video Recording - without ecs schema fields - multi target fields

Screen.Recording.2025-11-30.at.16.19.41.mov

Screenshots

single events not enriched:

image

single event not enriched - disabled 'show entity details' option:

image

single event enriched - enabled 'show entity details' option:

image

single actor with a single target field with multiple values - not enriched and entity store disabled:

image

single actor with two target fields - each target field has two values - not enriched and entity store disabled:

image

single actor with two target fields - each target field has two values - each target is enriched with entity data:

image

grouped node:

image

grouped events:

image

single entity node - not enriched:

image

single event:

image

grouped entity node - enriched:

image

@alexreal1314 alexreal1314 force-pushed the 14516-support-filtering-new-ecs-fields branch from bfa5d96 to 68ea2bb Compare November 20, 2025 22:57
@alexreal1314 alexreal1314 changed the title 14516 support filtering new ecs fields [Contextual Security] add support for new actor and target ECS schema fields including query builder and filtering logic Nov 20, 2025
@github-actions
Copy link
Contributor

🔍 Preview links for changed docs

@alexreal1314 alexreal1314 force-pushed the 14516-support-filtering-new-ecs-fields branch 5 times, most recently from e3e3d27 to da26c57 Compare November 27, 2025 15:51
@alexreal1314 alexreal1314 force-pushed the 14516-support-filtering-new-ecs-fields branch from a9ea81c to 9432c33 Compare November 30, 2025 09:29
@alexreal1314 alexreal1314 self-assigned this Nov 30, 2025
@alexreal1314 alexreal1314 added Team:Cloud Security Cloud Security team related release_note:enhancement backport:skip This PR does not require backporting ci:build-serverless-image ci:cloud-deploy Create or update a Cloud deployment ci:cloud-redeploy Always create a new Cloud deployment labels Nov 30, 2025
@alexreal1314 alexreal1314 force-pushed the 14516-support-filtering-new-ecs-fields branch from 9432c33 to 5bb24da Compare November 30, 2025 10:24
@alexreal1314 alexreal1314 changed the title [Contextual Security] add support for new actor and target ECS schema fields including query builder and filtering logic [Contextual Security] align graph visualization with ECS entity namespace fields for actor/target identification Nov 30, 2025
@alexreal1314 alexreal1314 changed the title [Contextual Security] align graph visualization with ECS entity namespace fields for actor/target identification [Contextual Security] align graph visualization with ECS entity namespace fields for actor and target identification Nov 30, 2025
@alexreal1314 alexreal1314 force-pushed the 14516-support-filtering-new-ecs-fields branch from 9c1516f to f81df85 Compare December 1, 2025 08:02
@kibanamachine
Copy link
Contributor

Project deployments require a Github label, please add one or more of ci:project-deploy-(elasticsearch|observability|security) and trigger the job through the checkbox again.

@kibanamachine
Copy link
Contributor

PR Cloud deployment started at: https://buildkite.com/elastic/kibana-deploy-cloud-from-pr/builds/602

@alexreal1314 alexreal1314 marked this pull request as ready for review December 1, 2025 10:35
@alexreal1314 alexreal1314 requested review from a team as code owners December 1, 2025 10:35
@elasticmachine
Copy link
Contributor

Pinging @elastic/contextual-security-apps (Team:Cloud Security)

- Add NON_ENRICHED_ENTITY_TYPE constant
- Update entity group/type fallback to use 'Entities'
- Add actorEntitySubType and targetEntitySubType to STATS aggregation
- Add new label logic after STATS based on entity count

Verification:
- ESLint: PASS
- Type check: PASS
- Unit tests: PASS
Tests verify ESQL query string contains:
- 'Entities' fallback for entity groups and types
- Label logic after STATS based on entity count
- actorEntitySubType and targetEntitySubType in STATS aggregation

Verification:
- ESLint: PASS
- Type check: PASS
- Unit tests: PASS (10 tests passing)
alexreal1314 and others added 18 commits December 1, 2025 18:43
…and targetsDocData object

fixed and added unit tests to fetchGraph and parseRecords, updated also graph api integration tests
enhanced the graph visualization API to handle multiple target
@alexreal1314 alexreal1314 force-pushed the 14516-support-filtering-new-ecs-fields branch from f81df85 to 931ec5b Compare December 1, 2025 20:23
Copy link
Contributor

@PhilippeOberti PhilippeOberti left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the 4 files impacting the @elastic/security-threat-hunting-investigations team. I did not look at any of the other 25k lines added by this PR 😅 (which seems that a lot tbh)

Copy link
Contributor

@albertoblaz albertoblaz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes because of the regression spotted eliminating COALESCE from the query

Comment on lines 247 to 258
actorEntityName IS NOT NULL OR actorEntityType IS NOT NULL OR actorEntitySubType IS NOT NULL,
CONCAT(",\\"entity\\":", "{",
"\\"name\\":\\"", actorEntityName, "\\"",
",\\"type\\":\\"", actorEntityType, "\\"",
",\\"sub_type\\":\\"", actorEntitySubType, "\\"",
CASE(
actorHostIp IS NOT NULL,
CONCAT(",\\"host\\":", "{", "\\"ip\\":\\"", TO_STRING(actorHostIp), "\\"", "}"),
""
),
"}",
"}")
| EVAL targetDocData = CONCAT("{",
"\\"id\\":\\"", COALESCE(target.entity.id, ""), "\\"",
",\\"type\\":\\"", "${DOCUMENT_TYPE_ENTITY}", "\\"",
",\\"entity\\":", "{",
"\\"name\\":\\"", COALESCE(targetEntityName, ""), "\\"",
",\\"type\\":\\"", COALESCE(targetEntityType, ""), "\\"",
",\\"sub_type\\":\\"", COALESCE(targetEntitySubType, ""), "\\"",
CASE (
"}"),
""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a regression.

If we find an non-null actorEntityType and actorEntitySubType but actorEntityName is NULL, CONCAT will fail because it doesn't support NULL values so the whole JSON won't be build.

We need to restore COALESCE(<field>, "undefined") on each line instead (@kfirpeled pointed out to better fallback to "undefined" than "")

Comment on lines 298 to 308
// Create minimal actor and target data with entityFieldNamespace even without enrichment
| EVAL actorDocData = CONCAT("{",
"\\"id\\":\\"", actorEntityId, "\\"",
",\\"type\\":\\"", "${DOCUMENT_TYPE_ENTITY}", "\\"",
",\\"entityFieldNamespace\\":\\"", actorEntityFieldHint, "\\"",
"}")
| EVAL targetDocData = CONCAT("{",
"\\"id\\":\\"", COALESCE(targetEntityId, ""), "\\"",
",\\"type\\":\\"", "${DOCUMENT_TYPE_ENTITY}", "\\"",
",\\"entityFieldNamespace\\":\\"", targetEntityFieldHint, "\\"",
"}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see this is duplicated in both enriched and non-enriched logic branches and could be simplified.

I would:

  • Set actorEntityField to the JSON object in the enriched branch
  • Set actorEntityField to "" in the non-enriched branch
  • Move actorDocData and targetDocData below, after the conditional (where we evaluate sourceIps), and evaluate it as you do at lines 274-284, so we concat the actorEntityField whatever it is:
| EVAL actorDocData = CONCAT("{",
    "\\"id\\":\\"", actorEntityId, "\\"",
    ",\\"type\\":\\"", "${DOCUMENT_TYPE_ENTITY}", "\\"",
    ",\\"entityFieldNamespace\\":\\"", actorEntityFieldHint, "\\"",
    actorEntityField,
  "}")

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

removed duplication, thanks.

Comment on lines 209 to 214
// Target: Use node ID from ES|QL or UUID for unknown
const targetId =
targetIdsCount === 0
? `unknown-${uuidv4()}` // Multiple unknown target nodes possible - differentiate via UUID
: targetNodeId!; // Use node ID from ES|QL (we know it's not null here)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you restore this file? I think it was simpler before since it avoids an extra conditional

Copy link
Contributor Author

@alexreal1314 alexreal1314 Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so many conflicts in this file and at the end this is the same code.

entity.target.id,
MV_DEDUPE(MV_APPEND(targetEntityId, entity.target.id))
)
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very unfortunate MV_APPEND doesn't work with NULL :( This could be simplified so much

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, a good suggestion maybe for esql team to expand the capability of this function.

})
),
entity: schema.maybe(entitySchema),
entityFieldNamespace: schema.maybe(schema.string()),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I'd name this field entityNamespace or entityTypeNamespace since it denotes the category/basket the entity type belongs to. Having the "field" term in the name feels too generic to me

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had a feeling this name would cause a debate, i wanted to add the 'field' to indicate that is some kind of a hint that means that the entity field in the document resides inside a namespace, e.g user/host/service etc.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Naming is hard 😅 not a strong opinion, just a nit. Maybe @kfirpeled has a better thought

targetHostIp = VALUES(targetHostIp),
targetsDocData = VALUES(targetDocData)
targetsDocData = VALUES(targetDocData),
targetEntityFieldHint = VALUES(targetEntityFieldHint)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should remove both actorEntityFieldHint and targetEntityFieldHint from this STATS block because we don't use them elsewhere, they're only as part of the JSON docs in actorsDocData and targetsDocData above. That would allow us to have a cleaner record interface.

So I'd advocate to remove them from here as well as in types.ts and all test cases in parse_records.test.ts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you are correct, these are leftovers because i moved the field to documentsData

@elasticmachine
Copy link
Contributor

elasticmachine commented Dec 3, 2025

💛 Build succeeded, but was flaky

Failed CI Steps

Metrics [docs]

Public APIs missing comments

Total count of every public API that lacks a comment. Target amount is 0. Run node scripts/build_api_docs --plugin [yourplugin] --stats comments for more detailed information.

id before after diff
@kbn/cloud-security-posture-common 218 220 +2
@kbn/cloud-security-posture-graph 44 45 +1
total +3

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 11.1MB 11.1MB +1.3KB
Unknown metric groups

API count

id before after diff
@kbn/cloud-security-posture-common 220 222 +2
@kbn/cloud-security-posture-graph 65 67 +2
total +4

History

cc @alexreal1314

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport:skip This PR does not require backporting ci:build-serverless-image ci:cloud-deploy Create or update a Cloud deployment ci:cloud-redeploy Always create a new Cloud deployment release_note:enhancement Team:Cloud Security Cloud Security team related

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants