[aws_securityhub] Initial release of AWS Security Hub #15932

brijesh-elastic · 2025-11-11T11:18:04Z

Proposed commit message

aws_securityhub: Initial release of AWS Security Hub

the integration collects findings in OCSF format using GetFindingsV2 API, it supports
OCSF v1.5 classes(2002, 2003, 2004, 2006).

ECS mapping and transforms have also been added to facilitate with the
Cloud Native Vulnerability Management (CNVM)[1] workflow.

[1] https://www.elastic.co/guide/en/security/current/vuln-management-overview.html

Note

To Reviewers:

Please refer to this guide: https://docs.elastic.dev/security-solution/cloud-security/cdr/3p-dev-guide which was used for proposed changes.

Checklist

I have reviewed tips for building integrations and this pull request is aligned with them.
I have verified that all data streams collect metrics or logs.
I have added an entry to my package's changelog.yml file.
I have verified that Kibana version constraints are current according to guidelines.
I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

Clone integrations repo.
Install elastic package locally.
Start elastic stack using elastic-package.
Move to integrations/packages/aws_securityhub directory.
Run the following command to run tests.

elastic-package test

Related issues

…issions (#137866) Adding logs-aws_securityhub.finding-* data stream indices to the kibana_system privileges. This is required for the latest transform to work. Related: elastic/integrations#15932

elasticmachine · 2025-12-02T04:12:40Z

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

elastic-vault-github-plugin-prod · 2025-12-02T13:51:10Z

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

elasticmachine · 2025-12-02T13:51:13Z

💚 Build Succeeded

Buildkite Build
Commit: aeebca9

History

💔 Build #34903 failed 60d0198
💔 Build #34463 failed 5d9dd72
💔 Build #33954 failed be3cd17

cc @brijesh-elastic

efd6 · 2025-12-02T21:19:47Z

packages/aws_securityhub/_dev/build/docs/README.md

+
+#### Finding
+
+{{/* {{event "finding"}} */}}


Is this omitted for size?

efd6 · 2025-12-02T21:24:41Z

packages/aws_securityhub/data_stream/finding/agent/stream/cel.yml.hbs

+          }]
+        },
+        "MaxResults": state.batch_size,
+        ?"NextToken": state.?next_token,


Does the API accept a null NextToken? What is the behaviour in that case?

efd6 · 2025-12-02T21:30:08Z

...aws_securityhub/data_stream/finding/elasticsearch/ingest_pipeline/pipeline_object_device.yml

+      type: string
+      ignore_missing: true
+
+#   # populate ECS fields


Is this intentional?

efd6 · 2025-12-02T21:41:50Z

packages/aws_securityhub/data_stream/finding/elasticsearch/ingest_pipeline/default.yml

+            - ISO8601
+            - UNIX_MS
+            - yyyy-MM-dd HH:mm:ss[.SSSSSSSSS][.SSSSSSSS][.SSSSSSS][.SSSSSS][.SSSSS][.SSSS][.SSS][.SS][.S]X
+          on_failure:


Are these failures expected or exceptional? If they are exceptional, we should add a message to error.message on failure.

efd6 · 2025-12-02T21:41:57Z

packages/aws_securityhub/data_stream/finding/elasticsearch/ingest_pipeline/default.yml

+          target_field: _ingest._value.created_time
+          formats:
+            - UNIX_MS
+          on_failure:


efd6 · 2025-12-02T22:23:15Z

...ityhub/data_stream/finding/elasticsearch/ingest_pipeline/pipeline_object_vulnerabilities.yml

+            - ISO8601
+            - UNIX_MS
+            - yyyy-MM-dd HH:mm:ss[.SSSSSSSSS][.SSSSSSSS][.SSSSSSS][.SSSSSS][.SSSSS][.SSSS][.SSS][.SS][.S]X
+          on_failure:


Expected or exceptional? (also below)

efd6 · 2025-12-02T22:24:06Z

packages/aws_securityhub/data_stream/finding/fields/ecs.yml

Do we want to provide the values here?

No, because they need to be in _source for the Findings page to work: #11608 (comment)

efd6 · 2025-12-02T22:24:54Z

packages/aws_securityhub/data_stream/finding/fields/fields.yml

This is autogenerated from the OCSF schema, correct?

efd6 · 2025-12-02T22:25:45Z

packages/aws_securityhub/elasticsearch/transform/latest_cdr_vulnerabilities/fields/fields.yml

Autogenerated?

efd6 · 2025-12-02T22:26:32Z

packages/aws_securityhub/elasticsearch/transform/latest_findings/fields/fields.yml

Autogenerated?

efd6 · 2025-12-03T02:56:56Z

I think also that the document that forms the basis of the OCSF to ECS mapping should be made public and linked in the commit message.

kcreddy · 2025-12-03T09:55:52Z

packages/aws_securityhub/_dev/build/docs/README.md

+
+### How it works
+
+The **finding** data stream uses the `/findingsv2` endpoint to gather all findings starting from the configured initial interval. Subsequently, it fetches the recent findings available at each specified interval.


Suggested change

The **finding** data stream uses the `/findingsv2` endpoint to gather all findings starting from the configured initial interval. Subsequently, it fetches the recent findings available at each specified interval.

The **finding** data stream uses the `/findingsv2` endpoint to gather all findings starting from the configured `Initial Interval`. Subsequently, it fetches the recent findings available at each specified `Interval`.

kcreddy · 2025-12-03T09:56:47Z

packages/aws_securityhub/_dev/build/docs/README.md

+
+## What data does this integration collect?
+
+The AWS Security Hub integration collects log messages of the following types:


Suggested change

The AWS Security Hub integration collects log messages of the following types:

The AWS Security Hub integration collects logs of the following types:

or events

kcreddy · 2025-12-03T09:57:47Z

packages/aws_securityhub/_dev/build/docs/README.md

+### Supported use cases
+Integrating AWS Security Hub with Elastic SIEM provides a comprehensive view of the security state of your AWS resources. Leveraging AWS Security Hub integration helps you analyze security trends to identify and prioritize security issues across your AWS environment.


Suggested change

### Supported use cases

Integrating AWS Security Hub with Elastic SIEM provides a comprehensive view of the security state of your AWS resources. Leveraging AWS Security Hub integration helps you analyze security trends to identify and prioritize security issues across your AWS environment.

### Supported use cases

Integrating AWS Security Hub with Elastic SIEM provides a comprehensive view of the security state of your AWS resources. Leveraging AWS Security Hub integration helps you analyze security trends to identify and prioritize security issues across your AWS environment.

Also adding CNVM use case would help

maxcold · 2025-12-03T11:00:47Z

@kcreddy @brijesh-elastic Tested the env with data ingested. All good except missing host.name . In the mapping sheet we discussed that we want to provide the instance id in the host.name. Did we miss this requirement or were there any technical challenges doing so?

brijesh-elastic · 2025-12-03T12:14:28Z

@kcreddy @brijesh-elastic Tested the env with data ingested. All good except missing host.name . In the mapping sheet we discussed that we want to provide the instance id in the host.name. Did we miss this requirement or were there any technical challenges doing so?

It looks like I missed it. I’ll update this in the next commit, along with the cluster.

kcreddy

To be merged after #16195

kcreddy · 2025-12-03T12:01:29Z

packages/aws_securityhub/_dev/build/docs/README.md

+
+#### Collecting data from AWS Security Hub API
+
+Users can authenticate using permanent security credentials, as well as temporary security credentials. They can also select `shared_credential_file`, `credential_profile_name` to retrieve credentials. Additionally, they can use `role_arn` to specify which AWS IAM role to assume for generating temporary credentials. An `external_id` can also be provided when assuming a role in another account.


I think a link to documentation is important to understand what all these options mean, and where to configure them. Also use the user facing text (title instead of name).

kcreddy · 2025-12-03T12:02:58Z

packages/aws_securityhub/_dev/build/docs/README.md

+1. In the top search bar in Kibana, search for **Integrations**.
+2. In the search bar, type **AWS Security Hub**.
+3. Select the **AWS Security Hub** integration from the search results.
+4. Select **Add AWS Security Hubs** to add the integration.


Suggested change

4. Select **Add AWS Security Hubs** to add the integration.

4. Select **Add AWS Security Hub** to add the integration.

kcreddy · 2025-12-03T12:03:45Z

packages/aws_securityhub/_dev/build/docs/README.md

+
+### Validation
+
+#### Transforms healthy


Could we also add dashboards?

kcreddy · 2025-12-03T12:05:05Z

packages/aws_securityhub/data_stream/finding/_dev/test/pipeline/test-common-config.yml

+  # This can be removed after ES 8.14 is the minimum version.
+  # Relates: https://github.com/elastic/elasticsearch/pull/105689
+  url.extension: '^.*$'


not required I guess?

kcreddy · 2025-12-03T12:07:09Z

packages/aws_securityhub/_dev/deploy/docker/files/config.yml

+    request_headers:
+      Content-Type:
+        - "application/json"
+    request_body: /^\{"Filters":\{"CompositeFilters":\[\{"DateFilters":\[\{"FieldName":"finding_info\.modified_time_dt","Filter":\{"End":.*,"Start":.*\}\}\]\}\]\},"MaxResults":2,"NextToken":"next_token_2","SortCriteria":\[\{"Field":"finding_info\.modified_time_dt","SortOrder":"asc"\}\]\}/


Can we also test empty results case? Do we get empty "Findings": [] array?

kcreddy · 2025-12-03T12:38:30Z

packages/aws_securityhub/data_stream/finding/manifest.yml

+      index:
+        mapping:
+          total_fields:
+            limit: 2000


How many mapped fields do we have right now?

kcreddy · 2025-12-03T12:40:33Z

packages/aws_securityhub/data_stream/finding/manifest.yml

+        title: Interval
+        description: Duration between requests to the AWS Security Hub API. Supported units for this parameter are h/m/s.
+        default: 1h
+        max_duration: 24h


What is the intention for having this max_duration on Interval?

kcreddy · 2025-12-03T12:41:23Z

packages/aws_securityhub/data_stream/finding/manifest.yml

+        multi: false
+        required: true
+        show_user: false
+        default: 30s


Do you think we should we should increase the default based on our testing?

kcreddy · 2025-12-03T12:46:36Z

packages/aws_securityhub/data_stream/finding/elasticsearch/ingest_pipeline/default.yml

+      if: ctx.aws_securityhub?.finding?.class_uid != null && ['2002','2003','2004','2006'].contains(ctx.aws_securityhub.finding.class_uid)
+      value: alert


I am not sure if we want to categorise vulnerability and misconfiguration as alert. CDR guide maps them as state.

kcreddy · 2025-12-03T12:50:25Z

packages/aws_securityhub/data_stream/finding/elasticsearch/ingest_pipeline/default.yml

+  - set:
+      field: aws_securityhub.finding.transform_unique_id
+      tag: set_transform_unique_id
+      value: '{{event.id}}|{{vulnerability.id}}|{{resource.id}}|{{package.name}}|{{package.version}}'


Can you use condition that its populated only for vulnerabilities?

Initial release

0e62d7d

brijesh-elastic self-assigned this Nov 11, 2025

Update changelog entry and codeowners

be3cd17

brijesh-elastic mentioned this pull request Nov 11, 2025

Add aws_securityhub.finding source indices to kibana_system role permissions elastic/elasticsearch#137866

Merged

andrewkroh added the New Integration Issue or pull request for creating a new integration package. label Nov 11, 2025

brijesh-elastic and others added 2 commits November 21, 2025 13:12

Merge branch 'main' into package-aws_securityhub

5d9dd72

Fix bugs and add transform for latest findings

60d0198

brijesh-elastic marked this pull request as ready for review December 2, 2025 04:12

brijesh-elastic requested a review from a team as a code owner December 2, 2025 04:12

Resolve build failure

aeebca9

efd6 reviewed Dec 2, 2025

View reviewed changes

kcreddy reviewed Dec 3, 2025

View reviewed changes


		### How it works

		The finding data stream uses the `/findingsv2` endpoint to gather all findings starting from the configured initial interval. Subsequently, it fetches the recent findings available at each specified interval.


		## What data does this integration collect?

		The AWS Security Hub integration collects log messages of the following types:

		### Supported use cases
		Integrating AWS Security Hub with Elastic SIEM provides a comprehensive view of the security state of your AWS resources. Leveraging AWS Security Hub integration helps you analyze security trends to identify and prioritize security issues across your AWS environment.


		#### Collecting data from AWS Security Hub API

		Users can authenticate using permanent security credentials, as well as temporary security credentials. They can also select `shared_credential_file`, `credential_profile_name` to retrieve credentials. Additionally, they can use `role_arn` to specify which AWS IAM role to assume for generating temporary credentials. An `external_id` can also be provided when assuming a role in another account.

	4. Select Add AWS Security Hubs to add the integration.
	4. Select Add AWS Security Hub to add the integration.

		if: ctx.aws_securityhub?.finding?.class_uid != null && ['2002','2003','2004','2006'].contains(ctx.aws_securityhub.finding.class_uid)
		value: alert

[aws_securityhub] Initial release of AWS Security Hub #15932

Are you sure you want to change the base?

[aws_securityhub] Initial release of AWS Security Hub #15932

Uh oh!

Conversation

brijesh-elastic commented Nov 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Proposed commit message

Checklist

How to test this PR locally

Related issues

Uh oh!

elasticmachine commented Dec 2, 2025

Uh oh!

elastic-vault-github-plugin-prod bot commented Dec 2, 2025

🚀 Benchmarks report

Uh oh!

elasticmachine commented Dec 2, 2025

💚 Build Succeeded

History

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

efd6 commented Dec 3, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maxcold commented Dec 3, 2025

Uh oh!

brijesh-elastic commented Dec 3, 2025

Uh oh!

kcreddy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

brijesh-elastic commented Nov 11, 2025 •

edited

Loading