From fc2195bd714521ebfc8a84bf85ea5b5c94970faf Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Thu, 6 Nov 2025 17:54:36 +0200 Subject: [PATCH 1/7] Update ecf aws documentation structure and add cloudtrail log type --- docs/reference/edot-cloud-forwarder/aws.md | 217 +++++++++++++++------ 1 file changed, 162 insertions(+), 55 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index fd4f436c..10117f26 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -24,6 +24,7 @@ products: | --- | --- | | Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic | | Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer | +| AWS CloudTrail | [CloudTrail Logs](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html) to record account activity | % | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch | Read on to learn how to set up {{edot-cf}} for AWS. @@ -65,6 +66,15 @@ To collect Elastic Load Balancer (ELB) Access logs, you need: ::: +:::{tab-item} CloudTrail + +To collect AWS CloudTrail logs, you need: + +- A trail that delivers account events as log files to an Amazon S3 bucket +- An S3 bucket to store the trail logs + +::: + + +Each method achieves the same result and uses CloudFormation templates under the hood. Choose the method that best fits your workflow. + ## Deployment considerations Before deploying {{edot-cf}} for AWS, keep these points in mind: @@ -98,9 +122,9 @@ Before deploying {{edot-cf}} for AWS, keep these points in mind: - Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket. - The CloudFormation stack deployment region must match the region of the S3 bucket. -## Download the template [download-templates] +## CloudFormation templates -Download the CloudFormation template to deploy the appropriate stack based on your log source: +The CloudFormation templates are hosted in a public Amazon S3 bucket and can be accessed via HTTPS URL. You can reference these templates directly during deployment or download them for local use. | Log Source | CloudFormation template | | --- | ------------------------------------------------ | @@ -111,7 +135,7 @@ For specific versions, edit `latest` in the URL to the required version in the f ## Configure the template -{{edot-cf}} for AWS uses a CloudFormation template to deploy the EDOT Collector as a Lambda function. +Before deploying {{edot-cf}} for AWS, configure the CloudFormation template parameters based on your specific requirements. The template uses settings below to deploy and configure the EDOT Collector Lambda function. ### Required settings @@ -119,10 +143,9 @@ These are the required settings you need: | Setting | Description | | -------------------------------------- | ----------- | -| `region` | AWS region where the CloudFormation stack is to be deployed | -| `stack-name` | Name of the CloudFormation stack, for example, `vpc-edot-cf`
Do not use the same name for different sources. | -| `OTLPEndpoint` | The OTLP endpoint URL used for data ingestion | -| `ElasticApiKey` | API key for authentication with Elastic | +| `stack-name` | Name of the CloudFormation stack, for example, `vpc-edot-cf`
Do not use the same name for different stacks. | +| `OTLPEndpoint` | The OTLP endpoint URL used for data ingestion, obtained from {{serverless-full}}. | +| `ElasticApiKey` | API key for authentication with Elastic, obtained from {{serverless-full}}. | ### Log source settings @@ -136,7 +159,7 @@ For S3 logs, use the following settings: | Setting | Description | | ------------------ | --- | -| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpcflow`: VPC Flow Logs
- `elbaccess`: ELB Access logs| +| `EdotCloudForwarderS3LogsType` | The encoding format for logs in the S3 bucket. Supported options:
- `vpcflow`: VPC Flow Logs
- `elbaccess`: ELB Access logs
- `cloudtrail`: CloudTrail Logs| | `SourceS3BucketARN` | Amazon Resource Name (ARN) of the S3 bucket where logs are stored. This bucket will trigger the `edot-cloud-forwarder` Lambda function automatically. | % | `S3LogsJsonEncodingMode` | _(Required if `EdotCloudForwarderS3LogsType` is `json`)_
Defines how JSON logs are structured:
- `body` _(default)_: Stores logs in the request body
- `body_with_inline_attributes`: Logs include inline attributes | @@ -166,19 +189,24 @@ These are optional settings you can set in the CloudFormation template: | Setting | Description | | ------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds. Minimum value is `1` second. Maximum value is `900` seconds. | +| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds (5 minutes). Minimum value is `1` second. Maximum value is `900` seconds. **When to increase**: If you experience Lambda timeouts when processing large log files. Most files process in 1-3 seconds, but larger files may need more time. | | `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.2.4`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. | -| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `512` MB. Minimum value is `128` MB. Maximum value is `10240` MB. | -| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. Make sure this value doesn't exceed your AWS account's concurrency limit. | -| `EdotCloudForwarderExporterMaxQueueSize` | Set the internal OTLP exporter queue size. Default is `50` MB. You may incease this based on the data volume. | +| `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `512` MB. Minimum value is `128` MB. Maximum value is `10240` MB. **When to increase**: If memory usage consistently exceeds 80% or if you observe Lambda timeouts and processing times increase significantly. More memory also increases CPU, which improves processing speed and reduces timeouts. | +| `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. **When to increase**: If new log files arrive frequently and you observe Lambda processing times consistently exceeding 1-2 seconds, consider increasing concurrent executions. Concurrent Lambda invocations are only spawned when needed (when new files arrive). This allows multiple log files to be processed in parallel instead of sequentially, avoiding processing delays. Make sure this value doesn't exceed your AWS account's concurrency limit. | +| `EdotCloudForwarderExporterMaxQueueSize` | Set the internal OTLP exporter queue size, measured in bytes. Default is `50000000` (50 MB). **When to increase**: If you see "element size too large" errors in your Lambda CloudWatch logs, indicating the queue is full and data is being dropped. This prevents data loss during export. | + +The default values provided have been determined through extensive load testing across different log types and data volumes. For most use cases, these defaults provide a good balance between cost and performance. Adjust these parameters only if you observe performance issues such as Lambda timeouts, throttling, high memory usage or data being dropped. If you need assistance tuning these parameters for your specific workload, contact Elastic Support. -Default values of `EdotCloudForwarderMemorySize` and `EdotCloudForwarderConcurrentExecutions` are sufficient for most use cases. Key indications for the need of tuning these parameters are Lambda throttling and Lambda timeouts. Along with these, you might also need to adjust `EdotCloudForwarderExporterMaxQueueSize` to export higher data volumes. +## Deploy using CloudFormation (AWS CLI) -## Deployment examples +Use the AWS CLI to deploy the EDOT Cloud Forwarder with CloudFormation templates. This method is ideal for automation and infrastructure-as-code workflows. -The following examples show how to deploy the ECF Cloud Forwarder using AWS CloudFormation. Copy and paste these commands after replacing the placeholder values with your actual configuration. +### Deployment examples + +The following examples show how to deploy the ECF Cloud Forwarder using AWS CloudFormation CLI. Copy and paste these commands after replacing the placeholder values with your actual configuration. - Use the `--template-url` flag to reference a template hosted on S3. +- Use the `--region` flag to specify the AWS region where the CloudFormation stack will be deployed. The CloudFormation stack deployment region must match the region of the S3 bucket where your logs are stored. - To always use the most recent stable templates, use the `latest` path. For example, `v0/latest`. - To pin a specific version, replace `latest` with the desired version tag. For example, `v0/v{{version.edot-cf-aws}}`. @@ -221,41 +249,41 @@ aws cloudformation create-stack \ ``` :::: - ## CloudFormation stack resources @@ -485,6 +546,7 @@ To set up data visualization in {{kib}}: |------------------|---------------------|-----------------| | ELB Access Logs | **AWS ELB OpenTelemetry Assets** | Dashboards and visualizations for Elastic Load Balancer logs | | VPC Flow Logs | **AWS VPC Flow Logs OpenTelemetry Assets** | Dashboards and visualizations for VPC flow log data | +| CloudTrail Logs | **AWS CloudTrail Logs OpenTelemetry Assets** | Dashboards and visualizations for CloudTrail log data | 4. Once installed, navigate to **Dashboard** to view the pre-built dashboards for your AWS log data. @@ -525,29 +587,26 @@ However, if a timeout occurs, you need to run the custom event multiple times to ## Delete a CloudFormation stack -If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using the following command: +If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using either the AWS CLI or the AWS Console. -```sh -aws cloudformation delete-stack \ - --stack-name \ - --region -``` +### Important considerations -Deleting a stack will remove all AWS resources created by that stack. However, if you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket will not be deleted if it contains objects, because CloudFormation doesn't force-delete non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it. +Deleting a stack will remove all AWS resources created by that stack. However: -If you specified an existing bucket through the `S3FailureBucketARN` parameter instead, that bucket will not be deleted because it is not managed by the stack. +- If you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket will **not** be deleted if it contains objects, because CloudFormation doesn't force-delete non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it. +- If you specified an existing bucket through the `S3FailureBucketARN` parameter, that bucket will **not** be deleted because it is not managed by the stack. -:::{dropdown} Example: Deleting a stack +### Delete using AWS CLI -The following command deletes the `edot-cloud-forwarder-vpc` stack: +Use the following command to delete a stack: ```sh aws cloudformation delete-stack \ - --stack-name edot-cloud-forwarder-vpc \ - --region eu-central-1 + --stack-name \ + --region ``` -You can monitor the deletion progress in the AWS Console under **CloudFormation** → **Stacks**, or through this command: +You can monitor the deletion progress through this command: ```sh aws cloudformation describe-stacks \ @@ -559,10 +618,58 @@ If the stack deletion fails and remains in a `DELETE_FAILED` state, you can retr ```sh aws cloudformation delete-stack \ - --stack-name edot-cloud-forwarder-vpc \ - --region eu-central-1 \ + --stack-name \ + --region \ --deletion-mode FORCE_DELETE_STACK ``` This forcibly deletes the stack's resources, except any that cannot be deleted, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion. + +:::{dropdown} Example: Deleting a stack via AWS CLI + +The following command deletes the `edot-cloud-forwarder-vpc` stack: + +```sh +aws cloudformation delete-stack \ + --stack-name edot-cloud-forwarder-vpc \ + --region eu-central-1 +``` + +Monitor the deletion progress: + +```sh +aws cloudformation describe-stacks \ + --stack-name edot-cloud-forwarder-vpc \ + --region eu-central-1 +``` + ::: + +### Delete using AWS Console + +To delete a stack through the AWS Management Console: + +1. Navigate to **CloudFormation** in the AWS Management Console. +2. Select the stack you want to delete from the list. +3. Click **Delete** at the top of the stack details page. +4. Monitor the deletion progress on the **Events** tab or wait until the stack disappears from the stack list (indicating deletion is complete). + +## Monitoring and Troubleshooting + +### Monitor Lambda performance + +To monitor your {{edot-cf}} Lambda function performance and troubleshoot issues: + +1. **CloudWatch Metrics Explorer**: View Lambda metrics such as: + - Duration + - ConcurrentExecutions + - Errors + - Throttles + - Invocations + +2. **CloudWatch Logs**: Check the Lambda function logs for: + - Processing errors + - Configuration issues + - Data export failures + +The `LambdaLogGroup` resource created by the CloudFormation stack stores all Lambda execution logs. Look for error messages or warnings that indicate configuration or performance issues. From 559ba373444d11012add652311aa15bfd1b9f7ed Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Thu, 6 Nov 2025 19:49:15 +0100 Subject: [PATCH 2/7] Update action --- .github/workflows/vale-linting.yml | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/.github/workflows/vale-linting.yml b/.github/workflows/vale-linting.yml index 14df0b25..7ebb6dad 100644 --- a/.github/workflows/vale-linting.yml +++ b/.github/workflows/vale-linting.yml @@ -1,21 +1,27 @@ name: Vale Documentation Linting on: - pull_request: + pull_request_target: # ← This is the key change + types: [opened, synchronize, reopened] paths: - '**.md' - '**.adoc' +permissions: + contents: read + pull-requests: write + jobs: vale: runs-on: ubuntu-latest steps: - - name: Checkout + - name: Checkout PR branch uses: actions/checkout@v5 with: + ref: ${{ github.event.pull_request.head.sha }} # ← Checkout the PR's code fetch-depth: 0 - name: Run Elastic Vale Linter uses: elastic/vale-rules@main with: - reporter: github-pr-review + github_token: ${{ secrets.GITHUB_TOKEN }} \ No newline at end of file From 4aa4afd3db2e923e6d9c10f73fdd938d0d550647 Mon Sep 17 00:00:00 2001 From: Michalis Katsoulis Date: Fri, 7 Nov 2025 11:50:31 +0200 Subject: [PATCH 3/7] Update --- docs/reference/edot-cloud-forwarder/aws.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index 10117f26..a0fc6a76 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -189,7 +189,7 @@ These are optional settings you can set in the CloudFormation template: | Setting | Description | | ------------------- |--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds (5 minutes). Minimum value is `1` second. Maximum value is `900` seconds. **When to increase**: If you experience Lambda timeouts when processing large log files. Most files process in 1-3 seconds, but larger files may need more time. | +| `EdotCloudForwarderTimeout` | Maximum execution time for the Lambda function, measured in seconds. Default value is `300` seconds (5 minutes). Minimum value is `1` second. Maximum value is `900` seconds. **When to increase**: If you experience Lambda timeouts. The default timeout is sufficient for most log file sizes. | | `EdotCloudForwarderVersion` | Version of the EDOT Cloud Forwarder. Expected format is semantic versioning, for example `0.2.4`. Defaults to the latest available patch version. Don't change this value unless advised by Elastic Support. | | `EdotCloudForwarderMemorySize` | Set the allocated memory for the Lambda function, measured in megabytes. Default value is `512` MB. Minimum value is `128` MB. Maximum value is `10240` MB. **When to increase**: If memory usage consistently exceeds 80% or if you observe Lambda timeouts and processing times increase significantly. More memory also increases CPU, which improves processing speed and reduces timeouts. | | `EdotCloudForwarderConcurrentExecutions` | Set the maximum number of reserved concurrent executions for the Lambda function. Default value is `5`. **When to increase**: If new log files arrive frequently and you observe Lambda processing times consistently exceeding 1-2 seconds, consider increasing concurrent executions. Concurrent Lambda invocations are only spawned when needed (when new files arrive). This allows multiple log files to be processed in parallel instead of sequentially, avoiding processing delays. Make sure this value doesn't exceed your AWS account's concurrency limit. | From ea46e854ef96494c9fa1f2d992fbcf7e152051e1 Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri-Benedetti Date: Fri, 7 Nov 2025 11:18:51 +0100 Subject: [PATCH 4/7] Edits --- docs/reference/edot-cloud-forwarder/aws.md | 68 +++++++++++----------- 1 file changed, 35 insertions(+), 33 deletions(-) diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index a0fc6a76..f10b1722 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -16,16 +16,16 @@ products: # EDOT Cloud Forwarder for AWS -{{edot-cf}} (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to Elastic Observability on {{serverless-full}}. +{{edot-cf}} (CF) for AWS provides the EDOT Collector as a Lambda function that collects and forwards logs to {{product.observability}} on {{serverless-full}}. {{edot-cf}} for AWS supports the following log sources: | AWS Service | Telemetry Description | | --- | --- | -| Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic | -| Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer | -| AWS CloudTrail | [CloudTrail Logs](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html) to record account activity | -% | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch | +| Virtual Private Cloud (VPC) | [VPC Flow Logs](https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html) to capture information about IP traffic. | +| Elastic Load Balancer (ELB) | [Access logs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html) for your Application Load Balancer. | +| AWS CloudTrail | [CloudTrail Logs](https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-log-file-examples.html) to record account activity. | +% | CloudWatch {applies_to}`product: planned` | Logs generated by AWS CloudWatch. | Read on to learn how to set up {{edot-cf}} for AWS. @@ -38,7 +38,7 @@ We are working to support other popular log types and sources. Get in touch to l ::::{important} {{edot-cf}} for AWS requires a Managed OTLP endpoint and an API key. Managed OTLP is available for {{serverless-full}} and will soon be available for {{ech}}. -For self-managed deployments, set up an EDOT Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode) that ingests OTel data from the edge setup into the self-managed Elastic Stack. +For self-managed deployments, set up an EDOT Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode) that ingests OTel data from the edge setup into the self-managed {{stack}}. :::: @@ -100,23 +100,23 @@ Trim the API key from `Authorization=ApiKey MYKEYVALUE...` to just `MYKEYVALUE.. ::: :::: -## Deployment options +## Deployment methods {{edot-cf}} for AWS can be deployed using any of the following methods: | Deployment Method | Description | | --- | --- | -| **CloudFormation (AWS CLI)** | Deploy using AWS CLI commands with CloudFormation templates. | -| **CloudFormation (AWS Console)** | Deploy using the AWS Management Console UI. | +| CloudFormation (AWS CLI) | Deploy using AWS CLI commands with CloudFormation templates. | +| CloudFormation (AWS Console) | Deploy using the AWS Management Console UI. | -Each method achieves the same result and uses CloudFormation templates under the hood. Choose the method that best fits your workflow. +Each method achieves the same result and uses CloudFormation templates. Choose the method that best adapts to your workflow. ## Deployment considerations -Before deploying {{edot-cf}} for AWS, keep these points in mind: +Before deploying {{edot-cf}} for AWS, consider the following actions: - Deploy a separate CloudFormation stack for each log type, for example VPC Flow Logs or ELB Access Logs. Each CloudFormation stack can only process one log type and format at a time. - Logs stored in S3 must be placed in separate buckets. Each log type should reside in its own dedicated bucket. @@ -124,18 +124,20 @@ Before deploying {{edot-cf}} for AWS, keep these points in mind: ## CloudFormation templates -The CloudFormation templates are hosted in a public Amazon S3 bucket and can be accessed via HTTPS URL. You can reference these templates directly during deployment or download them for local use. +The CloudFormation templates are hosted in a public Amazon S3 bucket and can be accessed through HTTPS URL. You can reference these templates directly during deployment or download them for local use. -| Log Source | CloudFormation template | -| --- | ------------------------------------------------ | -| S3 logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | +| Log type | Log source | CloudFormation template | +| --- | --- | ------------------------------------------------ | +| VPC | S3 | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | +| ELB | S3 | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | +| CloudTrail | S3 | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/s3_logs-cloudformation.yaml` | % | CloudWatch logs | `https://edot-cloud-forwarder.s3.amazonaws.com/v0/latest/cloudformation/cloudwatch_logs-cloudformation.yaml` | For specific versions, edit `latest` in the URL to the required version in the format `vX.Y.Z`. ## Configure the template -Before deploying {{edot-cf}} for AWS, configure the CloudFormation template parameters based on your specific requirements. The template uses settings below to deploy and configure the EDOT Collector Lambda function. +Before deploying {{edot-cf}} for AWS, configure the CloudFormation template parameters based on your specific requirements. The template uses the following settings to deploy and configure the EDOT Collector Lambda function. ### Required settings @@ -155,7 +157,7 @@ Set the following settings based on the log source: ::::{tab-item} S3 -For S3 logs, use the following settings: +For logs sourced from S3, use the following settings: | Setting | Description | | ------------------ | --- | @@ -168,7 +170,7 @@ For S3 logs, use the following settings: + +## Datastreams + +Logs collected by {{edot-cf}} for AWS are stored in Elasticsearch datastreams in OpenTelemetry native format. The following table shows which datastreams are used for each log type: + +| **AWS Log Type** | **Datastream Dataset** | **Description** | +|------------------|------------------------|-----------------| +| VPC Flow Logs | `aws.vpcflow.otel` | VPC Flow Log records | +| ELB Access Logs | `aws.elbaccess.otel` | ELB Access Log records (ALB, NLB, CLB) | + +The logs are produced in OpenTelemetry native format. For detailed information about the field mappings and structure of each log type, refer to the following documentation: + +- **VPC Flow Logs**: See [VPC Flow Log record fields](https://github.com/occamshub-dev/opentelemetry-collector-contrib/blob/main/extension/encoding/awslogsencodingextension/README.md#vpc-flow-log-record-fields) for the complete field mapping. +- **ELB Access Logs**: See [ELB Access Log fields](https://github.com/occamshub-dev/opentelemetry-collector-contrib/blob/main/extension/encoding/awslogsencodingextension/README.md#elb-access-log-fields) for the complete field mapping. + ## Kibana integration setup After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations. From 8038a1350b38e2e3944b3f4613a54108fae6b2bf Mon Sep 17 00:00:00 2001 From: Fabrizio Ferri Benedetti Date: Tue, 11 Nov 2025 15:31:28 +0100 Subject: [PATCH 7/7] Writer edits --- .../_snippets/find-motlp-endpoint.md | 20 ++++++ docs/reference/edot-cloud-forwarder/aws.md | 70 +++++++++---------- docs/reference/edot-cloud-forwarder/azure.md | 2 +- docs/reference/motlp.md | 20 +----- 4 files changed, 55 insertions(+), 57 deletions(-) create mode 100644 docs/reference/_snippets/find-motlp-endpoint.md diff --git a/docs/reference/_snippets/find-motlp-endpoint.md b/docs/reference/_snippets/find-motlp-endpoint.md new file mode 100644 index 00000000..db25e6c5 --- /dev/null +++ b/docs/reference/_snippets/find-motlp-endpoint.md @@ -0,0 +1,20 @@ +To retrieve your {{motlp}} endpoint address and API key, follow these steps: + +::::{applies-switch} +:::{applies-item} serverless: +1. In {{ecloud}}, create an Observability project or open an existing one. +2. Go to **Add data**, select **Applications** and then select **OpenTelemetry**. +3. Copy the endpoint and authentication headers values. + +Alternatively, you can retrieve the endpoint from the **Manage project** page and create an API key manually from the **API keys** page. +::: + +:::{applies-item} ess: +{applies_to}`stack: preview 9.2` +1. In {{ecloud}}, create an {{ech}} deployment or open an existing one. +2. Go to **Add data**, select **Applications** and then select **OpenTelemetry**. +3. Copy the endpoint and authentication headers values. + +Alternatively, you can retrieve the endpoint from the **Manage project** page and create an API key manually from the **API keys** page. +::: +:::: \ No newline at end of file diff --git a/docs/reference/edot-cloud-forwarder/aws.md b/docs/reference/edot-cloud-forwarder/aws.md index 798adbd9..46037f75 100644 --- a/docs/reference/edot-cloud-forwarder/aws.md +++ b/docs/reference/edot-cloud-forwarder/aws.md @@ -4,8 +4,9 @@ description: Set up the EDOT Cloud Forwarder for AWS to bring your AWS logs to E applies_to: serverless: observability: preview -# deployment: -# ess: preview + deployment: + ess: preview + self: unavailable product: edot_cf_aws: preview products: @@ -29,19 +30,16 @@ products: Read on to learn how to set up {{edot-cf}} for AWS. -::::{note} -We are working to support other popular log types and sources. Get in touch to let us know of any specific requirements that could influence our plans. -:::: +:::{note} +We are working to support other popular log types and sources. [Contact us](docs-content://troubleshoot/ingest/opentelemetry/contact-support.md) to let us know of any specific requirements that could influence our plans. +::: ## Prerequisites ::::{important} -{{edot-cf}} for AWS requires a Managed OTLP endpoint and an API key. Managed OTLP is available for {{serverless-full}} and will soon be available for {{ech}}. - -For self-managed deployments, set up an EDOT Collector in [Gateway mode](elastic-agent://reference/edot-collector/config/default-config-standalone.md#gateway-mode) that ingests OTel data from the edge setup into the self-managed {{stack}}. +{{edot-cf}} for AWS requires a Managed OTLP endpoint and an API key. Managed OTLP is available for {{serverless-full}} and {{ech}}. :::: - To collect logs using {{edot-cf}} for AWS, you need: ::::{tab-set} @@ -86,11 +84,9 @@ To collect CloudWatch logs, you need: --> :::: -In addition, you need to know the URL of the managed OTLP endpoint and the API key for authentication. - -::::{dropdown} Steps to retrieve the OTLP endpoint and API key +You also need to know the URL of the managed OTLP endpoint and the API key for authentication. -:::{include} ../_snippets/serverless-endpoint-api.md +:::{include} ../_snippets/find-motlp-endpoint.md ::: In the CloudFormation templates, the OTLP endpoint is set as `OTLPEndpoint`, and the API key is set as `ElasticAPIKey`. @@ -199,7 +195,9 @@ These are optional settings you can set in the CloudFormation template: The default values provided have been determined through extensive load testing across different log types and data volumes. For most use cases, these defaults provide a good balance between cost and performance. -Adjust these parameters only if you notice performance issues such as Lambda timeouts, throttling, high memory usage or data being dropped. If you need assistance tuning these parameters for your specific workload, contact Elastic Support. +:::{tip} +Adjust these parameters only if you notice performance issues such as Lambda timeouts, throttling, high memory usage or dropped data. If you need assistance tuning these parameters for your specific workload, refer to [Contact support](docs-content://troubleshoot/ingest/opentelemetry/contact-support.md). +::: ## Deploy using CloudFormation (AWS CLI) @@ -537,7 +535,7 @@ CloudWatch Log Groups help monitor execution performance and debug issues. IAM p ## Datastreams -Logs collected by {{edot-cf}} for AWS are stored in Elasticsearch datastreams in OpenTelemetry native format. The following table shows which datastreams are used for each log type: +Logs collected by {{edot-cf}} for AWS are stored in {{es}} datastreams in OpenTelemetry native format. The following table shows which datastreams are used for each log type: | **AWS Log Type** | **Datastream Dataset** | **Description** | |------------------|------------------------|-----------------| @@ -551,12 +549,12 @@ The logs are produced in OpenTelemetry native format. For detailed information a ## Kibana integration setup -After {{edot-cf}} for AWS is successfully running and forwarding logs to Elastic Observability, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations. +After {{edot-cf}} for AWS is successfully running and forwarding logs to {{product.observability}}, install the {{kib}} integrations to visualize your data with out-of-the-box dashboards and visualizations. To set up data visualization in {{kib}}: -1.Log into your Elastic Cloud deployment and open Kibana. -2. Go to **Management** → **Integrations** in the Kibana navigation menu. +1.Log into your {{ecloud}} deployment and open {{kib}} +2. Go to **Management** → **Integrations** in the {{kib}} navigation menu. 3. Search for the appropriate integration based on your log type and install it: | **AWS Log Type** | **Integration Name** | **Description** | @@ -602,23 +600,23 @@ With AWS CLI, you can use `--timeout` to increase currently configured Lambda ti However, if a timeout occurs, you need to run the custom event multiple times to fully process all error events from the bucket. ::: -## Delete a CloudFormation stack +## Remove a CloudFormation stack -If you no longer need a deployed stack and want to clean up all associated resources, you can delete it using either the AWS CLI or the AWS Console. +If you no longer need a deployed stack and want to clean up all associated resources, you can remove it using either the AWS CLI or the AWS Console. ### Important considerations -Deleting a stack will remove all AWS resources created by that stack. However: +Deleting a stack removes all AWS resources created by that stack. However: -- If you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket will not be deleted if it contains objects, because CloudFormation doesn't force-delete non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it. -- If you specified an existing bucket through the `S3FailureBucketARN` parameter, that bucket will not be deleted because it is not managed by the stack. +- If you allowed the stack to automatically create a dedicated S3 bucket for failed Lambda invocations, that bucket is not removed if it contains objects, because CloudFormation doesn't force-remove non-empty buckets. To remove the bucket entirely, you must empty it manually before deleting it. +- If you specified an existing bucket through the `S3FailureBucketARN` parameter, that bucket is not removed because it is not managed by the stack. -### Delete using AWS CLI +### Remove using AWS CLI -Use the following command to delete a stack: +Use the following command to remove a stack: ```sh -aws cloudformation delete-stack \ +aws cloudformation remove-stack \ --stack-name \ --region ``` @@ -634,20 +632,20 @@ aws cloudformation describe-stacks \ If the stack deletion fails and remains in a `DELETE_FAILED` state, you can retry the deletion with force mode: ```sh -aws cloudformation delete-stack \ +aws cloudformation remove-stack \ --stack-name \ --region \ --deletion-mode FORCE_DELETE_STACK ``` -This forcibly deletes the stack's resources, except any that cannot be deleted, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion. +This forcibly removes the stack's resources, except any that cannot be removed, like the failure S3 bucket if it still contains objects. For a complete cleanup, empty the bucket manually before retrying deletion. :::{dropdown} Example: Deleting a stack using AWS CLI -The following command deletes the `edot-cloud-forwarder-vpc` stack: +The following command removes the `edot-cloud-forwarder-vpc` stack: ```sh -aws cloudformation delete-stack \ +aws cloudformation remove-stack \ --stack-name edot-cloud-forwarder-vpc \ --region eu-central-1 ``` @@ -662,18 +660,16 @@ aws cloudformation describe-stacks \ ::: -### Delete using AWS Console +### Remove using AWS Console -To delete a stack using the AWS Management Console: +To remove a stack using the AWS Management Console: 1. Navigate to **CloudFormation** in the AWS Management Console. -2. Select the stack you want to delete from the list. -3. Click **Delete** at the top of the stack details page. +2. Select the stack you want to remove from the list. +3. Click **Remove** at the top of the stack details page. 4. Monitor the deletion progress on the **Events** tab or wait until the stack disappears from the stack list (indicating deletion is complete). -## Monitoring and Troubleshooting - -### Monitor Lambda performance +## Monitoring and troubleshooting To monitor your {{edot-cf}} Lambda function performance and troubleshoot issues: diff --git a/docs/reference/edot-cloud-forwarder/azure.md b/docs/reference/edot-cloud-forwarder/azure.md index e917deae..2a87c75c 100644 --- a/docs/reference/edot-cloud-forwarder/azure.md +++ b/docs/reference/edot-cloud-forwarder/azure.md @@ -89,7 +89,7 @@ To find out the URL of the managed OTLP endpoint and the API key for authenticat ::::{dropdown} Steps to retrieve the OTLP endpoint and API key -:::{include} ../_snippets/serverless-endpoint-api.md +:::{include} ../_snippets/find-motlp-endpoint.md ::: In the Bicep templates, the OTLP endpoint is set as `elasticsearchOtlpEndpoint`, and the API key is set as `elasticsearchApiKey`. diff --git a/docs/reference/motlp.md b/docs/reference/motlp.md index cf432316..70b7444d 100644 --- a/docs/reference/motlp.md +++ b/docs/reference/motlp.md @@ -57,25 +57,7 @@ To send data to Elastic through the {{motlp}}, follow the [Send data to the Elas ### Find your {{motlp}} endpoint -To retrieve your {{motlp}} endpoint address, follow these steps: - -::::{applies-switch} -:::{applies-item} serverless: -1. In {{ecloud}}, select the project name and then select **Manage project**. -2. Locate the **Connection alias** and select **Edit**. -3. Copy the Managed OTLP endpoint URL. -::: - -:::{applies-item} ess: -{applies_to}`stack: preview 9.2` -1. In {{ecloud}}, select **Add data** and then select **Application: OpenTelemetry** -2. Copy the Managed OTLP endpoint URL from `OTEL_EXPORTER_OTLP_ENDPOINT`. You can use this endpoint for both OpenTelemetry collectors and SDKs. -::: -:::: - -:::{note} -:applies_to: { ess:, stack: preview 9.2 } -The Managed OTLP endpoint might not be available in all {{ech}} regions during the Technical Preview. +:::{include} _snippets/find-motlp-endpoint.md ::: ### Configure SDKs to send data directly