Skip to content

Commit 6201f02

Browse files
committed
fix the dropdown, fix the features sections
1 parent f3150a4 commit 6201f02

File tree

1 file changed

+37
-38
lines changed
  • docs/reference/edot-cloud-forwarder

1 file changed

+37
-38
lines changed

docs/reference/edot-cloud-forwarder/gcp.md

Lines changed: 37 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -59,21 +59,21 @@ You can refer to [Send data to Elastic](../motlp.md#send-data-to-elastic) docume
5959

6060
You should have the following permissions on your Google Cloud project:
6161

62-
<DocAccordion buttonContent="Project IAM Admin" initialIsOpen>
62+
:::{dropdown} Project IAM Admin
6363
The principal should be granted the built-in `roles/resourcemanager.projectIamAdmin` role, allowing them to manage IAM policies and roles at the project level.
64-
</DocAccordion>
64+
:::
6565

66-
<DocAccordion buttonContent="Storage" initialIsOpen>
66+
:::{dropdown} Storage
6767
The following permissions are needed for Cloud Storage management:
6868
- `storage.buckets.create`
6969
- `storage.buckets.delete`
7070
- `storage.buckets.get`
7171
- `storage.buckets.getIamPolicy`
7272
- `storage.buckets.setIamPolicy`
7373
- `storage.buckets.update`
74-
</DocAccordion>
74+
:::
7575

76-
<DocAccordion buttonContent="Secret Manager" initialIsOpen>
76+
:::{dropdown} Secret Manager
7777
The following permissions are needed for Secret Manager management:
7878
- `secretmanager.secrets.create`
7979
- `secretmanager.secrets.delete`
@@ -86,9 +86,9 @@ The following permissions are needed for Secret Manager management:
8686
- `secretmanager.versions.destroy`
8787
- `secretmanager.versions.enable`
8888
- `secretmanager.versions.get`
89-
</DocAccordion>
89+
:::
9090

91-
<DocAccordion buttonContent="Pub/Sub" initialIsOpen>
91+
:::{dropdown} Pub/Sub
9292
The following permissions are needed for Pub/Sub management:
9393
- `pubsub.subscriptions.create`
9494
- `pubsub.subscriptions.delete`
@@ -105,9 +105,9 @@ The following permissions are needed for Pub/Sub management:
105105
- `pubsub.topics.getIamPolicy`
106106
- `pubsub.topics.setIamPolicy`
107107
- `pubsub.topics.update`
108-
</DocAccordion>
108+
:::
109109

110-
<DocAccordion buttonContent="Cloud Run" initialIsOpen>
110+
:::{dropdown} Cloud Run
111111
The following permissions are needed for Cloud Run management:
112112
- `run.operations.get`
113113
- `run.services.create`
@@ -116,9 +116,9 @@ The following permissions are needed for Cloud Run management:
116116
- `run.services.getIamPolicy`
117117
- `run.services.setIamPolicy`
118118
- `run.services.update`
119-
</DocAccordion>
119+
:::
120120

121-
<DocAccordion buttonContent="Service Account" initialIsOpen>
121+
:::{dropdown} Service Account
122122
The following permissions are needed for Service Account management:
123123
- `iam.serviceAccountKeys.create`
124124
- `iam.serviceAccountKeys.get`
@@ -127,9 +127,9 @@ The following permissions are needed for Service Account management:
127127
- `iam.serviceAccounts.get`
128128
- `iam.serviceAccounts.update`
129129
- `iam.serviceAccounts.actAs`
130-
</DocAccordion>
130+
:::
131131

132-
<DocAccordion buttonContent="Artifact Registry" initialIsOpen>
132+
:::{dropdown} Artifact Registry
133133
The following permissions are needed:
134134
- `artifactregistry.repositories.create`
135135
- `artifactregistry.repositories.delete`
@@ -138,51 +138,50 @@ The following permissions are needed:
138138
- `artifactregistry.repositories.setIamPolicy`
139139
- `artifactregistry.repositories.update`
140140
- `artifactregistry.repositories.downloadArtifacts`
141-
</DocAccordion>
141+
:::
142142

143143

144144
## Quick start
145145

146+
You can deploy {{edot-cf}} for GCP using the Terraform module...
147+
148+
149+
146150
% TODO Publish https://github.com/elastic/terraform-google-edot-cloud-forwarder on terraform public registry
151+
% Issue: https://elasticco.atlassian.net/browse/ENGPRD-1866
147152

148153

149154
## Features
150155

151-
The {{edot-cf}} is designed for reliability and observability.
156+
The {{edot-cf}} is engineered for high-throughput, reliable ingestion, and simplified observability.
152157

153158
### Flexible ingestion
154159

155-
Logs can be sent:
156-
157-
- Directly to a Pub/Sub topic.
158-
- To a file placed in a GCS bucket. This will trigger an event notification to Pub/Sub which in turn will trigger the {{edot-cf}}.
160+
The {{edot-cf}} supports two primary event-driven ingestion patterns on GCP:
161+
- Direct Pub/Sub: Ideal for logs streamed directly to a Pub/Sub topic by custom applications or other GCP services.
162+
- GCS file notifications: Automatically ingests batch logs (like VPC Flow Logs or Audit Logs) placed in a file into a Google Cloud Storage bucket. The system listens for the `OBJECT_FINALIZE` event, reads the file content, and processes it.
159163

160-
### Reliability & Recovery
164+
### Reliability
161165

162-
- Dead letter queue (DLQ): If a log entry fails to process or send to Elastic after the configured retries, it is not lost. The
163-
{{edot-cf}} automatically routes failed messages to a dedicated GCS bucket for later analysis.
164-
- Smart retries: Built-in exponential backoff for transient network issues.
166+
Reliability is built-in to prevent data loss or infinite retry loops.
167+
- Message acknowledgment: The service only acknowledges (ACKs) a Pub/Sub message upon successful forwarding to Elastic, ensuring that failed messages are automatically placed back in the queue for retry (or sent to the dead letter topic).
168+
- Smart retries: The underlying Pub/Sub subscription is configured with exponential backoff. This prevents overwhelming the service with repeated failed messages during transient issues like network instability.
169+
- Dead letter topic and failure bucket: If a message fails to be processed or forwarded after the configured maximum number of attempts, the {{edot-cf}} guarantees the message is sent to the dead letter topic. Messages sent to the dead letter topic are later archived in a dedicated GCS bucket. This prevents data loss and allows for later inspection.
165170

166-
### Observability & Metadata
167171

168-
- Self-telemetry: You can configure the collector to send its own internal telemetry to the {{motlp}}.
169-
- Enrich metadata: You can enable `include_metadata` to enrich your logs with context from the transport layer, including:
170-
- `bucket`
171-
- `object`
172-
- `subscription`
173-
- `message_id`
174-
- `delivery_attempt`
172+
### Observability and data enrichment
175173

176-
## Performance
174+
The {{edot-cf}} provides detailed context about its own health and the data it processes.
175+
- Self-telemetry: You can enable the OpenTelemetry collector's internal metrics, allowing you to monitor the service's health.
176+
- Metadata enrichment: By enabling the `include_metadata` option, logs are automatically enriched with context from the Pub/Sub and GCS transport layers, enabling better troubleshooting and correlation:
177+
- `bucket` and `object`, for logs coming from a GCS bucket.
178+
- `subscription` and `message_id`.
179+
- delivery_attempt, useful for tracking retries.
177180

181+
% Best effort, maybe not present for Tech Preview
182+
% ## Performance
178183
% TODO
179184

180185
## Limitations
181186

182187
The current retry logic treats all failures the same way, whether they're temporary or permanent errors like an invalid log format. This means a message that can't ever be processed correctly will still go through all configured retries before finally being sent to the dead-letter topic and archived in the GCS bucket. While this ensures resilience against transient failures, it does mean you might incur unnecessary processing costs for messages that were never going to succeed.
183-
184-
## Changelog
185-
186-
% How to link the CHANGELOG.md file if it is in a private repository?
187-
% https://github.com/elastic/edot-cloud-forwarder-gcp/blob/main/CHANGELOG.md
188-

0 commit comments

Comments
 (0)