You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/edot-cloud-forwarder/gcp.md
+37-38Lines changed: 37 additions & 38 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -59,21 +59,21 @@ You can refer to [Send data to Elastic](../motlp.md#send-data-to-elastic) docume
59
59
60
60
You should have the following permissions on your Google Cloud project:
61
61
62
-
<DocAccordionbuttonContent="Project IAM Admin"initialIsOpen>
62
+
:::{dropdown} Project IAM Admin
63
63
The principal should be granted the built-in `roles/resourcemanager.projectIamAdmin` role, allowing them to manage IAM policies and roles at the project level.
The {{edot-cf}} is designed for reliability and observability.
156
+
The {{edot-cf}} is engineered for high-throughput, reliable ingestion, and simplified observability.
152
157
153
158
### Flexible ingestion
154
159
155
-
Logs can be sent:
156
-
157
-
- Directly to a Pub/Sub topic.
158
-
- To a file placed in a GCS bucket. This will trigger an event notification to Pub/Sub which in turn will trigger the {{edot-cf}}.
160
+
The {{edot-cf}} supports two primary event-driven ingestion patterns on GCP:
161
+
- Direct Pub/Sub: Ideal for logs streamed directly to a Pub/Sub topic by custom applications or other GCP services.
162
+
- GCS file notifications: Automatically ingests batch logs (like VPC Flow Logs or Audit Logs) placed in a file into a Google Cloud Storage bucket. The system listens for the `OBJECT_FINALIZE` event, reads the file content, and processes it.
159
163
160
-
### Reliability & Recovery
164
+
### Reliability
161
165
162
-
- Dead letter queue (DLQ): If a log entry fails to process or send to Elastic after the configured retries, it is not lost. The
163
-
{{edot-cf}} automatically routes failed messages to a dedicated GCS bucket for later analysis.
164
-
- Smart retries: Built-in exponential backoff for transient network issues.
166
+
Reliability is built-in to prevent data loss or infinite retry loops.
167
+
- Message acknowledgment: The service only acknowledges (ACKs) a Pub/Sub message upon successful forwarding to Elastic, ensuring that failed messages are automatically placed back in the queue for retry (or sent to the dead letter topic).
168
+
- Smart retries: The underlying Pub/Sub subscription is configured with exponential backoff. This prevents overwhelming the service with repeated failed messages during transient issues like network instability.
169
+
- Dead letter topic and failure bucket: If a message fails to be processed or forwarded after the configured maximum number of attempts, the {{edot-cf}} guarantees the message is sent to the dead letter topic. Messages sent to the dead letter topic are later archived in a dedicated GCS bucket. This prevents data loss and allows for later inspection.
165
170
166
-
### Observability & Metadata
167
171
168
-
- Self-telemetry: You can configure the collector to send its own internal telemetry to the {{motlp}}.
169
-
- Enrich metadata: You can enable `include_metadata` to enrich your logs with context from the transport layer, including:
170
-
-`bucket`
171
-
-`object`
172
-
-`subscription`
173
-
-`message_id`
174
-
-`delivery_attempt`
172
+
### Observability and data enrichment
175
173
176
-
## Performance
174
+
The {{edot-cf}} provides detailed context about its own health and the data it processes.
175
+
- Self-telemetry: You can enable the OpenTelemetry collector's internal metrics, allowing you to monitor the service's health.
176
+
- Metadata enrichment: By enabling the `include_metadata` option, logs are automatically enriched with context from the Pub/Sub and GCS transport layers, enabling better troubleshooting and correlation:
177
+
-`bucket` and `object`, for logs coming from a GCS bucket.
178
+
-`subscription` and `message_id`.
179
+
- delivery_attempt, useful for tracking retries.
177
180
181
+
% Best effort, maybe not present for Tech Preview
182
+
% ## Performance
178
183
% TODO
179
184
180
185
## Limitations
181
186
182
187
The current retry logic treats all failures the same way, whether they're temporary or permanent errors like an invalid log format. This means a message that can't ever be processed correctly will still go through all configured retries before finally being sent to the dead-letter topic and archived in the GCS bucket. While this ensures resilience against transient failures, it does mean you might incur unnecessary processing costs for messages that were never going to succeed.
183
-
184
-
## Changelog
185
-
186
-
% How to link the CHANGELOG.md file if it is in a private repository?
0 commit comments