Skip to content

Commit f129ecf

Browse files
committed
[ECF GCP] Add documentation for tech preview
1 parent af2950f commit f129ecf

File tree

2 files changed

+964
-0
lines changed

2 files changed

+964
-0
lines changed
Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
---
2+
navigation_title: GCP
3+
description: Set up the EDOT Cloud Forwarder for GCP to bring your GCP logs to Elastic Observability.
4+
applies_to:
5+
serverless:
6+
observability: preview
7+
deployment:
8+
ess: preview
9+
product:
10+
edot_cf_gcp: preview
11+
products:
12+
- id: cloud-serverless
13+
- id: observability
14+
- id: edot-cf
15+
---
16+
17+
# EDOT Cloud Forwarder for GCP
18+
19+
{{edot-cf}} for GCP provides a serverless, scalable way to ingest Google Cloud Platform logs into Elastic. It deploys the EDOT Collector as a Google Cloud Run service that listens for Pub/Sub push subscriptions, processes the logs, and forwards them to {{motlp}}.
20+
21+
## Architecture overview
22+
23+
The architecture for the {{edot-cf}} GCP is as pictured:
24+
25+
![EDOT Cloud Forwarder GCP overview](../images/edot-cloud-forwarder-gcp-overview.svg)
26+
27+
### Data flow
28+
29+
- Ingestion: Logs are sent to a Pub/Sub topic (either directly or via GCS Bucket notifications).
30+
- Processing: A push subscription triggers the Cloud Run service, where the {{edot-cf}} is running.
31+
- Forwarding: The service processes the data and exports it to Elastic via the {{motlp}}.
32+
- Failure Handling: If processing fails (after retries), messages are routed to a dead letter topic and archived in a GCS Bucket for future analysis.
33+
34+
## Supported log types
35+
36+
Currently, {{edot-cf}} for GCP supports the following log types:
37+
38+
% TODO constanca-m MAYBE add permalink to otel version we will use in ECF GCP image in the table
39+
40+
| Log | OTel mapping |
41+
|-----------------|-----------------|
42+
| Cloud Audit Log | Cloud Audit Log |
43+
| VPC Flow Log | Access logs |
44+
45+
:::{note}
46+
We are working to support other popular log types and sources. [Contact us](docs-content://troubleshoot/ingest/opentelemetry/contact-support.md) to let us know of any specific requirements that could influence our plans.
47+
:::
48+
49+
50+
## Prerequisites
51+
52+
### Elastic requirements
53+
54+
- Access to {{motlp}} endpoint.
55+
- Valid API key with ingest permissions.
56+
57+
You can refer to [Send data to Elastic](../motlp.md#send-data-to-elastic) documentation for more details.
58+
59+
60+
### GCP permissions
61+
62+
% TODO constanca-m joecompute https://github.com/elastic/edot-cloud-forwarder-gcp/pull/225
63+
64+
65+
## Quick start
66+
67+
% TODO Publish https://github.com/elastic/terraform-google-edot-cloud-forwarder on terraform public registry
68+
69+
70+
## Features
71+
72+
The {{edot-cf}} is designed for reliability and observability.
73+
74+
### Flexible ingestion
75+
76+
Logs can be sent:
77+
78+
- Directly to a Pub/Sub topic.
79+
- To a file placed in a GCS bucket. This will trigger an event notification to Pub/Sub which in turn will trigger the {{edot-cf}}.
80+
81+
### Reliability & Recovery
82+
83+
- Dead letter queue (DLQ): If a log entry fails to process or send to Elastic after the configured retries, it is not lost. The
84+
{{edot-cf}} automatically routes failed messages to a dedicated GCS bucket for later analysis.
85+
- Smart retries: Built-in exponential backoff for transient network issues.
86+
87+
### Observability & Metadata
88+
89+
- Self-telemetry: You can configure the collector to send its own internal telemetry to the {{motlp}}.
90+
- Enrich metadata: You can enable `include_metadata` to enrich your logs with context from the transport layer, including:
91+
- `bucket`
92+
- `object`
93+
- `subscription`
94+
- `message_id`
95+
- `delivery_attempt`
96+
97+
## Performance
98+
99+
% TODO
100+
101+
## Limitations
102+
103+
% TODO Add information on: permanent errors don't stop the retries
104+
105+
## Changelog
106+
107+
% How to link the CHANGELOG.md file if it is in a private repository?
108+
% https://github.com/elastic/edot-cloud-forwarder-gcp/blob/main/CHANGELOG.md
109+

0 commit comments

Comments
 (0)