|
| 1 | +--- |
| 2 | +navigation_title: GCP |
| 3 | +description: Set up the EDOT Cloud Forwarder for GCP to bring your GCP logs to Elastic Observability. |
| 4 | +applies_to: |
| 5 | + serverless: |
| 6 | + observability: preview |
| 7 | + deployment: |
| 8 | + ess: preview |
| 9 | + product: |
| 10 | + edot_cf_gcp: preview |
| 11 | +products: |
| 12 | + - id: cloud-serverless |
| 13 | + - id: observability |
| 14 | + - id: edot-cf |
| 15 | +--- |
| 16 | + |
| 17 | +# EDOT Cloud Forwarder for GCP |
| 18 | + |
| 19 | +{{edot-cf}} for GCP provides a serverless, scalable way to ingest Google Cloud Platform logs into Elastic. It deploys the EDOT Collector as a Google Cloud Run service that listens for Pub/Sub push subscriptions, processes the logs, and forwards them to {{motlp}}. |
| 20 | + |
| 21 | +## Architecture overview |
| 22 | + |
| 23 | +The architecture for the {{edot-cf}} GCP is as pictured: |
| 24 | + |
| 25 | + |
| 26 | + |
| 27 | +### Data flow |
| 28 | + |
| 29 | +- Ingestion: Logs are sent to a Pub/Sub topic (either directly or via GCS Bucket notifications). |
| 30 | +- Processing: A push subscription triggers the Cloud Run service, where the {{edot-cf}} is running. |
| 31 | +- Forwarding: The service processes the data and exports it to Elastic via the {{motlp}}. |
| 32 | +- Failure Handling: If processing fails (after retries), messages are routed to a dead letter topic and archived in a GCS Bucket for future analysis. |
| 33 | + |
| 34 | +## Supported log types |
| 35 | + |
| 36 | +Currently, {{edot-cf}} for GCP supports the following log types: |
| 37 | + |
| 38 | +% TODO constanca-m MAYBE add permalink to otel version we will use in ECF GCP image in the table |
| 39 | + |
| 40 | +| Log | OTel mapping | |
| 41 | +|-----------------|-----------------| |
| 42 | +| Cloud Audit Log | Cloud Audit Log | |
| 43 | +| VPC Flow Log | Access logs | |
| 44 | + |
| 45 | +:::{note} |
| 46 | +We are working to support other popular log types and sources. [Contact us](docs-content://troubleshoot/ingest/opentelemetry/contact-support.md) to let us know of any specific requirements that could influence our plans. |
| 47 | +::: |
| 48 | + |
| 49 | + |
| 50 | +## Prerequisites |
| 51 | + |
| 52 | +### Elastic requirements |
| 53 | + |
| 54 | +- Access to {{motlp}} endpoint. |
| 55 | +- Valid API key with ingest permissions. |
| 56 | + |
| 57 | +You can refer to [Send data to Elastic](../motlp.md#send-data-to-elastic) documentation for more details. |
| 58 | + |
| 59 | + |
| 60 | +### GCP permissions |
| 61 | + |
| 62 | +% TODO constanca-m joecompute https://github.com/elastic/edot-cloud-forwarder-gcp/pull/225 |
| 63 | + |
| 64 | + |
| 65 | +## Quick start |
| 66 | + |
| 67 | +% TODO Publish https://github.com/elastic/terraform-google-edot-cloud-forwarder on terraform public registry |
| 68 | + |
| 69 | + |
| 70 | +## Features |
| 71 | + |
| 72 | +The {{edot-cf}} is designed for reliability and observability. |
| 73 | + |
| 74 | +### Flexible ingestion |
| 75 | + |
| 76 | +Logs can be sent: |
| 77 | + |
| 78 | +- Directly to a Pub/Sub topic. |
| 79 | +- To a file placed in a GCS bucket. This will trigger an event notification to Pub/Sub which in turn will trigger the {{edot-cf}}. |
| 80 | + |
| 81 | +### Reliability & Recovery |
| 82 | + |
| 83 | +- Dead letter queue (DLQ): If a log entry fails to process or send to Elastic after the configured retries, it is not lost. The |
| 84 | + {{edot-cf}} automatically routes failed messages to a dedicated GCS bucket for later analysis. |
| 85 | +- Smart retries: Built-in exponential backoff for transient network issues. |
| 86 | + |
| 87 | +### Observability & Metadata |
| 88 | + |
| 89 | +- Self-telemetry: You can configure the collector to send its own internal telemetry to the {{motlp}}. |
| 90 | +- Enrich metadata: You can enable `include_metadata` to enrich your logs with context from the transport layer, including: |
| 91 | + - `bucket` |
| 92 | + - `object` |
| 93 | + - `subscription` |
| 94 | + - `message_id` |
| 95 | + - `delivery_attempt` |
| 96 | + |
| 97 | +## Performance |
| 98 | + |
| 99 | +% TODO |
| 100 | + |
| 101 | +## Limitations |
| 102 | + |
| 103 | +% TODO Add information on: permanent errors don't stop the retries |
| 104 | + |
| 105 | +## Changelog |
| 106 | + |
| 107 | +% How to link the CHANGELOG.md file if it is in a private repository? |
| 108 | +% https://github.com/elastic/edot-cloud-forwarder-gcp/blob/main/CHANGELOG.md |
| 109 | + |
0 commit comments