You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/architecture/hosts_vms.md
+23-32Lines changed: 23 additions & 32 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -13,58 +13,49 @@ products:
13
13
14
14
# Hosts and VMs environments
15
15
16
-
On host or virtual machine environments, deploy local, per-host OpenTelemetry Collector instances, here referred to as OTel Collector in Agent Mode.
16
+
On host or virtual machine environments, deploy local, per-host OpenTelemetry Collector instances at these [edge](index.md#understanding-edge-deployment) host or VMs where your applications are running.
17
17
18
-
19
-
20
-
These collectors have two main purposes:
18
+
Collectors deployed on these edge environments have two main purposes:
21
19
22
20
1. The collection of local logs and infrastructure metrics. Refer to [this sample config file](https://github.com/elastic/elastic-agent/blob/main/internal/pkg/otel/samples/linux/managed_otlp/platformlogs_hostmetrics.yml) for recommended Collector receiver configurations for hostmetrics and logs.
23
-
2. Enriching application telemetry from OTel SDKs that run within the instrumented applications on corresponding hosts with resource information.
24
-
25
-
## Deployment scenarios
26
-
27
-
See the recommended architectures per Elastic deployment scenarios:
21
+
2. Enriching application telemetry from OTel SDKs with the corresponding hosts resource attributes.
28
22
29
23
:::{note}
30
-
Elastic's Observability solution is technically compatible with edge setups that are fully based on contrib OTel components as long as the ingestion path follows the recommendations outlined in the following sections.
24
+
Elastic Observability is technically compatible with edge setups that are fully based on contrib OTel components as long as the ingestion path follows the recommendations outlined in the following sections.
31
25
:::
32
26
33
-
### Elastic Cloud Serverless
34
-
35
-
{{serverless-full}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data.
Users can send their OTel data from the edge setup in OTel-native format through OTLP without any additional requirements for self-managed preprocessing of data.
29
+
The following sections outline the recommended architectures for different Elastic deployment scenarios.
40
30
41
-
### Elastic Cloud Hosted
31
+
### {{serverless-full}}
42
32
43
-
As of Elastic Stack version 9.0 on {{ech}} (ECH), you need to run a self-hosted EDOT Collector in Gateway Mode to ingest OTel data from the edge setup in OTel-native format into the Elastic-hosted {{es}}.
33
+
{{serverless-full}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data.
44
34
45
-
35
+
46
36
47
-
The EDOT Collector in Gateway mode enriches and pre-aggregates the data for a seamless experience in the Elastic Observability solution before ingesting it directly into {{es}}.
37
+
Users can send data direct from the Collectors or SDKs deployed on the edge environment through OTLP without any additional requirements for managing an ingestion layer.
48
38
49
-
If required, users can build their custom, EDOT-like Collector
50
-
[following these instructions](elastic-agent://reference/edot-collector/custom-collector.md).
39
+
### {{ech}}
40
+
```{applies_to}
41
+
ess:
42
+
stack: preview 9.2
43
+
```
51
44
52
-
:::{note}
53
-
The EDOT Gateway Collector does not send data through Elastic's Integration / APM Server on ECH to ingest data into {{es}}.
54
-
:::
45
+
{{ech}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data. Users can send data direct from the Collectors or SDKs deployed on the edge environment through OTLP without any additional requirements for managing an ingestion layer.
55
46
56
-
:::{important}
57
-
If self-managing an EDOT Gateway is not a valid option for you, refer to [Elastic's classic ingestion path for OTel data on ECH](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md).
58
-
:::
47
+
59
48
60
49
### Self-managed
61
50
62
-
In a self-managed deployment scenario, you need to host an EDOT Collector in Gateway mode that pre-processes and ingests OTel data from the edge setup into the self-managed Elastic Stack.
51
+
In a self-managed Elastic deployment, we recommend running an EDOT Collector in gateway mode as a unified ingestion layer for OTLP telemetry from OpenTelemetry collectors or SDKs running at the edge. This gateway can receive all signals (logs, metrics and traces), apply processing as needed, and cover the same use cases that previously required components like APM Server or Logstash.
52
+
53
+
Depending on your scalability and durability needs, this can be a single collector that scales horizontally, or a chain of collectors where each tier handles a specific concern. For high availability and stronger durability guarantees, you can insert Kafka between tiers so that ingestion is buffered and resilient to downstream outages.
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Gateway Collector there is no need for an APM Server anymore.
58
+
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Collector in gateway mode there is no need for {{product.apm-server}} anymore.
68
59
69
-
Refer to [EDOT data streams compared to classic APM](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
60
+
Refer to [data streams compared to classic {{product.apm}}](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
Copy file name to clipboardExpand all lines: docs/reference/architecture/index.md
+50Lines changed: 50 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -18,3 +18,53 @@ The following sections outline the recommended architectures for Elastic Distrib
18
18
-[Hosts and VMs](hosts_vms.md)
19
19
-[Kubernetes](k8s.md)
20
20
21
+
## Understanding edge deployment
22
+
23
+
In the context of OpenTelemetry architecture, 'edge' refers to the environment where applications are running: individual hosts, virtual machines, or as daemon sets on Kubernetes nodes. Collectors deployed in these environments perform two primary functions:
24
+
25
+
1. They gather logs, infrastructure metrics, application traces, profiles, and other telemetry from the local environment.
26
+
2. They enrich application telemetry from OpenTelemetry SDKs running on the same host or node with resource metadata, such as host information and Kubernetes attributes.
27
+
28
+
### Vendor-neutral collector deployments on edge environments
29
+
30
+
You can use any OpenTelemetry Collector distribution at the edge, including:
31
+
32
+
- The contrib OpenTelemetry Collector.
33
+
- Custom-built Collector distributions.
34
+
- The {{edot}} Collector.
35
+
- Any shipper capable of sending valid OTLP data.
36
+
37
+
The only requirement is that collectors deployed on edge environments send data using valid OpenTelemetry Protocol (OTLP) to {{ecloud}} (using managed OTLP endpoint) or to the OTLP receiver of a gateway EDOT collector for self-managed deployments.
38
+
39
+
While any OTLP-compatible collector works at the edge, using EDOT provides a more streamlined experience with:
40
+
41
+
- Preconfigured components optimized for {{product.observability}}.
42
+
- Curated receivers and processors for common use cases.
43
+
- Simplified setup and configuration.
44
+
45
+
## Understanding the {{product.observability}} backend [understanding-the-elastic-observability-backend]
46
+
47
+
The {{product.observability}} backend is the ingestion and storage layer where your telemetry data is processed, indexed, and made available for analysis. This includes:
48
+
49
+
-**{{es}}**: Stores and indexes your telemetry data.
50
+
-**{{kib}}**: Provides visualization and analysis interfaces.
51
+
-**Managed OTLP Endpoint** (for {{serverless-full}} and {{ech}}): A managed ingestion service that receives OTLP data.
52
+
-**Gateway Collector** (for self-managed, ECE, and ECK): An EDOT Collector running in gateway mode that serves as the ingestion layer.
53
+
54
+
:::{note}
55
+
When a Collector in gateway mode is deployed, it is considered part of the {{product.observability}} backend, not part of the edge deployment. The Collector in gateway mode sits between your edge collectors and {{es}}, acting as a centralized ingestion and preprocessing layer.
56
+
:::
57
+
58
+
### When gateway mode is required
59
+
60
+
The need for an EDOT Collector in gateway mode as part of your Elastic backend depends on your deployment type:
61
+
62
+
**{{serverless-full}} and {{ech}}**: Edge collectors can send OTLP data directly to the [Managed OTLP Endpoint](/reference/motlp.md) without requiring a self-managed Gateway Collector. The Managed OTLP Endpoint provides a fully managed ingestion layer as part of the Elastic backend. You can optionally deploy an EDOT Collector in gateway mode as part of your edge environment if you need additional processing before data reaches the Managed OTLP Endpoint.
63
+
64
+
**Self-managed, ECE, and ECK deployments**: An EDOT Collector in gateway mode is **required as part of your {{product.observability}} backend**. This Gateway Collector exposes the OTLP endpoint that edge collectors send data to, and handles essential preprocessing, including:
65
+
66
+
- Metrics aggregation for traces and logs, which improves {{product.apm}} UI performance with lower latency.
67
+
- Format conversion for optimal storage in {{es}}.
68
+
69
+
For detailed information about Agent and Gateway modes and their specific requirements, refer to the [deployment mode documentation](elastic-agent://reference/edot-collector/modes.md).
Copy file name to clipboardExpand all lines: docs/reference/architecture/k8s.md
+43-40Lines changed: 43 additions & 40 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -11,81 +11,84 @@ products:
11
11
- id: edot-collector
12
12
---
13
13
14
-
# Kubernetes Environments
14
+
# Kubernetes environments
15
15
16
-
The recommended OTel architecture for Kubernetes clusters includes a set of OpenTelemetry collectors in different forms. The following diagram shows the different forms:
16
+
In Kubernetes environments, deploy OpenTelemetry Collectors as [edge](index.md#understanding-edge-deployment) collectors within your cluster where your applications are running. The recommended architecture includes different types of collectors deployed as various Kubernetes resource types (DaemonSets, Deployments) to handle specific collection needs.
17
17
18
-
18
+
Collectors deployed in these edge environments have several main purposes:
19
19
20
-
## Daemon form
20
+
1. The collection of node-level logs and infrastructure metrics through DaemonSet collectors running on each node.
21
+
2. The collection of cluster-level metrics through a dedicated Cluster collector.
22
+
3. Enriching application telemetry from OTel SDKs with corresponding host and Kubernetes resource attributes.
23
+
4. Centralizing data through a gateway collector before sending to the Elastic backend.
21
24
22
-
The Collector in Daemon form is deployed on each Kubernetes node to collect nodes-local logs and host metrics.
25
+
## Types of collectors
23
26
24
-
The daemon collector also receives telemetry from applications instrumented with OTel SDKs and running on corresponding nodes.
27
+
The recommended OTel architecture for Kubernetes clusters includes a set of OpenTelemetry collectors in different forms.
28
+
29
+
### DaemonSet collectors
30
+
31
+
The Collector as a DaemonSet resource kind, deploys an instance on each Kubernetes node as an [edge collector](index.md#understanding-edge-deployment) (close to your data sources) to collect node-local logs and host metrics.
32
+
33
+
The DaemonSet collectors also can funnel and enrich telemetry from applications instrumented with OTel SDKs and running on corresponding nodes.
25
34
26
35
That Collector enriches the application telemetry with resource information such as host and Kubernetes metadata.
27
36
28
-
All data is then being sent through OTLP to the OTel or EDOT Gateway Collector.
37
+
All data is then sent through OTLP to the gateway Collector running in the cluster.
29
38
30
-
## Cluster form
39
+
###Cluster collector
31
40
32
-
The Collector in Cluster form collects Kubernetes cluster-level metrics and sends them to the OTel or EDOT Gateway Collector using OTLP.
41
+
The Cluster collector is a Deployment resource kind that collects Kubernetes cluster-level metrics and sends them to the gateway Collector using OTLP.
33
42
34
-
## Gateway form
43
+
This instance of the collector helps with collecting cluster level metrics which otherwise would be duplicated by the DaemonSet instances.
35
44
36
-
The OTel or EDOT Collector in Gateway form gathers the OTel data from all other collectors and ingests it into the Elastic backend.
45
+
### Gateway collector (on edge)
37
46
38
-
For self-managed and {{ech}} deployment models the Gateway Collector does some additional pre-processing of data.
47
+
The gateway Collector deployed in the Kubernetes cluster centralizes OTel data from the DaemonSet and Cluster collectors. This gateway collector runs as part of your edge environment, not as part of the Elastic backend. It can be any OpenTelemetry Collector distribution as long as it forwards data using OTLP to the appropriate backend endpoint:
48
+
49
+
- For {{serverless-full}} and {{ech}}: Sends OTLP data to the Managed OTLP Endpoint.
50
+
- For self-managed, ECE, and ECK: Sends OTLP data to an EDOT Collector running in gateway mode on the backend side (refer to [backend architecture](index.md#understanding-the-elastic-observability-backend)).
39
51
40
52
## Deployment scenarios
41
53
42
-
See the recommended architectures per Elastic deployment scenarios:
54
+
The following sections outline the recommended architectures for different Elastic deployment scenarios.
43
55
44
56
::::{note}
45
-
Elastic's Observability solution is technically compatible with setups that are fully based on contrib OTel components, as long as the ingestion path follows the recommendations outlined in following sub-sections for the different Elastic deployment options.
57
+
{{product.observability}} is technically compatible with setups that are fully based on contrib OTel components, as long as the ingestion path follows the recommendations outlined in following sections for the different Elastic deployment options.
46
58
::::
47
59
48
-
### Elastic Cloud Serverless
60
+
### {{serverless-full}}
49
61
50
62
{{serverless-full}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data.
For a Kubernetes setup, that means the Gateway Collector passes through the OTel data in native format using the OTLP protocol to the managed OTLP endpoint. There is no need for the Gateway Collector to do any Elastic-specific pre-processing.
55
-
56
-
### Elastic Cloud Hosted
64
+
57
65
58
-
With {{ech}} (ECH), OTel data is being directly ingested into the Elastic-hosted {{es}} instance.
66
+
For a Kubernetes setup, the gateway Collector in your cluster sends OTLP data directly to the Managed OTLP Endpoint. There is no need for an EDOT gateway collector on the backend side. You can optionally deploy an EDOT Collector in gateway mode as part of your edge environment if you need additional processing before data reaches the Managed OTLP Endpoint.
59
67
60
-
68
+
### {{ech}}
69
+
```{applies_to}
70
+
ess:
71
+
stack: preview 9.2
72
+
```
61
73
62
-
The Gateway Collector needs to do some preprocessing, aggregation of metrics and, finally, it uses the {{es}} exporter to ingest data into ECH.
74
+
{{ech}} provides a [Managed OTLP Endpoint](/reference/motlp.md) for ingestion of OpenTelemetry data.
63
75
64
-
While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendor agnostic or upstream, the Gateway Collector needs to be either an EDOT Collector or a [custom, EDOT-like Collector](elastic-agent://reference/edot-collector/custom-collector.md) containing the
65
-
[required components and pre-processing pipelines](elastic-agent://reference/edot-collector/config/default-config-k8s.md#direct-ingestion-into-elasticsearch).
76
+
For a Kubernetes setup, the gateway Collector in your cluster sends OTLP data directly to the Managed OTLP Endpoint. There is no need for an EDOT gateway collector on the backend side.
66
77
67
-
If required, users can build their custom, EDOT-like Collector [following these instructions](elastic-agent://reference/edot-collector/custom-collector.md).
68
-
69
-
::::{note}
70
-
The EDOT Gateway Collector does not send data through Elastic's Integration / APM Server on ECH to ingest data into {{es}}.
71
-
::::
72
-
73
-
::::{important}
74
-
If self-managing an EDOT Gateway is not a valid option for you, refer to [Elastic's classic ingestion path for OTel data on ECH](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md).
75
-
::::
78
+
76
79
77
80
### Self-managed
78
81
79
-
With a self-managed scenario the Gateway Collector ingests data directly into the self-managed {{es}} instance.
82
+
With a self-managed deployment, you need an EDOT Collector running in gateway mode as part of your {{product.observability}} backend (not in the Kubernetes cluster). This backend gateway collector receives OTLP data from the Kubernetes gateway collector and ingests it into {{es}}.
The Gateway Collector does some preprocessing and aggregation of OTel data before ingesting it into {{es}}.
86
+
The backend gateway Collector processes {{product.apm}} data and logs to improve latency on solution UIs before ingesting into {{es}}.
84
87
85
-
While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay fully vendoragnostic or upstream, the Gateway Collector needs to be either an EDOT Collector or a [custom, EDOT-like Collector](elastic-agent://reference/edot-collector/custom-collector.md) containing the [required components and pre-processing pipelines](elastic-agent://reference/edot-collector/config/default-config-k8s.md#direct-ingestion-into-elasticsearch).
88
+
The Kubernetes cluster components (DaemonSet collectors, Cluster collectors, gateway collector, and OTel SDKs) can all use fully vendor-agnostic or upstream OpenTelemetry components. Only the backend gateway Collector needs to be either an EDOT Collector or a [custom, EDOT-like Collector](elastic-agent://reference/edot-collector/custom-collector.md) containing the [required components and preprocessing pipelines](elastic-agent://reference/edot-collector/config/default-config-k8s.md#direct-ingestion-into-elasticsearch).
86
89
87
90
::::{note}
88
-
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Gateway Collector there is no need for an APM Server anymore.
91
+
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT gateway Collector there is no need for {{product.apm-server}} anymore.
89
92
90
-
Refer to [EDOT data streams compared to classic APM](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
93
+
Refer to [EDOT data streams compared to classic {{product.apm}}](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
0 commit comments