Skip to content

Commit 93354be

Browse files
alexandra5000alexandra5000
authored
Add EDOT vs. classic APM comparison doc to "Compatibility" section (#391)
* Add EDOT vs. APM data streams comparison * Apply comments * Fix link * Apply review comments * Fix link * Apply comments * Fix build error * Add examples
1 parent 9a3efb3 commit 93354be

File tree

6 files changed

+129
-1
lines changed

6 files changed

+129
-1
lines changed

‎docs/reference/architecture/hosts_vms.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -64,5 +64,7 @@ In a self-managed deployment scenario, you need to host an EDOT Collector in Gat
6464
![VM-self-managed](../images/arch-vm-self-managed.png)
6565

6666
:::{note}
67-
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Gateway Collector there is no need for an APM Server anymore.
67+
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Gateway Collector there is no need for an APM Server anymore.
68+
69+
Refer to [EDOT data streams compared to classic APM](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
6870
:::

‎docs/reference/architecture/k8s.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,4 +86,6 @@ While the Daemon and Cluster collectors, as well as the OTel SDKs, can stay full
8686

8787
::::{note}
8888
Compared to [Elastic's classic ingestion paths](docs-content://solutions/observability/apm/use-opentelemetry-with-apm.md) for OTel data, with the EDOT Gateway Collector there is no need for an APM Server anymore.
89+
90+
Refer to [EDOT data streams compared to classic APM](../compatibility/data-streams.md) for a detailed comparison of data streams, mappings, and storage models.
8991
::::

‎docs/reference/compatibility/data-streams.md‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
---
2+
navigation_title: Data streams comparison
3+
description: Learn how EDOT optimizes telemetry storage and query performance in Elastic Observability compared to classic APM.
4+
applies_to:
5+
stack:
6+
serverless:
7+
observability:
8+
products:
9+
- id: cloud-serverless
10+
- id: observability
11+
- id: edot-collector
12+
- id: edot-sdk
13+
---
14+
15+
# OpenTelemetry data streams compared to classic APM
16+
17+
The Elastic Distribution of OpenTelemetry (EDOT) stores telemetry data using a storage model optimized for OpenTelemetry signals. When `mapping_mode: otel` is enabled on the Elasticsearch exporter (which is the default setting), EDOT writes logs, traces, and metrics to specialized data streams aligned with OpenTelemetry semantics.
18+
19+
This architecture is designed for scalable observability workloads. It supports dynamic attributes, reduces mapping complexity, and avoids issues like mapping explosions or manual dimension setup.
20+
21+
EDOT uses Elasticsearch’s [Logs data stream (LogsDB)](docs-content://manage-data/data-store/data-streams/logs-data-stream.md) and [Time Series Data Streams (TSDS)](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md) as storage backends. These are purpose-built to handle the scale and variety of observability data and improve the storage efficiency.
22+
23+
## Logs and traces in LogsDB
24+
25+
Log and trace data is stored in [LogsDB](docs-content://manage-data/data-store/data-streams/logs-data-stream.md), a storage engine optimized for high-ingest, semi-structured observability data. Benefits include:
26+
27+
* Storage efficiency
28+
* Optimized field handling for dynamic fields (for example, `attributes`)
29+
30+
## Metrics in TSDS
31+
32+
Metric data is stored using Elasticsearch’s [TSDS](docs-content://manage-data/data-store/data-streams/time-series-data-stream-tsds.md). Benefits include:
33+
34+
* Efficient storage using columnar compression
35+
* Fast aggregations
36+
* Automatic detection of metric dimensions (no need to manually define `time_series_dimension` in field mappings)
37+
38+
## Query compatibility with classic APM data streams
39+
40+
EDOT is designed to make OpenTelemetry data queryable using many of the same field names as classic APM (ECS-based) data streams. This helps preserve compatibility with existing dashboards, saved searches, and queries.
41+
42+
Query compatibility is achieved through:
43+
44+
* **`passthrough` fields:** Make nested OpenTelemetry fields available at the top level so they can be queried. For example, while the service name is stored at `resource.attributes.service.name`, you can query it as `service.name` (the same field name as the one used in the classic APM data stream).
45+
* **Field aliases:** Map fields with different names in ECS and OpenTelemetry semantic conventions to a common query name to make migration easier.
46+
47+
### Limitations
48+
49+
Query compatibility is not complete:
50+
51+
* Not all ECS fields have aliases. Some integration-specific fields may require query changes.
52+
* Custom attributes and labels are stored differently.
53+
54+
These differences may require updates to certain queries or visualizations.
55+
56+
Refer to [ECS & OpenTelemetry](ecs://reference/ecs-opentelemetry.md) for details on the available aliases and field mappings.
57+
58+
## Comparison with classic APM data streams
59+
60+
This table highlights key differences between classic Elastic APM data streams and EDOT with `mapping_mode: otel`:
61+
62+
| Feature | Classic APM (ECS-based) | EDOT (`mapping_mode: otel`) |
63+
|---|---|---|
64+
| Index mode | General-purpose data streams (logs, traces, metrics) <br><br> TSDS is not supported for classic APM. | LogsDB (logs/traces), TSDS (metrics) |
65+
| Mapping style | Nested objects are mapped as structured fields. Some exceptions exist, such as `labels.*` and `numeric_labels.*`, where dots in field names are replaced with underscores. <br><br> ECS supports multiple field types (keyword, long, double, date, boolean, etc.) as defined in the schema. | Native OpenTelemetry fields with `passthrough`, preserving types and structure. |
66+
| Attribute handling | Dynamic mapping. Custom attributes are stored under `labels.*` (strings) or `numeric_labels.*` (numbers); dots in field names are replaced with underscores. <br><br> See [Document examples - classic APM](#classic-apm) | Dynamic mapping with native types under `attributes.*`, preserving dots in field names. <br><br> See [Document examples - EDOT](#edot) |
67+
68+
69+
### Document examples
70+
71+
#### Classic APM:
72+
73+
```yaml
74+
"@timestamp": "2025-08-14T05:29:43.922Z"
75+
data_stream:
76+
type: logs
77+
dataset: apm.app.cart-service
78+
namespace: default
79+
service:
80+
name: "cart-service"
81+
host:
82+
ip: ["127.0.0.1", "0.0.0.0"]
83+
kubernetes:
84+
namespace: "ecommerce"
85+
labels:
86+
customer_id: "fc2d1b03-b307-4ae3-a19e-df2804c49fc2"
87+
numeric_labels:
88+
order_id: 4711
89+
cart_items: 42
90+
cart_total_amount: 42.0
91+
message: "Order was successfully created"
92+
log:
93+
level: INFO
94+
```
95+
96+
#### EDOT:
97+
98+
```yaml
99+
"@timestamp": "2025-08-14T05:29:43.922Z"
100+
data_stream:
101+
type: logs
102+
dataset: generic.otel
103+
namespace: default
104+
resource:
105+
attributes:
106+
service.name: "cart-service"
107+
host.ip: ["127.0.0.1", "0.0.0.0"]
108+
k8s.namespace.name: "ecommerce"
109+
attributes:
110+
customer.id: "fc2d1b03-b307-4ae3-a19e-df2804c49fc2"
111+
order.id: 4711
112+
cart.items: 42
113+
cart.total_amount: 42.0
114+
body:
115+
text: "Order was successfully created"
116+
severity_text: INFO
117+
```

‎docs/reference/compatibility/limitations.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,8 @@ The Elastic Distributions of OpenTelemetry (EDOT) come with a new way of ingesti
1818

1919
While EDOT and OTel-native data collection already covers most of the core Observability use cases, the following limitations apply compared to data collection with classic Elastic data collection mechanisms.
2020

21+
Refer to [EDOT data streams compared to classic APM](../compatibility/data-streams.md) for an overview of how these ingestion paths differ.
22+
2123
## Centralized parsing and processing of data
2224

2325
With OTel-native ingestion of data, for example through the EDOT Collector or the [Managed OTLP endpoint](/reference/motlp.md), [{{es}} Ingest Pipelines](docs-content://manage-data/ingest/transform-enrich/ingest-pipelines.md) are not supported.

‎docs/reference/motlp.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,10 @@ This diagram shows data ingest using {{edot}} and the {{motlp}}:
2929
:width: 100%
3030
:::
3131

32+
For a detailed comparison of how EDOT data streams differ from classic Elastic APM data streams, refer to [EDOT data streams compared to classic APM](../reference/compatibility/data-streams.md).
33+
34+
## Prerequisites
35+
3236
Telemetry is stored in Elastic in OTLP format, preserving resource attributes and original semantic conventions. If no specific dataset or namespace is provided, the data streams are: `traces-generic.otel-default`, `metrics-generic.otel-default`, and `logs-generic.otel-default`.
3337

3438
You don't need to use APM Server when ingesting data through the Managed OTLP Endpoint. The APM integration (`.apm` endpoint) is a legacy ingest path that only supports traces and translates OTLP telemetry to ECS, whereas {{motlp}} natively ingests OTLP data for logs, metrics, and traces.

‎docs/reference/toc.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ toc:
4141
- file: compatibility/edot-vs-upstream.md
4242
- file: compatibility/limitations.md
4343
- file: compatibility/nomenclature.md
44+
- file: compatibility/data-streams.md
4445
- file: edot-collector/index.md
4546
children:
4647
- file: edot-collector/download.md

0 commit comments

Comments
 (0)