Skip to content

docs(otel): document delivery guarantees for OTel mode #11560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
+43 −0
Open

docs(otel): document delivery guarantees for OTel mode #11560

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions docs/hybrid-agent-beats-receivers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,7 @@ receivers:
paths:
- /var/log/*.log
type: filestream
queue.mem.flush.timeout: 0s
metricbeatreceiver:
metricbeat:
modules:
Expand All @@ -202,6 +203,7 @@ receivers:
metricsets:
- cpu
module: system
queue.mem.flush.timeout: 0s
exporters:
elasticsearch/_agent-component/default:
api_key: placeholder
Expand All @@ -214,6 +216,24 @@ exporters:
enabled: true
mapping:
mode: bodymap

retry:
enabled: true
initial_interval: 1s
max_interval: 1m0s
max_retries: 5
Copy link
Member

@cmacknz cmacknz Dec 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be equal to filebeat, we need the ability to retry indefinitely. This is covered by elastic/beats#47892 so we probably want to reference that.


sending_queue:
enabled: true
wait_for_result: true
block_on_overflow: true
num_consumers: 1
queue_size: 3200
batch:
max_size: 1600
min_size: 0
flush_timeout: 10s
sizer: items
service:
pipelines:
logs:
Expand All @@ -223,3 +243,26 @@ service:
- filebeatreceiver
- metricbeatreceiver
```

### Beats receivers delivery guarantees in OTel mode

When Beat receivers are used in OTel mode, event delivery guarantees depend on the configuration of the OpenTelemetry Collector `sending_queue` and retry settings.
Unlike standalone Beats, the EDOT pipeline allows users to customize queue behavior through the Collector configuration.
This flexibility is useful, but it also means that not every option combination is compatible with reliable delivery.

Elastic Agent in OTel mode provides an **at least once** delivery guarantee for Beat receivers **only when using the supported `sending_queue` settings described below**.
These settings mirror Beats pipeline behavior closely enough to preserve durability expectations.

If users provide arbitrary `sending_queue` or Beat queue overrides, delivery semantics become **undefined** and **at least once delivery cannot be guaranteed**.
These combinations are not tested and may result in event loss during backpressure or shutdown.

To achieve the intended delivery guarantee, the exporter that receives events from Beat receivers must define a `sending_queue` with the following characteristics:

- `enabled: true`: The queue must be active.
- `wait_for_result: true`: The pipeline must wait for the exporter response before removing events.
- `block_on_overflow: true`: Prevents event drops when the queue is full.
- A bounded queue with enough capacity and batch configured with explicit limits.

Additionally, the `retry` seetings must be enabled on the exporter, with a backoff policy that retries until success.

Beat receivers also require the Beat-internal memory queue to work in synchronous mode, which is achieved by setting `queue.mem.flush.timeout: 0s` in each receiver configuration, as shown in the example above.
Loading