From 58c4cdf3cce5083c0e4a134c3dee7ded9b5a5512 Mon Sep 17 00:00:00 2001 From: Mauri de Souza Meneguzzo Date: Wed, 3 Dec 2025 09:40:28 -0300 Subject: [PATCH] docs(otel): document delivery guarantees for OTel mode When running in OTel mode, delivery guarantees for Beats receivers are only possible with a specific combination of retry settings, the sending queue, and Beats queue options, so document that. --- docs/hybrid-agent-beats-receivers.md | 43 ++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/docs/hybrid-agent-beats-receivers.md b/docs/hybrid-agent-beats-receivers.md index 65e99f08a38..f10576ee635 100644 --- a/docs/hybrid-agent-beats-receivers.md +++ b/docs/hybrid-agent-beats-receivers.md @@ -193,6 +193,7 @@ receivers: paths: - /var/log/*.log type: filestream + queue.mem.flush.timeout: 0s metricbeatreceiver: metricbeat: modules: @@ -202,6 +203,7 @@ receivers: metricsets: - cpu module: system + queue.mem.flush.timeout: 0s exporters: elasticsearch/_agent-component/default: api_key: placeholder @@ -214,6 +216,24 @@ exporters: enabled: true mapping: mode: bodymap + + retry: + enabled: true + initial_interval: 1s + max_interval: 1m0s + max_retries: 5 + + sending_queue: + enabled: true + wait_for_result: true + block_on_overflow: true + num_consumers: 1 + queue_size: 3200 + batch: + max_size: 1600 + min_size: 0 + flush_timeout: 10s + sizer: items service: pipelines: logs: @@ -223,3 +243,26 @@ service: - filebeatreceiver - metricbeatreceiver ``` + +### Beats receivers delivery guarantees in OTel mode + +When Beat receivers are used in OTel mode, event delivery guarantees depend on the configuration of the OpenTelemetry Collector `sending_queue` and retry settings. +Unlike standalone Beats, the EDOT pipeline allows users to customize queue behavior through the Collector configuration. +This flexibility is useful, but it also means that not every option combination is compatible with reliable delivery. + +Elastic Agent in OTel mode provides an **at least once** delivery guarantee for Beat receivers **only when using the supported `sending_queue` settings described below**. +These settings mirror Beats pipeline behavior closely enough to preserve durability expectations. + +If users provide arbitrary `sending_queue` or Beat queue overrides, delivery semantics become **undefined** and **at least once delivery cannot be guaranteed**. +These combinations are not tested and may result in event loss during backpressure or shutdown. + +To achieve the intended delivery guarantee, the exporter that receives events from Beat receivers must define a `sending_queue` with the following characteristics: + +- `enabled: true`: The queue must be active. +- `wait_for_result: true`: The pipeline must wait for the exporter response before removing events. +- `block_on_overflow: true`: Prevents event drops when the queue is full. +- A bounded queue with enough capacity and batch configured with explicit limits. + +Additionally, the `retry` seetings must be enabled on the exporter, with a backoff policy that retries until success. + +Beat receivers also require the Beat-internal memory queue to work in synchronous mode, which is achieved by setting `queue.mem.flush.timeout: 0s` in each receiver configuration, as shown in the example above.