[otelconsumer] evaluate at least once delivery and queue settings

[leehinman]

To have at least once delivery guarantee the otel exporter should use the exportbatcher. But this can create poor performance

Example 1

If there is just one event, we have to wait for the Beats queue timeout before it is sent to the exportbatcher queue, and then wait for the exportbatcher queue timeout before it is sent to the final destination. This means wait for both timeouts.

Example 2

If the beats bulk_max_size is less than the exportbatcher min_size_items then the batch will wait in the exportbatcher queue until the exportbatcher timeout. This means waiting for one timeout even though the beats code is blocked and can't progress.

We need to evaluate how best to remove negative interactions between the two queues.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

For additional context, the batchprocessor is deprecated and runs asynchronously, which means we can't confirm actual delivery. The exporterhelper batcher is the new experimental batching mechanism; it uses queues and provides delivery guarantees. The batcher helper is expected to become the default batching mechanism in the future.

It seems we could limit batching and timeouts to one side to prevent stacking. It might also be necessary to dynamically define these conditions.

[mauri870]

Related changes and discussion around batcher options: #42797

[VihasMakwana]

upstream batcher implementation had missing validations and as a result, we always waited for 30 seconds to flush the request, regardless of batch size.
PR to add validation: open-telemetry/opentelemetry-collector-contrib#38072