[exporter/elasticsearch] Use batch sender exporter helper for reliability

[carsonip]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

This is a way to fix the issue reported in #30792.

In case of a collector crash, the events buffered within the bulk indexer in memory will be lost, even with a persistent queue.

Describe the solution you'd like

Move the buffering from bulk indexer to exporterhelper batch sender introduced in open-telemetry/opentelemetry-collector#8122, such that events are deleted from the queue only when they are flushed by bulk indexer.

Describe alternatives you've considered

No response

Additional context

I'm already working on a draft and related PRs are on their way.

exporterhelper batch sender open-telemetry/opentelemetry-collector#8122 is WIP and interface may change. Some features (e.g. batching by bytes) may not be available yet.

Blockers

• [exporter/elasticsearch] Replace go-elasticsearch with go-docappender BulkIndexer to prepare for reliability fixes #32378

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @ycombinator

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[crobert-1]

This enhancement makes sense to me, and sounds like a good idea. I'll defer to code owners in case there are component-specific thoughts here.

Removing needs triage.

[ycombinator]

This enhancement makes sense to me, and sounds like a good idea. I'll defer to code owners in case there are component-specific thoughts here.

Agreed, +1