Introduce new exporter helper with batching option #8122

dmitryax · 2023-07-22T02:57:46Z

This is a tracking issue for introducing the new exporter helper and migrating the existing exporters to use it.

The primary reason for introducing the new exporter helper is to move the batching to the exporter side and deprecate the batch processor as part of making the delivery pipeline reliable, as reported in #7460. More details about moving batching to the exporter helper can be found in #4646.

Shifting batching to the exporter side grants us the opportunity to leverage the exporter's data model instead of relying on OTLP. As a result, we can achieve the following benefits:

Ability to place failed requests back into the queue without the need for converting them back to OTLP format.
Enhanced control in counting queue and batch sizes using basic items (like spans, data points, or log records for OTLP) tailored to different exporters, resolving the concern raised in issue Processor: Splitting summary metrics into timeseries. opentelemetry-collector-contrib#7134.
Optional counting of queue and batch sizes in bytes of serialized data.

Adapting to the new exporter helper requires exporter developers to implement at least two functions:

Converter: to translate pdata Metrics/Traces/Logs into a user-defined Request.
Request sender: to send the user-defined Request

Design document: https://docs.google.com/document/d/1uxnn5rMHhCBLP1s8K0Pg_1mAs4gCeny8OWaYvWcuibs

Essential sub-issues to mark this as complete:

Additional sub-issues to get feature parity with the batch processor:

[exporterhelper] Add an ability to batch by a key from pdata or context #10825

Introduce a new exporter helper that operates over client-provided requests instead of pdata. The helper user now has to provide `Converter` - an interface with a function implementing translation of pdata Metrics/Traces/Logs into a user-defined `Request`. `Request` is an interface with only one required function `Export`. It opens a door for moving batching to the exporter, where batches will be built from client data format, instead of pdata. The batches can be properly sized by custom request size, which can be different from OTLP. The same custom request sizing will be applied to the sending queue. It will also improve the performance of the sending queue retries for non-OTLP exporters, they don't need to translate pdata on every retry. This is an implementation alternative to #7874 as suggested in #7874 (comment) Tracking Issue: #8122 --------- Co-authored-by: Alex Boten <[email protected]>

This change adds collector's internal metrics and tracing to the new request-based exporter helpers. Only those metrics and traces are added that are already adopted by the existing exporter helpers for backward compatibility. The new exporter helpers can and should expose more metrics in the future, e.g. for tracking converter errors. Tracking Issue: #8122

bogdandrutu · 2023-08-22T19:23:55Z

Some feedback about the interface:

[Traces|Metrics|Logs]Converter can be replaced with a simple func instead of having an interface. If you need in the future to extend capabilities of that, they can be options. Comparing interfaces with nil is problematic sometimes, see https://mangatmodi.medium.com/go-check-nil-interface-the-right-way-d142776edef1, referring to https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/metrics.go#L142
Can you make the old NewMetricsExporter to call into NewMetricsRequestExporter to remove duplicate code and to actually start testing the new path more intensively? Also removes lots of duplicate tests.

dmitryax · 2023-08-22T19:51:51Z

Thanks for the feedback!

Can you make the old NewMetricsExporter to call into NewMetricsRequestExporter to remove duplicate code and to actually start testing the new path more intensively? Also removes lots of duplicate tests.

That's what I wanted to do after #8248 is merged 👍

As proposed in open-telemetry#8122 (comment)

…on (#8764) As proposed in #8122 (comment) If we need backward conversion, we will use an optional argument to the helper function instead of an optional interface.

#9164) Introduce an option to limit the queue size by the number of items instead of number of requests. This is preliminary step for having the exporter helper v2 with a batcher sender placed after the queue sender. Otherwise, it'll be hard for the users to estimate the queue size based on the number of requests without batch processor in front of it. This change doesn't effect the existing functionality and the items based queue limiting cannot be utilized yet. Updates #8122 Alternative to #9147

Introduce a way to enable queue in the new exporter helper with a developer interface suggested in #8248 (comment). The new configuration interface for the end users provides a new `queue_size_items` option to limit the queue by a number of spans, log records, or metric data points. The previous way to limit the queue by number of requests is preserved under the same field, `queue_size,` which will later be deprecated through a longer transition process. Tracking issue: #8122

Add opt-in support for the experimental batcher helper - open-telemetry/opentelemetry-collector#8122 When opting into this functionality the exporter's Consume* methods will make synchronous bulk requests to Elasticsearch, without additional batching/buffering in the exporter.

**Description:** Add opt-in support for the experimental batch sender (open-telemetry/opentelemetry-collector#8122). When opting into this functionality the exporter's `Consume*` methods will make synchronous bulk requests to Elasticsearch, without additional batching/buffering in the exporter. By default the exporter continues to use its own buffering, which supports thresholds for time, number of documents, and size of encoded documents in bytes. The batch sender does not currently support a bytes-based threshold, and is experimental, hence why we are not yet making it the default for the Elasticsearch exporter. This PR is based on #32632, but made to be non-breaking. **Link to tracking Issue:** #32377 **Testing:** Added unit and integration tests. Manually tested with Elasticsearch, using the following config to highlight that client metadata can now flow through all the way: ```yaml receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 include_metadata: true exporters: elasticsearch: endpoint: "http://localhost:9200" auth: authenticator: headers_setter batcher: enabled: false extensions: headers_setter: headers: - action: insert key: Authorization from_context: authorization service: extensions: [headers_setter] pipelines: traces: receivers: [otlp] processors: [] exporters: [elasticsearch] ``` I have Elasticsearch running locally, with an "admin" user with the password "changeme". Sending OTLP/HTTP to the collector with `telemetrygen traces --otlp-http --otlp-insecure http://localhost:4318 --otlp-header "Authorization=\"Basic YWRtaW46Y2hhbmdlbWU=\""`, I observe the following: - Without the `batcher` config, the exporter fails to index data into Elasticsearch due to an auth error. That's because the exporter is buffering and dropping the context with client metadata, so there's no Authorization header attached to the requests going out. - With `batcher: {enabled: true}`, same behaviour as above. Unlike the [`batch` processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor), the batch sender does not maintain client metadata. - With `batcher: {enabled: false}`, the exporter successfully indexes data into Elasticsearch. **Documentation:** Updated the README. --------- Co-authored-by: Carson Ip <[email protected]>

axw · 2024-08-14T01:45:14Z

As discussed in today's Collector SIG meeting, we (Elastic) have the same requirement as @verejoel described in #8122 (comment).

We intend to support a multi-tenant collector that receives data from different tenants, and sends the data to a tenant-specific Elasticsearch cluster. For performance reasons we should batch the data before exporting to Elasticsearch, which must necessarily be done per tenant.

Although partitioning may be done by the batch sender (#10825), each backend instance (in our case, Elasticsearch cluster) may have different performance (different load, different scaling, ...) or availability; with a single queue, this may lead to head-of-line blocking.

#### Description This PR adds opt-in support to oltp exporter for the experimental batch sender (#8122). By default batch sender is not enabled. Similar: * open-telemetry/opentelemetry-collector-contrib#34238 * open-telemetry/opentelemetry-collector-contrib#32563 #### Link to tracking issue Resolves #10834 #### Testing `exporter/otlpexporter/config_test.go` #### Documentation Updated the `oltpexporter` README to point to `exporterhelper` README for batching. --------- Co-authored-by: Dmitrii Anoshin <[email protected]>

… to a class function instead of a callback (#11338) #### Description Why this change? Each request from the queue contains multiple items, and those items could be merge-split into multiple batches when they are sent out (see #8122 for more about exporter batcher). We would like to book-keep those cases, and only call `onProcessingFinished` when all such batches has gone out. In this PR, `onProcessingFinished` is changed from a callback to a method function because it is easier to book keep index instead of functions. #### Link to tracking issue #8122 #10368 #### Testing `exporter/internal/queue/persistent_queue_test.go` #### Documentation This is an internal change invisible to the users. --------- Co-authored-by: Dmitrii Anoshin <[email protected]>

… to a class function instead of a callback (open-telemetry#11338) #### Description Why this change? Each request from the queue contains multiple items, and those items could be merge-split into multiple batches when they are sent out (see open-telemetry#8122 for more about exporter batcher). We would like to book-keep those cases, and only call `onProcessingFinished` when all such batches has gone out. In this PR, `onProcessingFinished` is changed from a callback to a method function because it is easier to book keep index instead of functions. #### Link to tracking issue open-telemetry#8122 open-telemetry#10368 #### Testing `exporter/internal/queue/persistent_queue_test.go` #### Documentation This is an internal change invisible to the users. --------- Co-authored-by: Dmitrii Anoshin <[email protected]>

#### Description This PR adds a public function `GetNextItem` to queue (both persistent queue and bounded memory queue) Why this change? Instead of blocking until consumption of the item is done, we would like to separate the API for reading and committing consumption. Before: `Consume(consumeFunc)` After: `idx, item = Read()` `OnProcessingFinished(idx)`  #### Link to tracking issue #8122 #10368  #### Testing  #### Documentation

… that operates on batch sender (#11448) #### Description As part of the effort to solve #10368, we no longer guarantee to initialize a `batchSender` when `batcher` is enabled. Therefore, we would like to remove the interface to set `mergeFunc` and `mergeSplitFunc` as a callback that operates on `batchSender`. Instead, users should use the alternative `WithBatchFuncs` that is a callback that operates `baseExporter`. Context: #11414 #### Link to tracking issue #8122 #10368 --------- Co-authored-by: Bogdan Drutu <[email protected]>

) #### Description This PR is a bare minimum implementation of a component called queue batcher. On completion, this component will replace `consumers` in `queue_sender`, and thus moving queue-batch from a pulling model instead of pushing model. Limitations of the current code * This implements only the case where batching is disabled, which means no merge of splitting of requests + no timeout flushing. * This implementation does not enforce an upper bound on concurrency All these code paths are marked as panic currently, and they will be replaced with actual implementation in coming PRs. This PR is split from #11507 for easier review. Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue #8122 #10368

#### Description This PR follows #11532 and implements support for limited worker pool for queue batcher. Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue #8122 #10368

…d exports (#11546) #### Description This PR follows #11540 and implements support for item-count based batching for queue batcher. Limitation: This PR supports merging request but not splitting request. In other words, it support specifying a minimum request size but not a maximum request size. Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue #8122 #10368

…batcher (#11580) #### Description This PR follows #11546 and add support for splitting (i.e. support setting a maximum request size) Design doc: https://docs.google.com/document/d/1y5jt7bQ6HWt04MntF8CjUwMBBeNiJs2gV4uUZfJjAsE/edit?usp=sharing #### Link to tracking issue #8122 #10368

dmitryax added the area:exporter label Jul 22, 2023

dmitryax self-assigned this Jul 22, 2023

dmitryax mentioned this issue Jul 22, 2023

[exporterhelper] New exporter helper for custom requests #7874

Closed

jmacd mentioned this issue Aug 16, 2023

Support for blocking OTel-Arrow pipeline open-telemetry/otel-arrow#11

Closed

This was referenced Aug 16, 2023

[exporterhelper] New exporter helper for custom requests #8178

Merged

[exporterhelper] Add observability to the new exporter helper #8244

Merged

[exporterhelper] Add queue options to the new exporter helper #8248

Closed

This was referenced Aug 24, 2023

add bytes limit feature to batch processor #8272

Open

[exporterhelper] Add WithRequestQueue option to the exporter #8275

Closed

dmitryax mentioned this issue Aug 31, 2023

[pkg/stanza] Make batching behavior configurable open-telemetry/opentelemetry-collector-contrib#21184

Closed

dmitryax mentioned this issue Sep 13, 2023

[exporterhelper] Convert internal request interface into a struct #8435

Closed

dmitryax mentioned this issue Sep 25, 2023

New Component: Encoding Extension open-telemetry/opentelemetry-collector-contrib#28686

Closed

This was referenced Oct 2, 2023

[exporterhelper] Add batch sender #8440

Closed

[exporterhelper] Introduce batching functionality #8685

Merged

dmitryax added a commit to dmitryax/opentelemetry-collector that referenced this issue Oct 26, 2023

[exporterhelper] Replace experimental converter interface with function

8c52bd5

As proposed in open-telemetry#8122 (comment)

dmitryax mentioned this issue Oct 26, 2023

[exporterhelper] Replace experimental converter interface with function #8764

Merged

dmitryax added a commit to dmitryax/opentelemetry-collector that referenced this issue Oct 26, 2023

[exporterhelper] Replace experimental converter interface with function

2e5b127

As proposed in open-telemetry#8122 (comment)

dmitryax mentioned this issue Oct 28, 2023

[DRAFT] New exporter helper #8762

Closed

dmitryax mentioned this issue Nov 12, 2023

[exporterhelper] Add WithRequestQueue option to the exporter #8853

Merged

atoulme added the release:required-for-ga Must be resolved before GA release label Dec 19, 2023

This was referenced Dec 19, 2023

[chore] [exporterhelper] Add an option for items based queue sizing #9147

Closed

Uniform defaults for exporterhelper #4455

Open

[chore] [exporterhelper] Items based queue sizing with bounded channel #9164

Merged

axw mentioned this issue Jul 24, 2024

[exporter/elasticsearch] add support for batcher config open-telemetry/opentelemetry-collector-contrib#34238

Merged

dmitryax mentioned this issue Aug 7, 2024

[otlp/exporter] Enable batcher in OTLP exporter #10834

Closed

sfc-gh-sili mentioned this issue Aug 9, 2024

Enable batch sender in oltpexporter #10846

Merged

jade-guiton-dd mentioned this issue Sep 20, 2024

[exporterhelper] Review usage of API and internalize anything not currently used #11142

Open

jmacd mentioned this issue Sep 23, 2024

Concurrent batch processor features → batch processor #11248

Closed

jade-guiton-dd mentioned this issue Sep 27, 2024

[exporterhelper, exporterqueue] Deprecate unused public symbols #11285

Open

jmacd mentioned this issue Sep 30, 2024

Batch processor concurrency and error transmission #11308

Open

sfc-gh-sili mentioned this issue Oct 3, 2024

[exporter] internal/persistent_queue::OnProcessingFinished is changed to a class function instead of a callback #11338

Merged

sfc-gh-sili mentioned this issue Oct 8, 2024

[exporter] exporter queue Read() #11396

Merged

This was referenced Oct 12, 2024

[exporter] moving queue sender from pushing model to pulling model #11414

Draft

[exporter] Disable the API to pass in configurations using a callback that operates on batch sender #11448

Merged

This was referenced Oct 22, 2024

[exporterqueue] Default Batcher that reads from the queue, batches and exports #11507

Closed

[exporterqueue] Bare minimum frame of queue batcher + unit test. #11532

Merged

[exporterqueue] Limited worker pool support for queue batcher #11540

Merged

sfc-gh-sili mentioned this issue Oct 28, 2024

[exporterqueue] Default Batcher that reads from the queue, batches and exports #11546

Merged

sfc-gh-sili mentioned this issue Oct 31, 2024

[chore] [exporterqueue] Add splitting to the experimental pull-based batcher #11580

Merged

swiatekm mentioned this issue Nov 5, 2024

[testbed] Add batcher performance tests open-telemetry/opentelemetry-collector-contrib#36206

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce new exporter helper with batching option #8122

Introduce new exporter helper with batching option #8122

dmitryax commented Jul 22, 2023 •

edited

Loading

bogdandrutu commented Aug 22, 2023

dmitryax commented Aug 22, 2023

axw commented Aug 14, 2024

Introduce new exporter helper with batching option #8122

Introduce new exporter helper with batching option #8122

Comments

dmitryax commented Jul 22, 2023 • edited Loading

bogdandrutu commented Aug 22, 2023

dmitryax commented Aug 22, 2023

axw commented Aug 14, 2024

dmitryax commented Jul 22, 2023 •

edited

Loading