Skip to content

[exporter/elasticsearch] enhancement request to support specifying pipeline in event #37419

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
leehinman opened this issue Jan 22, 2025 · 2 comments · Fixed by #37860
Closed

[exporter/elasticsearch] enhancement request to support specifying pipeline in event #37419

leehinman opened this issue Jan 22, 2025 · 2 comments · Fixed by #37860
Labels
enhancement New feature or request exporter/elasticsearch

Comments

@leehinman
Copy link
Contributor

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

Currently the elasticsearch exporter only allows you to set the pipeline in the configuration file for the exporter, so that all events go to the same pipeline. Beats allows for each event to specify which pipeline is run per event.

https://github.com/elastic/beats/blob/27adc68350051c939e403219c18a0bb3d6e585f8/libbeat/outputs/elasticsearch/client.go#L385-L390

Describe the solution you'd like

Much like #36882 we would like the ability to specify the pipeline in the event (probably an attribute) and have the exporter read the pipeline from there and use it to set the Elasticsearch Bulk API pipeline query parameter. This will also require changes to https://github.com/elastic/go-docappender to support passing in the pipeline per document.

Describe alternatives you've considered

No response

Additional context

This will be required to support use cases that rely on different pipelines within the same index.
@leehinman leehinman added enhancement New feature or request needs triage New item requiring triage labels Jan 22, 2025
@github-actions GitHub Actions
Copy link
Contributor

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@carsonip
Copy link
Contributor

/label -needs-triage

@github-actions github-actions bot removed the needs triage New item requiring triage label Jan 23, 2025
@leehinman leehinman mentioned this issue Jan 23, 2025
Merged
@github-actions github-actions bot mentioned this issue Jan 28, 2025
Closed
carsonip pushed a commit to elastic/go-docappender that referenced this issue Jan 28, 2025
@leehinman
Add per event pipeline (#227)
95f67c5 
Adds pipeline to the BulkIndexItem. This will be necessary to implement open-telemetry/opentelemetry-collector-contrib#37419
@leehinman leehinman mentioned this issue Feb 11, 2025
Merged
atoulme pushed a commit that referenced this issue Mar 13, 2025
@leehinman @andrzej-stencel @carsonip
[exporter/elasticsearch] add dynamic document pipeline support (#37860)
4272468 
#### Description

This PR adds a new config option `logs_dynamic_pipeline` that when set
to true reads the `elasticsearch.ingest_pipeline` attribute from each
log record and uses it as the ingest pipeline in Elasticsearch.

#### Link to tracking issue

Fixes #37419

#### Testing

Added tests to verify that the document pipeline attribute can be read
from the telemetry and that the pipeline is properly forwarded to
Elasticsearch. Also asserted that when there is no document pipeline
attribute the current behavior is retained.

Manual testing that setting the option resulted in the named ingest
pipeline being run on the document in Elasticsearch.

#### Documentation

Updated the readme to mention the new `logs_dynamic_pipeline` config
option.

---------

Co-authored-by: Andrzej Stencel <[email protected]>
Co-authored-by: Carson Ip <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request exporter/elasticsearch
Projects
None yet
Milestone
No milestone
2 participants
@carsonip @leehinman