Skip to content

Consider Removing third-party pipeline invokation and definition from Nginx, Apache, AWS CloudTrail, Zeek integration #15881

@herrBez

Description

@herrBez

In 2021, the third-party pipeline was introduced in elastic/integrations#545.
From my understanding, it was developed to integrate or collect logs from third-party sources (e.g., Splunk) into Elastic.
In 2025, we removed the third-party stream input from the same integration in elastic/integrations#14995.
However, the corresponding third-party ingest pipelines (and their references in the main pipeline) were not removed.
If these pipelines are no longer needed, the main pipeline is currently performing unnecessary checks.

I am opening this issue to ask whether it is expected to have these third-party ingest pipelines and invokations:

  • Nginx
  • Apache
  • AWS CloudTrail
  • Zeek

EXAMPLE

The NGINX Access pipeline is invoking the third-party pipeline here:

https://github.com/elastic/integrations/blob/685d79a58c5665422453dc57aeb4616f7317121d/packages/nginx/data_stream/access/elasticsearch/ingest_pipeline/default.yml#L5C3-L7C49

  - pipeline:
      if: ctx.message.startsWith('{')
      name: '{{ IngestPipeline "third-party" }}'

If the third-party pipeline should not exist in the first place, we can spare the first processor in the pipeline (and remove the invoked third-party pipeline)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions