[Blocked] Add test for running Log input as Filestream in Beatsreceivers

[belimawr]

What does this PR do?

This PR updates the Beats dependency to a version that can run the Filestream input for a Log input configuration and adds a test for this functionality in Beats receivers.

Why is it important?

It allows us to run the Filestream input instead of the Log input for a Log input configuraiton

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Author's Checklist

• Wait for [Filebeat] Filestream running as Log input under Elastic Agent or feature flag beats#46587 to be merged
• Remove overwrites
  • dev-tools/notice/overrides.json
  • go.mod
• Update comments on testing/integration/ess/logfile.go
• Replace testing/integration/ess/logfile.go with the functionality provided by Add TempDir and NewLogFile test utilities elastic-agent-libs#369

## Disruptive User Impact

How to test this PR locally

Running integration tests

Run the test:

1. Package the Elastic Agent for your OS
2. Run TestFilebeatReceiverLogAsFilestream from testing/integration/ess/otel_log_as_filestream_test.go

Manual test

Run the Elastic Agent in Otel mode setting features.log_input_run_as_filestream.enabled: true for the filebeatreceiver you want to test.

Here is an example configuration:

otel.yml

receivers:
  filebeatreceiver:
    filebeat:
      inputs:
        - type: log
          id: a-unique-id
          allow_deprecated_use: true
          paths:
            {{.LogFilepath}}
    output:
      otelconsumer:
    logging:
      level: debug
      selectors:
        - '*'
    features.log_input_run_as_filestream.enabled: true

exporters:
  elasticsearch:
    api_key: <your key>
    endpoint: <your Elasticsearch endpoint>
    logs_index: <your index>
    sending_queue:
      enabled: true
      wait_for_result: true # Avoid losing data on shutdown
      block_on_overflow: true
    mapping:
      mode: none

service:
  pipelines:
    logs:
      receivers: [filebeatreceiver]
      exporters: [elasticsearch]
  telemetry:
    logs:
      level: DEBUG
      encoding: json
      disable_stacktrace: true

Related issues

• Blocked by [Filebeat] Filestream running as Log input under Elastic Agent or feature flag beats#46587

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b log-run-as-filestream upstream/log-run-as-filestream
git merge upstream/main
git push upstream log-run-as-filestream

[mergify]

This pull request does not have a backport label. Could you fix it @belimawr? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-./d./d is the label that automatically backports to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b log-run-as-filestream upstream/log-run-as-filestream
git merge upstream/main
git push upstream log-run-as-filestream

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: d36fc95

Failed CI Steps

• Unit tests - Windows 10
• Win2022:non-sudo:default
• Win2022:non-sudo:default
• Win2025:non-sudo:default
• Win2025:non-sudo:default
• x86_64:non-sudo: default
• x86_64:non-sudo: default
• arm:sudo: default
• x86_64:non-sudo: default - platform-ingest-elastic-agent-debian-11-1762801856
• x86_64:non-sudo: default - platform-ingest-elastic-agent-debian-11-1762801856
• x86_64:non-sudo: default - platform-ingest-elastic-agent-debian-13-1762801856
• x86_64:non-sudo: default - platform-ingest-elastic-agent-debian-13-1762801856
• x86_64:sudo: default - platform-ingest-elastic-agent-debian-11-1762801856

History

• 💔 Build #30780 failed eadfc01
• 💔 Build #30674 failed b8c6446
• 💔 Build #30590 failed 1d9b666
• 💔 Build #30564 failed 6a85797

cc @belimawr

[belimawr]

TestOtelLogsIngestion is failing, there is already a PR open to fix it.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b log-run-as-filestream upstream/log-run-as-filestream
git merge upstream/main
git push upstream log-run-as-filestream

[belimawr]

Keeping CI green is a moving target with all the recent test flakiness and conflicts in go.mod every time some change in the Beats import happen.

I'll keep this PR as is until elastic/beats#46587 gets merged and I can put this PR in its final form.