[PoC] Filestream running as log input

[belimawr]

Proposed commit message

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Author's Checklist

• [ ]

How to test this PR locally

1. Create a log file with at least 1kb

   docker run -it --rm mingrammer/flog -n 20 > /tmp/flog.log
   

2. Start Filebeat with the following configuration

   filebeat.inputs:
     - type: log
       id: from-elastic-agent
       # be_filestream: true
       allow_deprecated_use: true
       paths:
         - /tmp/flog.log
   
   output.file:
     path: "${path.home}"
     filename: output
     rotate_on_startup: false  
   
   queue.mem:
     flush.timeout: 0
   
   logging:
     to_stderr: true
     level: debug
     selectors:
       - input
       - input.filestream
       - input.harvester
       - fswatcher

3. Look at the logs, you will see some logs from the Log input

   {
     "log.level": "debug",
     "@timestamp": "2025-09-12T12:28:06.899-0400",
     "log.logger": "input.harvester",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/filebeat/input/log.(*Log).Read",
       "file.name": "log/log.go",
       "file.line": 111
     },
     "message": "End of file reached: /tmp/flog.log; Backoff now.",
     "service.name": "filebeat",
     "input_id": "94a20b13-6927-4ff4-8f99-4f750469ed96",
     "source_file": "/tmp/flog.log",
     "state_id": "native::26052-40",
     "finished": false,
     "os_id": "26052-40",
     "harvester_id": "69128be5-d1f4-4493-935a-889d0461c95d",
     "ecs.version": "1.6.0"
   }

4. Stop Filebeat
5. Check the number of events published

   % wc -l output-*.ndjson        
   20 output-20250912.ndjson
   

6. Uncomment the be_filestream: true line in the configuration
7. Start Filebeat again, you'll see some logs from the Filestream input

   {
     "log.level": "debug",
     "@timestamp": "2025-09-12T12:31:07.586-0400",
     "log.logger": "input.filestream",
     "log.origin": {
       "function": "github.com/elastic/beats/v7/filebeat/input/filestream.(*logFile).Read",
       "file.name": "filestream/filestream.go",
       "file.line": 139
     },
     "message": "End of file reached: /tmp/flog.log; Backoff now.",
     "service.name": "filebeat",
     "id": "from-elastic-agent",
     "source_file": "filestream::from-elastic-agent::fingerprint::445d01af94a604742ab7bb9db8b5bceff4b780925c2f8c7729165076319fc016",
     "path": "/tmp/flog.log",
     "state-id": "fingerprint::445d01af94a604742ab7bb9db8b5bceff4b780925c2f8c7729165076319fc016",
     "ecs.version": "1.6.0"
   }

8. Check the number of events published, it should still be 20

   % wc -l output-*.ndjson        
   20 output-20250912.ndjson
   

Related issues

Use cases

Screenshots

Logs

[github-actions]

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

• run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit
• backport-active-all is the label that automatically backports to all active branches.
• backport-active-8 is the label that automatically backports to all active minor branches for the 8 major.
• backport-active-9 is the label that automatically backports to all active minor branches for the 9 major.