Beats do not log to stderr when running in a container or systemd environment

[belimawr]

For confirmed bugs, please report:

• Version: 8.15.0 and later versions
• Operating System: All
• Discuss Forum URL: https://discuss.elastic.co/t/no-output-to-stdout-stderr-from-docker-image-after-upgrading-to-v8-15-x/
• Steps to Reproduce: Run any beat (Filebeat as an exmaple) in a container or with -environment container

Reproducing with Docker:

docker run --rm docker.elastic.co/beats/filebeat:8.14.3

Will log to stderr

docker run --rm docker.elastic.co/beats/filebeat:8.15.2

Will produce no logs on stderr.

When using -environment systemd this issue can also be observed.

Workaround

Pass the -e flag when running the Beat or add logging.to_stderr: true to your configuration file.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mauri870]

@belimawr I'm confused, isn't this behaving as expected according to #38767?

6. Look at the logs (container) and assert no event data is going to stderr:

If you want to see the event logs going to stderr, uncomment the following lines from the configuration:

     # "-E",
     # "logging.event_data.to_stderr=true"

Edit: In a call with Tiago it is clear that this was not supposed to affect the container behavior.

[mauri870]

This seems to be a bug introduced in elastic/elastic-agent-libs#208, the default location for container logs was silently changed to files instead of stderr. It should be fixed by elastic/elastic-agent-libs#236.

[mauri870]

Reopening as we need to track the PRs updating the lib in Beats as well as backport PRs.

[mauri870]

FYI I updated this ticket to mention that -environment systemd is also affected.