Add support for logging.event_data in beatreceivers

[khushijain21]

What does this PR do?

This PR adds support for logging.event_data for beatreceivers running from EA. The idea is that all documents containing sensitive information should be logged to separate -event.log file.

Beats' adds a field called log.type:event to log messages containing sensitive information Ref and
typedLoggerCore looks for the presence of this key-value pair. If exists, it logs to -event.log file.

Documents rejected by ES-exporter also fall in this category. We add a new field called message on typedLogger. If a log entry contains this message, it is "also" logged using typedLogger.

Why is it important?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have added tests that prove my fix is effective or that my feature works

Related issues

• Closes [beat_receivers] Redact or drop logs that could contain event data in the monitoring filestream instance elastic-agent#8846

[mauri870]

I don't see anything out of place, it's just missing a test for ConfigureOTelCore.

[khushijain21]

typedMsg is a log message snippet that we expect to contain sensitive information

typedMsg should be passed as failed to index document; input may contain sensitive data to log failed docs from ES exporter to -event.log file.
[Ref] https://github.com/open-telemetry/opentelemetry-collector-contrib//blob/main/exporter/elasticsearchexporter/bulkindexer.go#L384

[pchila]

@khushijain21 Could you please explain why a logger package in a library should concern itself with Otel or event logging filtering ? Those are concepts that are specific to the related applications (elastic-agent in this case) so I feel that we are giving the library additional responsibilities that do not belong in a generic logging package

[khushijain21]

@pchila
So elastic-agent already uses typedLoggerCore to filter events based on provided key-value pair.
https://github.com/khushijain21/elastic-agent/blob/eventLogging/pkg/core/logger/logger.go#L133

We are only extending that functionality to include certain "log message snippets". If these log message exists - we log them using typedCore. This was specifically required for OTel hence the naming of new methods, but I could make them generic.

The logp package is concerned here because where and how events are logged comes from user configuration, and we want certain log message (coming from the OTel world) to use the same core. I hope that makes sense

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: cbf2dea

History

• 💚 Build elastic/beats#750 succeeded 04e2dc6
• 💚 Build elastic/beats#749 succeeded 8efd3d3
• 💔 Build elastic/beats#746 failed 5463f8b
• 💔 Build elastic/beats#745 failed 0fecdbd
• 💔 Build elastic/beats#744 failed 4d200a4

[VihasMakwana]

Wouldn't this print all the messages for this logger to core? defaultCore and typedCore are same.

[khushijain21]

Think of this as a core on top of an existing typeLoggerCore. The core passed to this method takes care of re-routing the logs correctly. All this does is add additional fields if a message exists.

The assumption of this method is that the output is already defined. The description also explains that