[FEATURE] adding annotation discovery preset

[nicolastakashi]

This pull request introduces support for annotation-based discovery of logs and metrics in the OpenTelemetry Collector Helm chart. It adds new configuration options, updates templates to handle the new presets, and ensures mutual exclusivity between certain features. Below is a summary of the most important changes:

New Feature: Annotation-Based Discovery

• Added a new annotationDiscovery preset in values.yaml that allows enabling discovery of logs and metrics based on Kubernetes annotations. (charts/opentelemetry-collector/values.yaml, charts/opentelemetry-collector/values.yamlR80-R85)
• Introduced new Helm template definitions for annotationDiscoveryConfig and applyAnnotationDiscoveryConfig to configure the OpenTelemetry Collector for annotation-based discovery. These include setting up pipelines and receivers for logs and metrics. (charts/opentelemetry-collector/templates/_config.tpl, charts/opentelemetry-collector/templates/_config.tplR269-R308)

Configuration Updates

• Updated _config.tpl to include applyAnnotationDiscoveryConfig when either logs or metrics discovery is enabled. (charts/opentelemetry-collector/templates/_config.tpl, [1] [2]
• Added a warning in NOTES.txt to ensure logsCollection and annotationDiscovery.logs are not enabled simultaneously, as they are mutually exclusive. (charts/opentelemetry-collector/templates/NOTES.txt, charts/opentelemetry-collector/templates/NOTES.txtR66-R69)

Kubernetes Role Adjustments

• Updated clusterrole.yaml to grant necessary permissions for annotation-based discovery, including access to pods, nodes, and related resources. (charts/opentelemetry-collector/templates/clusterrole.yaml, charts/opentelemetry-collector/templates/clusterrole.yamlR56-R68)

Pod Configuration

• Modified _pod.tpl to set the K8S_NODE_NAME environment variable when annotation-based discovery is enabled and to adjust volume mounts for logs collection. (charts/opentelemetry-collector/templates/_pod.tpl, [1] [2] [3]

[nicolastakashi]

Closes #1634

[nicolastakashi]

cc @ChrsMark

[ChrsMark]

Looks good overall with few comments.
Also should we add some content about this at https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/README.md?

[povilasv]

Should they be exclusive? Could we somehow by default collect both ways, as annotation one allows you to do custom parsing via file log, while logsCollection would be default and could somehow ignore the annotation based ones?

Maybe for the future though

[ChrsMark]

If both logsCollection and annotationDiscovery.logs are enabled logs of annotated Pods will be collected twice. This is log duplication that we have to avoid.

We don't have a way to configure logsCollection to skip annotated Pods :).

To collect all logs without extra parsing the following can be used:

      default_annotations:
        io.opentelemetry.discovery.logs/enabled: "true"

This will have the Collector to Collect all logs by default without any parsing applied as logsCollection would do. Then annotated Pods will be handled according to the provided annotations.

Is there any use-case this approach wouldn't cover?

[povilasv]

Could we make this configureable?

I think it useful to not collect all by default 🤔

[ChrsMark]

We could make this configurable but as explained at #1724 (comment) the reason for this is to act as drop-in replacement of logsCollection which collects all logs anyways.

[ChrsMark]

LGTM, thank's!

[TylerHelmuth]

Can you add a new test in the ci folder for this preset?

[TylerHelmuth]

Suggested change

	headers:
	[]
	headers: []

[TylerHelmuth]

Suggested change

	hostAliases:
	[]
	hostAliases: []

[TylerHelmuth]

Suggested change

	startupProbe:
	{}
	startupProbe: {}