[BUG] Tags added through DD_KUBERNETES_NAMESPACE_LABELS_AS_TAGS are not always attached to logs #38413
Description
Agent Environment
Agent 7.67.0, deployed in Kubernetes through the official Helm chart.
Describe what happened:
We've setup a namespace label to be added as tag by the agent through the DD_KUBERNETES_NAMESPACE_LABELS_AS_TAGS environment variable (injected through the namespaceLabelsAsTags value in the Helm chart).
Most of the time the tag is properly reported, but sometimes for long periods of time the tag disappears from our logs. This happened for more than ten hours last night, and several shorter periods in the past two weeks too.
Describe what you expected:
I expect the tag defined in DD_KUBERNETES_NAMESPACE_LABELS_AS_TAGS to be always attached to our logs.
The long periods of tag unavailability make this feature very unreliable.
Steps to reproduce the issue:
The start and stop of the issue match with restarts of the deployment, through updates of the agent or nodepool rollouts in GKE.
I suspect this happens when the cluster agent fails to query the labels for any reason on startup, due to transient issues. It acts as if the namespace labels are only queried once at the start of the agent (or cluster agent ?) and never periodically refreshed.
The namespace itself has a static configuration, its labels haven't changed for the last two years.
Additional environment details (Operating System, Cloud provider, etc):
GKE cluster, running the latest stable GKE release: 1.31.7-gke.1212000