Add lifecycle option for datadog containers

[hussambhivestack]

I propose adding the option to configure a lifecycle preStop hook with a sleep delay for datadog containers.

Currently, we introduce a 30-second sleep in our services to ensure zero-downtime deployments on EKS (https://glasskube.dev/blog/kubernetes-zero-downtime-deployments-aws-eks/). However, we've noticed that the Datadog agent container shuts down prematurely, resulting in the loss of trace and metric data during the shutdown process.

By enabling a configurable preStop hook, we can delay the termination of Datadog containers, allowing them to continue collecting and forwarding telemetry data until the primary services have shut down gracefully.

[celenechang]

Hi @hussambhivestack , thanks for opening the issue. Just to make sure I understand, in your set up are you running the Datadog Agent in a sidecar container?

[hussambhivestack]

We have datadog running as a daemonset on our EKS cluster. We are also using Karpenter for spot instance so we recycle nodes for cheaper ones frequently.
The problem is that the datadog agent on the instance that are recalled are terminating earlier than our applications. This is causing a gap in metrics. With a preStop hook I would like the delay the datadog agent until after the applications have gracefully shutdown.