User Request: Support for pod state disruptions

[nikos912000]

Is your feature request related to a problem? Please describe.
Pod state failures (e.g. graceful/non-graceful deletion) are common disruptions in the Chaos Engineering community.

The reasoning behind pod failures is that Kubernetes pods are ephemeral resources; they get destroyed, restarted, recreated.

This happens in many cases:

• When deploying a new version of an application
• In case the liveness probe of any container running inside the pod fails
• As a consequence of draining a node
• When the autoscaler updates the number of replicas of a deployment

Pod state disruptions can expose a number of reliability concerns including:

• Long-living pods and all the issues that may arise from them
• Cold start issues
• Scalability issues (e.g. autoscaling misconfigurations)
• Inconsistent/unknown startup times
• Uneven traffic distribution across pods
• Non-graceful shutdown
• Issues related to Java's DNS cache TTL leading to terminated pods still receiving requests
• Cascading failures
• We also wrote a blogpost on issues we found when using Kube Monkey

Describe the solution you'd like
Pod deletions can be executed in many different ways. The easiest is through the Kubernetes client which supports graceful and non-graceful deletions through its gracePeriodSeconds parameter. This is how tools like Kube Monkey and our internal controller execute that disruption.

The other option would be to do this at container level which provides more granularity. This is how Pumba executes these disruptions.

A few more implementation details/ideas:

• The level will always be pod for these disruptions.
• In the CRD this might get a bit confusing but one option is to introduce a podFailure, similar to nodeFailure, with options (e.g. graceful/non-graceful deletion).
• There is already a containers field which would allow targeting containers.

[nikos912000]

I'm happy to take a stab on this soon once we decide on these implementation details.

[Devatoria]

Hi, sorry for the very late reply but we couldn't find time to answer to this yet.

We never implemented this disruption in the controller because at Datadog we consider a pod being killed (either softly or violently) should not impact us at all as long as the pod can be rescheduled elsewhere to keep the scale. But we definitely agree that it could be an interesting feature outside of the Datadog usecase.

If you want to tackle this one, we can work together on a design document in this issue.

In the CRD, I'd go with a new field podFailure as you described (in comparison to the existing nodeFailure field). The level would always be pod (we should catch this in the admission webhook when someone tries to apply such a disruption with the level: node as it does not make sense).

In the injector, as we always try to be container engine agnostic, I'd go with signals which are often an issue (with apps in containers not catching termination signals so they end up with a SIGKILL when the graceful period is reached). The implementation of the primary use case of a podFailure disruption could be a SIGKILL sent to the target container(s) main PID which is a signal applications can't trap and could be sent for different reasons (OOMkill, end of the graceful period, etc.). An application should not cause any issue when terminated violently because it is unpredictable and uncatchable.

What do you think?

[nikos912000]

Sure thing.
In that case this will be a process/container termination that would lead to a pod termination achieving the same goal.
I'll go through the code in the following days and I will post an update for the design.