Unable to delete pod

[chilu49]

This pod is not getting deleted after its deployment is deleted. Even manual deletion is giving below error.
Error from server (InternalError): Internal error occurred: admission webhook "k-rail.cruise-automation.github.com" attempted to modify the object, which is not supported for this operation

I have added exemption but its still not working.

kubernetes version: v1.20

[Kaezon]

I just ran into the same issue trying to delete a Job pod

Kubernetes version: 1.19
k-rail version: v3.5.1

[chilu49]

I installed k-rail using helm.
So, i ended up doing helm uninstall k-rail after which I was able to delete the pod.
Not sure if this works for you or not.

[chilu49]

I just ran into the same issue trying to delete a Job pod

Kubernetes version: 1.19
k-rail version: v3.5.1

I installed k-rail using helm.
So, i ended up doing helm uninstall k-rail after which I was able to delete the pod.
Not sure if this works for you or not.

[Kaezon]

I installed k-rail using helm.
So, i ended up doing helm uninstall k-rail after which I was able to delete the pod.
Not sure if this works for you or not.

Oh, yes. I can remove k-rail to delete the pod; however, deleting k-rail every time I run a job doesn't seem like an ideal way to administrate my deployments :P

[tobymilne-haven]

I had the same issue as soon as I switched to reportonly false, in the end i hacked the helm chart, and disabled the webhook for "DELETE", that allows pods to be deleted, but i suspect rules about eviction etc wont work.

[Kaezon]

@chilu49 @tobymilne-haven
I created a temporary work-around in a branch: Kaezon/k-rail@9599c670d942f10547e276d8bf0056d34995b0c5

All I did was limit the webhook to processing DELETEs to CRDs. This was because it's the only thing I'm aware of that has a plugin which looks at deletes.
In the long run, I would not keep this solution in place since we probably want k-rail to be examining all requests anyways.

I'm going to see if I can figure out what the actual cause of the problem is and fix it.

[Kaezon]

After adding a lot of debug prints, I found what's happening at least.
It looks like k-rail is trying to attach some extra metadata to the DELETE request. Specifically "seccomp.security.alpha.kubernetes.io/pod:runtime/default"

I'm still not sure why though.

{"kind":"AdmissionReview","apiVersion":"admission.k8s.io/v1","request":{"uid":"f48f2da7-6e29-4d50-bd41-5843bd91a045","kind":{"group":"","version":"v1","kind":"Pod"},"resource":{"group":"","version":"v1","resource":"pods"},"requestKind":{"group":"","version":"v1","kind":"Pod"},"requestResource":{"group":"","version":"v1","resource":"pods"},"name":"banana-app-c74b498db-cps64","namespace":"default","operation":"DELETE","userInfo":{"username":"system:serviceaccount:argocd:argocd-server","uid":"3c627c97-ddae-4f57-baa4-3937d7abcdf4","groups":["system:serviceaccounts","system:serviceaccounts:argocd","system:authenticated"]},"object":null,"oldObject":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"banana-app-c74b498db-cps64","generateName":"banana-app-c74b498db-","namespace":"default","uid":"1a4921ba-456a-48ef-9e25-33a18177222a","resourceVersion":"76855","creationTimestamp":"2021-11-02T18:04:42Z","labels":{"app":"banana","pod-template-hash":"c74b498db"},"annotations":{"cni.projectcalico.org/podIP":"10.1.9.216/32","cni.projectcalico.org/podIPs":"10.1.9.216/32","sidecar.istio.io/inject":"true"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"banana-app-c74b498db","uid":"c1aa8378-137e-4c9a-a948-256236e889e4","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-11-02T18:04:42Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:sidecar.istio.io/inject":{}},"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"c1aa8378-137e-4c9a-a948-256236e889e4\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"banana-app\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8080,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:protocol":{}}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:runAsGroup":{},"f:runAsNonRoot":{},"f:runAsUser":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{".":{},"f:runAsGroup":{},"f:runAsNonRoot":{},"f:runAsUser":{}},"f:terminationGracePeriodSeconds":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2021-11-02T18:04:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-11-02T18:04:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.1.9.216\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-l4qr4","secret":{"secretName":"default-token-l4qr4","defaultMode":420}}],"containers":[{"name":"banana-app","image":"packages.bco.cudaops.com:443/docker-virtual/hashicorp/http-echo@sha256:ba27d460cd1f22a1a4331bdf74f4fccbc025552357e8a3249c40ae216275de96","args":["-listen=:8080","-text=banana"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"resources":{"limits":{"cpu":"550m","memory":"2560Mi"},"requests":{"cpu":"500m","memory":"2Gi"}},"volumeMounts":[{"name":"default-token-l4qr4","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"bcostabile-barracuda","securityContext":{"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-02T18:04:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-02T18:04:43Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-02T18:04:43Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-02T18:04:42Z"}],"hostIP":"192.168.1.189","podIP":"10.1.9.216","podIPs":[{"ip":"10.1.9.216"}],"startTime":"2021-11-02T18:04:42Z","containerStatuses":[{"name":"banana-app","state":{"running":{"startedAt":"2021-11-02T18:04:43Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"sha256:a6838e9a6ff6ab3624720a7bd36152dda540ce3987714398003e14780e61478a","imageID":"packages.bco.cudaops.com:443/docker-virtual/hashicorp/http-echo@sha256:ba27d460cd1f22a1a4331bdf74f4fccbc025552357e8a3249c40ae216275de96","containerID":"containerd://a756072973f9ba3a3d4d7b5222aaf53e1cef82ada12a5e4d82d0fa8575b7f183","started":true}],"qosClass":"Burstable"}},"dryRun":false,"options":{"kind":"DeleteOptions","apiVersion":"meta.k8s.io/v1","gracePeriodSeconds":30,"propagationPolicy":"Foreground"}}}

DEBUG: Printing list of patches
{add /metadata/annotations map[seccomp.security.alpha.kubernetes.io/pod:runtime/default]}

[Kaezon]

Ok, a little more debugging revealed it's the pod_default_seccomp_policy plugin.
I'll look at the code there next.

DEBUG: List of patches from pod_default_seccomp_policy policy
{add /metadata/annotations map[seccomp.security.alpha.kubernetes.io/pod:runtime/default]}

[funkypenguin]

I've found this problem as well, after enabling the pod_default_seccomp_policy. The pods were already running, and so thereafter any attempts to delete them caused the above-mentioned issue.

[mark-adams]

👋 The k-rail project has been deprecated and is no longer under active development. We recommend taking a look at OPA Gatekeeper to see if it might meet your needs going forward.

Thanks for your contribution(s) to the project!