rollback from ambient mode to sidecar proxy resulting in pods stuck in 'failed to find plugin \"istio-cni\"' #16748
Description
My cluster was running istio 1.24 with dataplane in sidecar mode. ( istio-injection=enabled)
I upgraded the chart and added ztunnel, and istio-cni.
I then enabled the dataplane in ambient mode. ( istio.io/dataplane-mode) I also used waypoint.
After testing, I removed ambient and waypoint annotations, undeployed waypoint, reverted back to sidecar mode.
I restarted all the pods in dataplance mode, observing all sidecar proxy running.
I then rolled back istio chart and uninstalled ztunnel and istio-cni.
Now I noticed when a pod is restarted, it is stuck in Terminating status.
Describing the pod shows the following error:
` Type Reason Age From Message
Warning FailedKillPod 3m4s (x46749 over 7d) kubelet error killing pod: failed to "KillPodSandbox" for "b3761d1f-41e9-4d24-9aaf-20e6d03e0c75" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_orcldatabase-adapters-6c7d585857-xgczn_integrationrt-1_b3761d1f-41e9-4d24-9aaf-20e6d03e0c75_0(0d63e3f7527107e232b10568c45feeacbe86a8fd07f15805dab9f8460fc2b060): error removing pod "xyz" from CNI network "cbr0": plugin type="istio-cni" name="istio-cni" failed (delete): failed to find plugin "istio-cni" in path [/opt/cni/bin /usr/libexec/cni /lib/cni]: failed to remove netns path: unlinkat /var/run/netns/3eb17c97-e80e-4066-98a8-937bab37937c: device or resource busy”`
What is causing that? What rollback step did I miss?
istioctl version
client version: 1.24.6
control plane version: 1.24.1