Kubelet Stuck Trying to Unmount SMB Volume When Multiple PVCs referencing the same smb share on the node

[MattPOlson]

What happened:

When multiple pods mount the same SMB PVC on the same node, everything works fine initially. However, when one of the pods terminates, the kubelet continuously tries to remove the mount, but fails because it is still in use by the other running pods.

This leads to repeated errors like this in the kubelet logs:

E0318 17:59:39.117298 3433 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/csi/smb.csi.k8s.io^bdreporting.local/prod#pvc-bef1050f-315f-4d4e-b6cd-e34852707289## podName: nodeName:}" failed. No retries permitted until 2025-03-18 18:01:41.117272093 +0000 UTC m=+512314.416251152 (durationBeforeRetry 2m2s). Error: GetDeviceMountRefs check failed for volume "pvc-bef1050f-315f-4d4e-b6cd-e34852707289" (UniqueName: "kubernetes.io/csi/smb.csi.k8s.io^bdreporting.local/prod#pvc-bef1050f-315f-4d4e-b6cd-e34852707289##") on node "k8s-se-platform-02-vktnv-worker-0-l7hmt" : the device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/c610e0382b580c4274e6cfd0f86f29b1b4a9fef6865ea6ff304f804c1cdffaf8/globalmount" is still mounted by other references [...]

This situation causes the kubelet to get stuck in a loop, retrying the operation but never succeeding, since other pods on the same node are still using the SMB mount.

In some cases, this appears to contribute to node instability, including kernel panics.

What you expected to happen:

Kubelet should correctly handle multiple pods using the same SMB volume on the same node.
Unmounting the volume for one pod should not interfere with the other pods still using it.
Kubelet should not get stuck in a loop trying to unmount a volume that is still in use.
No kernel panics or node instability should result from this scenario.

How to reproduce it:

Deploy a StatefulSet or multiple pods that mount the same SMB PVC on a single node.
Verify the mount paths:

mount | grep kubernetes.io/csi
You should see the same mount path used by multiple pods.

Delete one of the pods.
Check kubelet logs (journalctl -u kubelet -f) to observe errors.
See if the kubelet continuously tries to remove the mount but fails due to "still mounted by other references."

Environment:

• CSI Driver version: smbplugin:v1.17.0
• Kubernetes version (use kubectl version): v1.30.6-dirty
• OS (e.g. from /etc/os-release): CentOS Stream CoreOS
• Kernel: 5.14.0-536.el9.x86_64
• Install tools:
• Others:

[andyzhangx]

related to #353

[gnufied]

Is this still an issue? @andyzhangx you CCed an issue which is closed, but that seems like a case of closed without a fix, rather than actually closed.

[andyzhangx]

Is this still an issue? @andyzhangx you CCed an issue which is closed, but that seems like a case of closed without a fix, rather than actually closed.

@gnufied yes, I have reopened issue: #353

[andyzhangx]

@MattPOlson
I have reviewed the two issues, this issue is different from #353, in #353, it's using multiple PVCs and each PVC referencing the same smb share, the recommendation for mitigating #353 is using one PVC for one smb share on the node.

And I have tried one PVC used by multiple pods on the same node by using this example: https://github.com/kubernetes-csi/csi-driver-smb/blob/master/deploy/example/deployment.yaml, I don't see there is pod terminating error or related error in kubelet logs, if you only use one PVC used by 3 pods on the node, and if you kill one pod on the node, there would be only NodeUnpublishVolume actitivity happen in CSI driver (see below example), I don't see there is any issue here:

I0419 03:29:11.395640       1 utils.go:76] GRPC call: /csi.v1.Node/NodeUnpublishVolume
I0419 03:29:11.395658       1 utils.go:77] GRPC request: {"target_path":"/var/lib/kubelet/pods/53100026-ac66-4fa6-99b4-d4d90004607f/volumes/kubernetes.io~csi/pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9/mount","volume_id":"smb-server.default.svc.cluster.local/share#pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9##"}
I0419 03:29:11.395714       1 nodeserver.go:118] NodeUnpublishVolume: unmounting volume smb-server.default.svc.cluster.local/share#pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9## on /var/lib/kubelet/pods/53100026-ac66-4fa6-99b4-d4d90004607f/volumes/kubernetes.io~csi/pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9/mount
I0419 03:29:11.395736       1 mount_helper_common.go:93] unmounting "/var/lib/kubelet/pods/53100026-ac66-4fa6-99b4-d4d90004607f/volumes/kubernetes.io~csi/pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9/mount" (corruptedMount: false, mounterCanSkipMountPointChecks: true)
I0419 03:29:11.395748       1 mount_linux.go:412] Unmounting /var/lib/kubelet/pods/53100026-ac66-4fa6-99b4-d4d90004607f/volumes/kubernetes.io~csi/pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9/mount
I0419 03:29:11.398064       1 mount_helper_common.go:150] Deleting path "/var/lib/kubelet/pods/53100026-ac66-4fa6-99b4-d4d90004607f/volumes/kubernetes.io~csi/pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9/mount"
I0419 03:29:11.398113       1 nodeserver.go:123] NodeUnpublishVolume: unmount volume smb-server.default.svc.cluster.local/share#pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9## on /var/lib/kubelet/pods/53100026-ac66-4fa6-99b4-d4d90004607f/volumes/kubernetes.io~csi/pvc-00a18b4a-977d-4ce8-a911-fc661a7332f9/mount successfully
I0419 03:29:11.398126       1 utils.go:83] GRPC response: {}

[MattPOlson]

@andyzhangx
Thanks for the reply! The issue that we are seeing seems to be when when multiple PVCs (e.g., from different apps or Helm releases) are created, but all reference the same SMB share (same source, sometimes with different secrets).

In this case, the CSI driver generates multiple global mount paths (one per volume ID), even though they point to the same share. This leads to the kubelet seeing multiple refs from GetDeviceMountRefs() and refusing to unmount:

Error: the device mount path "/var/lib/kubelet/plugins/.../globalmount" is still mounted by other references [...]

The fix I'm implemented does the following:

• Adds credential-based hashing to the volume ID so PVCs using the same share + same credentials resolve to the same global mount.

• Uses NodeStageVolume + bind mounts to avoid duplicate mounts entirely.

• Updates NodeUnstageVolume to safely unmount the global share only when it's no longer referenced by any bind mount.

I'd love to upstream this or collaborate on a design if there's interest. It significantly improves correctness when dealing with shared SMB paths and Helm-based workflows that create multiple PVCs for the same underlying share.

[andyzhangx]

thanks, I have changed the issue title as Kubelet Stuck Trying to Unmount SMB Volume When Multiple PVCs referencing the same smb share on the node, that would be more accurate, and also left comments in the PR, let's make the change as minimal as possible