Patching k8 cluster, we consistently encounter "MountVolume.MountDevice " on one of the nodes. All others work OK.

[hbxx]

environment Ubuntu 22.04.5 and HPE CSI 2.52.

We are running a patching process that drains/cordons each node, patches it, reboots, then uncordons
On exactly one host in the cluster (which one appears to be random), we see the following error:
MountVolume.MountDevice failed for volume "pvc-fd5b1ac4-aff8-445b-b36b-38a288793c63" : rpc error: code = Internal desc = Failed to stage
volume pvc-fd5b1ac4-aff8-445b-b36b-38a288793c63, err: rpc error: code = Internal desc = Error creating device for volume pvc-fd5b1ac4-aff8-
445b-b36b-38a288793c63, err: device not found with serial 60002ac000000000000002bc000274c7 or target
We observe removing the VLUNs from the previous host run, but VLUN registration for the new host just don't happen.
Moving the broken pods to a different host fixes the issues as the VLUNs are created for that host
Moving the pods back to the first host works correctly (the VLUNs are removed/recreated as expected

[datamattsson]

Certain precautions need to be taken in scenarios where rolling node upgrades are being executed.

Since the HPE CSI Driver controller Pod is a single replica Deployment (do NOT attempt to scale it, it's not supported) the controller can NOT be co-mingled with the workloads subject to rolling node upgrades which has PVCs.

Draining a node with the controller and workloads will show symptoms that you're mentioning and shuffling around Pods should solve the issues.

[hbxx]

we tested with the CSI driver controller running in the management plane and not co-mingled with the workload. We had the same result

[datamattsson]

What storage platform is this and what protocol are you using? Regardless, we need HPE support engaged here to reach the proper channels to have this looked at.

[hbxx]

the environment is connected to HPE Primera using fiber channel protocol.
I opened a ticket with HPE storage. How do we get the two teams working together?

[datamattsson]

I opened a ticket with HPE storage. How do we get the two teams working together?

The support team will reach out the engineering team if needed and escalate a fix if needed.