CSI Driver fails to use existing LVM Volume Group on a Fibre Channel multipath device

[sbahmani]

I'm trying to use the CSI driver to manage an existing LVM Volume Group that is set up on a Fibre Channel SAN LUN. The LUN is managed by multipathd on all my Kubernetes nodes.

The driver does not seem to provision volumes in this configuration. PVCs remain in a Pending state with the WaitForFirstConsumer message, but even after a pod is created, the volume is never provisioned.

What happened?
I have a 2TB SAN LUN available on all 20 of my worker nodes as /dev/mapper/mpathb. I have successfully created a Volume Group on this device named csi-lvm-vg.

I installed the Helm chart with devicePattern set to "" and vgName set to csi-lvm-vg, which I believe is the correct configuration for using a pre-existing VG.

When I create a PersistentVolumeClaim using the auto-generated csi-driver-lvm-linear StorageClass, the PVC never gets bound. No Logical Volume is created in my csi-lvm-vg.

What did you expect to happen?
I expected the CSI driver to recognize the existing Volume Group (csi-lvm-vg) on the multipath device. When a PVC is created and a pod consumes it, I expected the driver to automatically create a new Logical Volume of the requested size within that VG, allowing the PVC to bind successfully.

How to reproduce it (as minimally and precisely as possible)?
On all nodes: Provision a SAN LUN and configure multipathd so the device is available (e.g., as /dev/mapper/mpathb).

On one node: Create the LVM structures on the shared device:

sudo pvcreate /dev/mapper/mpathb
sudo vgcreate csi-lvm-vg /dev/mapper/mpathb
Install the Helm chart with the following values.yaml:

# csi-values.yaml
lvm:
  vgName: csi-lvm-vg
  devicePattern: ""

helm install csi-driver-lvm csi-driver-lvm/csi-driver-lvm \
  --namespace kube-system \
  --values csi-values.yaml

Create a PVC using the automatically generated StorageClass:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-claim-on-multipath
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: csi-driver-lvm-linear
  resources:
    requests:
      storage: 10Gi

Create a consumer Pod:

apiVersion: v1
kind: Pod
metadata:
  name: storage-consumer-pod
spec:
  containers:
  - name: app
    image: nginx
  volumes:
  - name: my-storage
    persistentVolumeClaim:
      claimName: test-claim-on-multipath

Observe: The PVC remains Pending and kubectl describe pvc shows the WaitForFirstConsumer event repeatedly. The pod remains in a Pending state, waiting for the volume. No LV is created in the VG.

Anything else we need to know?
My suspicion is that the driver might have an issue interacting with a VG that resides on a /dev/mapper/mpath* device instead of a simple disk like /dev/sdb. Is this a supported configuration? Are there any special permissions or host configurations required for the CSI pods to correctly manage LVM on a multipath device?

Environment
CSI Driver version: [v0.6.1]

Kubernetes version: [e.g., v1.31.0]

Node OS: [Ubuntu 24.04]

multipath-tools version: [v0.9.4]

lvm2 version: [2.03.16]

Any guidance or suggestions would be greatly appreciated. Thank you!

[majst01]

Hi,

The default vg name csi-driver-lvm uses is csi-lvm. But there is no need manually create the VG beforehand. The csi driver will create it based on disk matching the configured devicePattern

[Gerrit91]

Never tried a pre-existing VG and neither a multipath device. Can you please check the logs? The driver usually logs the commands it runs and prints errors. Not even sure what an empty device pattern should resolve to?

[mhbahmani]

Hi,
In the provided scenario, since the /dev/mapper/mpathb device is connected to a SAN storage and therefore Logical Volumes are accessible from all the nodes, isn't it possible to remove Topology awareness of csi-provisioner and lvmplugin?

By removing topology awareness, one of the goals is to remove the nodeAffinity that is added to PV resources by csi-provisioner (in controller statefulset).

I imagine removing (or changing this affinity to match a set of nodes (instead of just one node)) requires some changes in CreateVolume function. If we disable --feature-gate=Topology=true in the csi-provisioner container, there would be no pod in req.GetAccessibilityRequirements().GetPreferred(), and [0] would probably cause a panic. Since setting the topology is not mandatory in csi.volume (if I'm being correct), it can be removed. (I'm not sure about the effect of setting Topology to false. I believe that Topology=false is exactly referring to this point that the volumes are accessible on all nodes)

	node := req.GetAccessibilityRequirements().GetPreferred()[0].GetSegments()[topologyKeyNode]
	topology := []*csi.Topology{{
		Segments: map[string]string{topologyKeyNode: node},
	}}
        ...
	return &csi.CreateVolumeResponse{
		Volume: &csi.Volume{
			VolumeId:           req.GetName(),
			CapacityBytes:      req.GetCapacityRange().GetRequiredBytes(),
			VolumeContext:      volumeContext,
			ContentSource:      req.GetVolumeContentSource(),
			AccessibleTopology: topology,
		},
	}, nil

Link to code

Alternatively, there could be a config in Helm values that sets a label-based affinity that can be added to csi.Volumes.

Do you have any concerns to consider about these changes?

[mhbahmani]

I changed the code and deployed my own version of lvmplugin daemonset (with the mentioned changes and some other minor changes), and in a multi-node setup, it's working just fine. I was able to move my pod to other nodes and still mount the same data. Of course, some other commands were needed that probably should be added to mountLV in lvm.go. Commands like partprobe and calling vgActivate function, to update the partition table on the new node.

I would test some other scenarios to make sure that there wouldn't be any multi-attach or data inconsistencies. I would be glad if you could give your opinion on this, and if MRs are welcome, I can send an MR to add this feature (generally, supporting a shared device)

[femrtnz]

The default vg name csi-driver-lvm uses is csi-lvm. But there is no need manually create the VG beforehand. The CSI driver will create it based on disk matching the configured devicePattern

This didn't work for me either on GKE, my alternative solution was to manually create using vgActivate as well.