Failed Mount Error (exit status 32) When Creating Pod with PVC using the csi-driver-smb

[hajedkh]

What happened:
Pod Creation Error with event:
Warning FailedMount 28s (x8 over 92s) kubelet MountVolume.MountDevice failed for volume "pvc-f03018a6-a450-41a9-b4f7-0609a57120e7" : rpc error: code = Internal desc = volume(viaps012-int.lia.int/archives#pvc-f03018a6-a450-41a9-b4f7-0609a57120e7##) mount "//<HOST>/archives" on "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/0a24123840085c6b252ac47fff4245d291dfda1381a23183f0c8b394e4183af5/globalmount" failed with mount failed: exit status 32 Mounting command: mount Mounting arguments: -t cifs -o dir_mode=0777,file_mode=0777,uid=1001,gid=1001,<masked> //viaps012-int.lia.int/archives /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/0a24123840085c6b252ac47fff4245d291dfda1381a23183f0c8b394e4183af5/globalmount Output: mount error(13): Permission denied Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)

What you expected to happen:
Volume provisionned and pod created

How to reproduce it:
Random after multiple volume mounts it fails for some and the pod stay blocked in ContainerCreationError.

Anything else we need to know?:
When we reschedule the pod in another node in the cluster it works fine (it happens with all worker nodes).
Environment:

• CSI Driver version: v1.16.0
• Kubernetes version (use kubectl version): v1.27.13+e709aa5
• OS (e.g. from /etc/os-release): CoreOS
• Install tools: Helm
• Others: Openshift cluster 4.14.26

[andyzhangx]

that's Permission denied error, does manual mount on the node work?

[hajedkh]

It is the exact same issue using the mount command on the node, randomly sometimes it passes sometimes no.

[andyzhangx]

then it's not the csi driver issue.

[pcking999]

@hajedkh do the shares you are connecting to happening to be DFS shares? i just had this exact same issue with the same error, permissions on the shares where fine and hadn't changed. but when i remoted into my worker node and ran journalctl -xe i noticed this error repeated multiple times. "the device mount path ... is still mounted by other references". it appears what happened was when our main file server went down for patching the DFS shares resolved to our backup file server. i could see shares where still mounted to the backup server on the worker host by running "cat /proc/mounts" and looking at the ip address. i think what happened is once the main file server was back online the system tried to mount them against the main fs when pods where brought up but it couldn't because it already had a connection to the backup file server. hence the mounted by other references error. i ended up changing everything to point to the server shares directly not though DFS but it would be nice if DFS worked seamlessly.

[kxs-jnadeau]

You can also consider the CIFS driver shipping with your worker node kernel. We've seen many, many instabilities in the CIFS driver in the shipping Linux kernel before upstream version 6.5 causing similar issues.

[hajedkh]

@kxs-jnadeau Could you please specify which versions of cifs do you recommend ? I am using CoreOS REHEL 9.2 and cifs module version is 2.37.

[kxs-jnadeau]

We have seen stability with CIFS driver as shipping by AKS on Ubuntu 22.04 but they are seemingly backporting it from kernel 6.5 on the 5.15 Linux baseline at version 2.44.

[pcking999]

just had this happen again but this time everything was pointing directly to the server shares and not using the DFS shares at all. i could see that 1 host had the shares mounted pointing to the ip address of our DR server. that doesn't seem to make a lot of sense to me since it wasn't using the DFS shares I'm not sure how that happened. i believe the permission error is a red haring and the true error to focus on is exit status 32.

Our hosts are running RHEL 8.9
Kubernetes version v1.27.2

[kingrichard2005]

I am having this same mount issue with exit status 32 and initially reported it over on the helm-samba4 repo. I just retried after updating to the latest, as of this date, v1.17 version of the csi driver and the issue still seems to persist with my deployment reporting the following

│ Output: mount error(13): Permission denied                                                                                                                           │
│ Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)                                                                         │
│   Warning  FailedMount  39s (x11 over 6m52s)  kubelet  MountVolume.MountDevice failed for volume "pvc-89f20fa4-6b1f-4169-a744-02c814804b22" : rpc error: code = Inte │
│ rnal desc = volume(mysamba-samba4.samba.svc.cluster.local/data/archive#pvc-89f20fa4-6b1f-4169-a744-02c814804b22#) mount "//mysamba-samba4.samba.svc.cluster.local/da │
│ ta/archive" on "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/87e09493baab1db6cd8f559ff4185b35ec7782bbd9e3ed67fe93b2145604815d/globalmount" failed with  │
│ mount failed: exit status 32                                                                                                                                         │
│ Mounting command: mount                                                                                                                                              │
│ Mounting arguments: -t cifs -o dir_mode=0755,file_mode=0644,vers=3.0,uid=1000,gid=1000,<masked> //mysamba-samba4.samba.svc.cluster.local/data/archive /var/lib/kubel │
│ et/plugins/kubernetes.io/csi/smb.csi.k8s.io/87e09493baab1db6cd8f559ff4185b35ec7782bbd9e3ed67fe93b2145604815d/globalmount                                             │
│ Output: mount error(13): Permission denied                                                                                                                           │
│ Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)                                                                         │
│   Warning  FailedMount  15s  kubelet  Unable to attach or mount volumes: unmounted volumes=[ptr-rw ptr-archive-rw], unattached volumes=[kube-api-access-swkml pta-se │
│ rets pta-public-key-for-uls ptr-rw ptr-archive-rw[]: timed out waiting for the condition

iMartyn/helm-samba4#5

Kubernetes version v1.26.13+k3s2

[kingrichard2005]

There is also this older issue reported in 548. The issue was closed with the reported mentioning issue was due to a firewall but they didn't provide any other details so hard to glean how they resolved it, but just sharing in case it provides any helpful hints

[kingrichard2005]

For anyone who finds this, if it helps, I was able to resolve my issue by manually setting my samba server user below, for reasons TBD the helm-samba4 chart I mentioned earlier at --version 4.0.1.2 seems to not properly set the password for users you pass in like in the below example values. This required that I manually set the user from a shell in my samba server like

/ # smbpasswd -a FAKE_USER
Registered MSG_REQ_POOL_USAGE
Registered MSG_REQ_DMALLOC_MARK and LOG_CHANGED
lp_load_ex: refreshing parameters
Initialising global parameters
Processing section "[global]"
No builtin backend found, trying to load plugin
load_module_absolute_path: Module '/usr/lib/samba/pdb/tdbsam.so' loaded

New SMB password:
Retype new SMB password:
Forcing Primary Group to 'Domain Users' for FAKE_USER

iMartyn/helm-samba4#5

# Default values for configuring the samba4 server. 
# No changes should be necessary in this section
image:
  tag: latest
persistence:
  enabled: true
  type: pvc
  pvc:
    storageClass: my-local-path
    accessMode: ReadWriteMany
# User information to access samba shares. Adjust as necessary.
samba:
  users:
  - username: FAKE_USER
    password: FAKE_PASSWORD <---Wasn't getting set even tho it's specified here??

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[knowk8s]

Hey, i am also getting the below error when trying to mount EFS filesystem on pod. Please help
`Warning FailedMount 114s (x24 over 35m) kubelet MountVolume.SetUp failed for volume "pvc-d1ebb949-e0e7-4a2d-a795-" : rpc error: code = Internal desc = Could not mount "fs-:/" at "/var/lib/kubelet/pods/7bb3a3b5-00a6-46fd-b5e5-74760a8b1aae/volumes/kubernetes.io~csi/pvc-d1ebb949-e0e7-4a2d-a795-*/mount": mount failed: exit status 32

Mounting command: mount

Mounting arguments: -t efs -o accesspoint=fsap-,tls,iam,nfsvers=4.1 fs-:/ /var/lib/kubelet/pods/7bb3a3b5-00a6-46fd-b5e5-74760a8b1aae/volumes/kubernetes.io~csi/pvc-d1ebb949-e0e7-4a2d-a795-*/mount

Output: Could not start amazon-efs-mount-watchdog, unrecognized init system "aws-efs-csi-dri"

b'mount.nfs4: access denied by server while mounting 127.0.0.1:/'

Warning: config file does not have fips_mode_enabled item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [fips_mode_enabled = False].Warning: config file does not have retry_nfs_mount_command item in section mount.. You should be able to find a new config file in the same folder as current config file /etc/amazon/efs/efs-utils.conf. Consider update the new config file to latest config file. Use the default value [retry_nfs_mount_command = True]. `