smb mount fails with mount error(5): Input/output error

[vignesh-hbk]

What happened:

I have issues mounting a NetApp SMB share onto my ubuntu 22.04 machine using csi-smb-driver. The storage has AD auth enabled and with smbclient, I am able to connect and authenticate to the storage. However when I try to mount using CIFS or CSI-smb-driver, I am running into issues.

Smb logs :
E0120 09:59:29.001256 1 utils.go:81] GRPC error: rpc error: code = Internal desc = volume(smb-01-vol) mount "//xxxx/yyy" on "/var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/9b400ffc4472d1e8195452d738fac44a1ff7719ad65879f69fda6f55599971b1/globalmount" failed with mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t cifs -o domain=fmee, //xxx/yyy /var/lib/kubelet/plugins/kubernetes.io/csi/smb.csi.k8s.io/9b400ffc4472d1e8195452d738fac44a1ff7719ad65879f69fda6f55599971b1/globalmount
Output: mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg)
dmesg log
[ 39.528383] kernel: FS-Cache: Netfs 'cifs' registered for caching
[ 39.638853] kernel: Key type cifs.spnego registered
[ 39.638869] kernel: Key type cifs.idmap registered
[ 39.639378] kernel: CIFS: Attempting to mount \xxx\yyy
[ 39.960848] kernel: kauditd_printk_skb: 53 callbacks suppressed
[ 39.960852] kernel: audit: type=1400 audit(1737111830.818:65): apparmor="DENIED" operation="capable" profile="/usr/sbin/cupsd" pid=1180 comm="cupsd" capability=12 capname="net_admin"
[ 40.188282] kernel: CIFS: VFS: \xxx\yyy Send error in SessSetup = -5
[ 40.189072] kernel: CIFS: VFS: cifs_mount failed w/return code = -5
[ 40.746703] kernel: audit: type=1400 audit(1737111831.602:66): apparmor="DENIED" operation="capable" profile="/usr/sbin/cups-browsed" pid=1259 comm="cups-browsed" capability=23 capname="sys_nice"
[ 44.913797] kernel: loop12: detected capacity change from 0 to 8 (edited)

What you expected to happen:

Mount to be successful

How to reproduce it:

Try mounting a SMB share on Ubuntu 22.04 using csi-smb driver.

Anything else we need to know?:

I am able to connect and authenticate and see the shares using smbclient on the same server. But when attempting to mount using CIFS or smb-driver, it fails with the below error.

Environment:

• CSI Driver version: v1.11.0
• Kubernetes version (use kubectl version): v1.31.3+k3s1
• OS (e.g. from /etc/os-release): Ubuntu 22.04

[vignesh-hbk]

PRETTY_NAME="Ubuntu 22.04.5 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.5 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy

[vignesh-hbk]

Could someone help with this?

[andyzhangx]

does manual mount work on the node?

mount -t cifs -o domain=fmee, //xxx/yyy /mountpath

[vignesh-hbk]

@andyzhangx - no - it does not. See the below error.
Output: mount error(5): Input/output error
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs) and kernel log messages (dmesg).

However, when we use smbclient, we are able to authenticate and list the shares etc.

[andyzhangx]

what is smbclient?

[vignesh-hbk]

@andyzhangx - It's an ftp-like client to access SMB/CIFS resources on servers - https://www.samba.org/samba/docs/4.9/man-html/smbclient.1.html

[vignesh-hbk]

@andyzhangx - Could you help with this issue? Please let me know if you need further info. Thanks.

[Akash-HBK]

@andyzhangx : Could you please help us with an input on this to proceed further in order to mount the storage on the Ubuntu server using CSI-SMB driver?

[andyzhangx]

smb csi driver only uses standard smb driver on linux, and I see smbclient is an unmaintained version, not sure how to leverage that smbclient in smb csi driver.

[vignesh-hbk]

@andyzhangx - I mentioned smbclient only to indicate that there are no connectivity or authentication issues. The core issue is that we are unable to mount the smb drive using the csi-smb driver.

[andyzhangx]

point is this smb csi driver only uses mount -t cifs command to mount file share.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten