XFS filesystem mounting failure on NVMe

[JakeSCahill]

I am getting a failure when attempting to mount an XFS filesystem on NVMe volumes using the csi-driver-lvm. This issue is due to incompatible XFS features that are not supported by the kernel on my worker nodes. Here are the details:

• Driver: csi-driver-lvm (v0.6.0)
• Kubernetes version: v1.27.16-eks-ce1d5eb
• Kernel version: 5.10.226-214.880.amzn2.x86_64 (Amazon Linux)
• StorageClass: XFS filesystem with striped LVM configuration on NVMe devices
• LVM configuration: Using striped storage class configured with csi-driver-lvm
• XFS rrror message: Superblock has unknown incompatible features (0x20) enabled

XFS (dm-0): Superblock has unknown incompatible features (0x20) enabled.
XFS (dm-0): Filesystem cannot be safely mounted by this kernel.
XFS (dm-0): SB validate failed with error -22.

The error suggests that the XFS filesystem superblock includes features not supported by the current kernel version. Specifically, the feature flag 0x20 appears incompatible, which prevents safe mounting.
These incompatible features might be due to the mkfs.xfs options applied by the LVM CSI driver.

Steps to reproduce

1. Deploy the csi-driver-lvm with striped storage class using the following parameters in StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: csi-driver-lvm-striped-xfs
provisioner: lvm.csi.metal-stack.io
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true
parameters:
  type: "striped"
  csi.storage.k8s.io/fstype: xfs

2. Attempt to mount the provisioned XFS filesystem on a Kubernetes node with kernel version 5.10.226-214.880.amzn2.x86_64.

Expected behavior

The XFS filesystem should mount successfully without errors.

Actual behavior

The mount fails with repeated errors about incompatible superblock features, as shown in the error message above.

Suggestion

Could we allow users to specify or restrict certain mkfs.xfs options to ensure compatibility with the underlying kernel.

[ligfx]

I believe this is due to the changes introduced in f3bb9a1c0f8d31ffb12d101fd97ceeed0bdfd745 and released starting in v0.4.8. That change updated mkfs.xfs from v5.16.0 to v6.8.0, and the newer mkfs.xfs is enabling nrext64 by default.

If mkfs.xfs is run manually, this can be solved by passing the -i nrext64=0 on the command line.

Could a possible solution be allowing passing extra mkfs options in the StorageClass config?

[majst01]

Why not using a more recent worker node image with a more recent kernel ? Here they state they have at least kernel 6.1.x available: https://docs.aws.amazon.com/linux/al2023/ug/compare-with-al2-kernel.html

[ligfx]

Yes, that's another option. Unfortunately eksctl still defaults to AL2 AMIs 😕 We've seen similar issues with Azure and GCP as well.