Fail to create cluster when using btrfs as storage driver for rootless docker.

[hzhangxyz]

Hello, I am using kind by rootless docker. Recently, I add an disk and mount it to docker storage by /etc/fstab as

UUID=47e0e0e3-783d-4bd8-8b33-5e943c0b2806 /home/hzhangxyz/.local/share/docker btrfs rw,relatime,space_cache=v2,subvol=/Docker     0      0

(I clear all data before switch to new storage, and I did NOT copy previous data to the new driver so the new storage is completely empty.)

and my config is simply as

{
    "registry-mirrors": ["https://docker.mirrors.kclab.cloud/"],
    "features": {"cdi": true}
}

But I try to use kind to create a cluster, an error raised as

Creating cluster "user" ...
DEBUG: docker/images.go:58] Image: kindest/node:v1.33.1@sha256:050072256b9a903bd914c0b2866828150cb229cea0efe5892e2b644d5dd3b34f present locally
 ✓ Ensuring node image (kindest/node:v1.33.1) 🖼
 ✗ Preparing nodes 📦 📦
Deleted nodes: ["user-worker" "user-control-plane"]
ERROR: failed to create cluster: command "docker run --name user-control-plane --hostname user-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --label io.x-k8s.kind.cluster=user --net kind --restart=on-failure:1 --init=false --cgroupns=private --volume /dev/mapper:/dev/mapper --device /dev/fuse --volume=/home/hzhangxyz/Cloud/Desktop/learn-k8s/docker.io.toml:/etc/containerd/certs.d/docker.io/hosts.toml --volume=/home/hzhangxyz/Cloud/Desktop/learn-k8s/registry.k8s.io.toml:/etc/containerd/certs.d/registry.k8s.io/hosts.toml --publish=127.0.0.1:34699:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.33.1@sha256:050072256b9a903bd914c0b2866828150cb229cea0efe5892e2b644d5dd3b34f" failed with error: exit status 126
Command Output: 89fc1420282fd6bb0eae93150ce328be2eafe3fc9447b49a7340959675cc0f8f
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error creating device nodes: open /home/hzhangxyz/.local/share/docker/overlay2/e5dbad9a0faf54e9e13570cced444379cd0c320d97746f2d676d7503ce1d94be/merged/dev/mapper/control: permission denied

Run 'docker run --help' for more information
Stack Trace:
sigs.k8s.io/kind/pkg/errors.WithStack
        sigs.k8s.io/kind/pkg/errors/errors.go:59
sigs.k8s.io/kind/pkg/exec.(*LocalCmd).Run
        sigs.k8s.io/kind/pkg/exec/local.go:124
sigs.k8s.io/kind/pkg/cluster/internal/providers/docker.createContainerWithWaitUntilSystemdReachesMultiUserSystem
        sigs.k8s.io/kind/pkg/cluster/internal/providers/docker/provision.go:410
sigs.k8s.io/kind/pkg/cluster/internal/providers/docker.planCreation.func2
        sigs.k8s.io/kind/pkg/cluster/internal/providers/docker/provision.go:115
sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1
        sigs.k8s.io/kind/pkg/errors/concurrent.go:30
runtime.goexit
        runtime/asm_amd64.s:1700

The config file is

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: user
nodes:
- role: control-plane
  extraMounts: &extraMounts
  - containerPath: /etc/containerd/certs.d/docker.io/hosts.toml
    hostPath: ./docker.io.toml
  - containerPath: /etc/containerd/certs.d/registry.k8s.io/hosts.toml
    hostPath: ./registry.k8s.io.toml
- role: worker
  extraMounts: *extraMounts
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"

I am sure:

1. If I do not use btrfs as storage, everything works well.
2. If I switch to use rootful docker, everything works well.

What should I do now?