running rootful buildkitd inside a container in rootless mode: `runc run failed: no cgroup mount found in mountinfo`

[apostasie]

Description

I have a rather complex build system, and for now cannot reduce it to a simple reproducer.

Pretty much, buildkitd is started inside a container, with --privileged, and then buildctl is used directly to communicate with it.

When the buildkitd container is started with docker, things work fine.

When the buildkitd container is started with nerdctl (rootless), any RUN instruction fails immediately with:
runc run failed: no cgroup mount found in mountinfo

I am not a cgroup specialist... @AkihiroSuda does the above ^ ring a bell for you by any chance? Any specific place I should look into?

If not, I will come up with a reproducer eventually, but need time to simplify it.

Steps to reproduce the issue

No response

Describe the results you received and expected

na

What version of nerdctl are you using?

2.0.0

Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

Host information

No response

[apostasie]

Here is a repro:

Dockerfile:

FROM debian
RUN echo whatever

mkdir sock
nerdctl run --name bk -d --network host -v $(pwd)/sock:/run/buildkit --privileged moby/buildkit

# Confirm this is working:
BUILDKIT_HOST=unix://$(pwd)/sock/buildkitd.sock buildctl debug workers

# Try to build
BUILDKIT_HOST=unix://$(pwd)/sock/buildkitd.sock nerdctl build -f Dockerfile .

Result:

 > [2/2] RUN echo whatever:
0.009 runc run failed: no cgroup mount found in mountinfo

This works if buildkitd was started with docker (rootful):

docker run --name bk -d --network host --privileged moby/buildkit
BUILDKIT_HOST=docker-container://bk buildctl debug workers
BUILDKIT_HOST=docker-container://bk nerdctl build -f Dockerfile .

Note that:

• this is a rootless issue - rootful works fine
• this is a host network issue - non-host works fine
  Version:
• nerdctl containerd runc > main
• lima VM on macOS

Maybe --privileged (to fully work) does need root (makes sense?) - but then, if that is the case and --privileged is not meant to be used with rootless, we should have a better error message.

Note that the error message is coming from runc, and somewhat hints at a cgroup mount/permission issue?

@AkihiroSuda my understanding of cgroup is rather lame - do you have an opinion on this?

[apostasie]

Ok... I am too lazy to wait for answers, so, doing my own debugging 😂

This is because:

https://github.com/containerd/nerdctl/blob/main/pkg/containerutil/container_network_manager.go#L743-L752

So, back to you @AkihiroSuda since you did look into this problem overall with opencontainers/runc#1869

What do you think?

[AkihiroSuda]

https://github.com/containerd/nerdctl/blob/main/pkg/containerutil/container_network_manager.go#L743-L752

Let's use a permalink: https://github.com/containerd/nerdctl/blob/v2.1.2/pkg/containerutil/container_network_manager.go#L743-L752

For recent kernel we could just bind-mount /sys from the host, but it is not likely to solve the issue either?

Maybe BK should just disable cgroups when running in UserNS && sysfs is unavailable?

cc @tonistiigi