Buildkit Rootless on Node:slim debian image is getting buildkitd: mkdir /run/buildkit: permission denied

[mjgit007]

I’m running buildkitd in rootless mode on top of node:slim image.

FROM node:slim

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl \
    ca-certificates \
    uidmap \
    slirp4netns \
    fuse-overlayfs \
    iproute2 \
    iptables \
    && rm -rf /var/lib/apt/lists/*

ARG BUILDKIT_VERSION=v0.24.0
ARG PLATFORM_ARCH=linux-arm64
ARG ROOTLESSKIT_VERSION=v2.3.5
ARG RK_ARCH=aarch64

RUN curl -sL "https://github.com/moby/buildkit/releases/download/${BUILDKIT_VERSION}/buildkit-${BUILDKIT_VERSION}.${PLATFORM_ARCH}.tar.gz" \
    -o buildkit.tar.gz && tar -xzf buildkit.tar.gz  -C /usr/local && \
    rm buildkit.tar.gz

RUN echo "Downloading RootlessKit version ${ROOTLESSKIT_VERSION}..." && \
    curl -sL  "https://github.com/rootless-containers/rootlesskit/releases/download/${ROOTLESSKIT_VERSION}/rootlesskit-${RK_ARCH}.tar.gz" \
    -o rootlesskit.tar.gz && tar -xzf rootlesskit.tar.gz -C /usr/local/bin/ && \
    chmod +x /usr/local/bin/rootlesskit && rm rootlesskit.tar.gz

ENV NODE_UID=1000
ENV NODE_GID=1000
ENV HOME=/home/node

RUN echo "node:100000:65536" > /etc/subuid && \
    echo "node:100000:65536" > /etc/subgid && \
    chown ${NODE_UID}:${NODE_GID} /etc/subuid /etc/subgid

RUN mkdir -p /etc/buildkit 

COPY start_buildkitd.sh /usr/local/bin/start_buildkitd.sh
RUN chmod +x /usr/local/bin/start_buildkitd.sh

ENV BUILDKIT_HOST=unix:///home/node/buildkit/buildkitd.sock
ENV XDG_RUNTIME_DIR=/home/node/run

USER node
WORKDIR ${HOME}

ENTRYPOINT ["/usr/local/bin/start_buildkitd.sh"]

Below is start_buildkitd.sh code.

#!/bin/bash

set -e

SOCKET_DIR="${HOME}/buildkit"
DATA_DIR="${HOME}/.local/share/buildkit"
STATE_DIR="${HOME}/run/buildkit"
SOCKET_HOST="unix://${SOCKET_DIR}/buildkitd.sock"

mkdir -p "${SOCKET_DIR}" || { echo "Error creating socket directory. Check permissions."; exit 1; }
mkdir -p "${DATA_DIR}" || { echo "Error creating home data directory."; exit 1; }
mkdir -p "${STATE_DIR}" || { echo "Error creating state directory. Check permissions."; exit 1; }


chmod 0700 "${SOCKET_DIR}"
chmod 0700 "${STATE_DIR}"

export XDG_RUNTIME_DIR="${HOME}/run"

echo "BUILDKIT_HOST $BUILDKIT_HOST"
echo "XDG_RUNTIME_DIR $XDG_RUNTIME_DIR"
echo $(ls -ltr /etc/buildkit/buildkitd.toml)
echo "Starting buildkitd (rootless mode)..."

exec /usr/local/bin/rootlesskit /usr/local/bin/buildkitd \
    --addr="${SOCKET_HOST}" \
    --root "${DATA_DIR}" \
    "$@"

When I try to run the Image , getting below error.

docker logs my-buil
BUILDKIT_HOST unix:///home/node/buildkit/buildkitd.sock
XDG_RUNTIME_DIR /home/node/run
-rw-r--r-- 1 node node 245 Oct 1 01:10 /etc/buildkit/buildkitd.toml
Starting buildkitd (rootless mode)...
buildkitd: mkdir /run/buildkit: permission denied
creating trace controller listener
main.runTraceController
/src/cmd/buildkitd/main.go:1008
main.newController
/src/cmd/buildkitd/main.go:799
main.main.func3
/src/cmd/buildkitd/main.go:362
github.com/urfave/cli.HandleAction
/src/vendor/github.com/urfave/cli/app.go:524
github.com/urfave/cli.(*App).Run
/src/vendor/github.com/urfave/cli/app.go:286
main.main
/src/cmd/buildkitd/main.go:429
runtime.main
/usr/local/go/src/runtime/proc.go:283
runtime.goexit
/usr/local/go/src/runtime/asm_arm64.s:1223
[rootlesskit:child ] error: command [/usr/local/bin/buildkitd --addr=unix:///home/node/buildkit/buildkitd.sock --root /home/node/.local/share/buildkit] exited: exit status 1
[rootlesskit:parent] error: child exited: exit status 1

Any set up I am missing here?

[AkihiroSuda]

buildkitd: mkdir /run/buildkit: permission denied
creating trace controller listener

Probably your ~/.config/buildkit/buildkitd.toml has:

[otel]
  socketPath = "/run/buildkit/otel-grpc.sock"

The path has to be removed or changed

[mjgit007]

Thank you @AkihiroSuda . I have have removed this config file ~/.config/buildkit/buildkitd.toml and running buildkitd as "exec rootlesskit buildkitd --addr=$SOCKET_HOST --root=$DATA_DIR --oci-worker-no-process-sandbox --otel-socket-path=$OTEL_SOCK" . This is working. But now when I try to build docker image using buildctl inside container getting error mkdir /run/runc: permission denied. Really appreciate inputs on this

[mjgit007]

@AkihiroSuda I am able to sort the error mkdir /run/runc: permission denied by chown 1000:1000 /run and its working fine. Is there any better secured solution available? .Also one more problem is container is running in --privileged mode and container root user is same as host user . If I try to run without --privileged mode getting error [rootlesskit:parent] error: failed to setup UID/GID map: newuidmap 35 [0 1000 1 1 100000 65536] failed: newuidmap: write to uid_map failed: Operation not permitted.
I try to set echo "kernel.unprivileged_userns_clone=1" >> /etc/sysctl.d/99-rootless.conf and sysctl --system but still no luck. How to run the container with true rootless mode by creating separate user namespace?

[AkihiroSuda]

I suggest using moby/buildkit:<VERSION>-rootless images
https://github.com/moby/buildkit/blob/master/docs/rootless.md#docker

You can use RUN apk add ... to install additional stuff there

[mjgit007]

Even I thought of it , but at the moment because of proxy issue I cannot use alpine distro . I want mainly node, rootlesskit and buildkit on the container, So I am using debian:node as base and trying to configure rootlesskit and buildkit on top of it