Podman not working in Forgejo runner jobs

[nalcabio-tom]

The error is:

    cannot clone: Permission denied
    Error: cannot re-exec process

From my research this should be caused by a mismatch between /etc/subuid and /etc/subguid on the host and guest. However, as best as I can tell, I've set them up so they do match (see below) and yet the problem persists.

I'm setting up Forgejo and a Forgejo runner using incus and podman. The runner spawns CI jobs in a podman container, and our CI jobs in turn call podman. It's this last call to podman inside podman which fails. Backing up even further, I'm running Ubuntu 24.04 on an actual (not virtual) machine, which runs the latest version of incus, which runs Forgejo in a container image, and the Forgejo runner and podman in the same Debian Trixie vm. Our CI jobs also run Debian Trixie. So in both cases podman is the same, version 5.2 (we use Debian Trixie specifically to use a recent podman version).

The Forgejo runner and podman vm are started by (again, on a real machine running Ubuntu 24.04):

    sudo incus launch -c security.protection.shift=false -c security.nesting=true images:debian/trixie forgejo-runner

This vm is setup by (this is run inside the Debian Trixie vm started by incus as above):

#!/usr/bin/env bash
# -*- coding: utf-8; mode: sh -*-
set -euo pipefail
IFS=$'\n\t'

sudo useradd -m -s /bin/bash -U runner

sudo loginctl enable-linger runner

sudo apt update
sudo apt dist-upgrade

PKGS=(
    curl
    fuse-overlayfs
    podman
    slirp4netns
    uidmap
)

sudo apt install "${PKGS[@]}"

sudo systemctl --no-pager disable --now podman.socket

sudo apt clean
sudo apt autoremove --purge

sudo mkdir -p /etc/systemd/system/podman.socket.d

cat << 'EOF' | sudo tee /etc/systemd/system/podman.socket.d/nalcabio.conf > /dev/null
[Socket]
SocketUser=
SocketUser=runner
SocketGroup=
SocketGroup=runner
EOF

mkdir -p /tmp/setup.d

curl -sL https://code.forgejo.org/forgejo/runner/releases/download/v4.0.1/forgejo-runner-4.0.1-linux-amd64 \
    -o /tmp/setup.d/runner
sudo install /tmp/setup.d/runner /usr/local/bin/runner

cat << 'EOF' | tee /tmp/setup.d/runner.yml > /dev/null
log:
  level: warn

runner:
  file: /home/runner/.runner
  labels: ["debian-trixie:docker://debian:trixie"]

container:
  docker_host: "-"
  options: --device /dev/fuse:rw --security-opt label=disable
  network: "bridge"
EOF

sudo install -o runner -g runner -m 0440 /tmp/setup.d/runner.yml /home/runner

runner -c /home/runner/runner.yml create-runner-file \
    --name runner --instance http://forgejo:3000 --connect --secret "${FORGEJO_SHARED_SECRET}"

sudo chown -R runner:runner /home/runner

# https://code.forgejo.org/forgejo/runner/src/branch/main/contrib/runner.service
cat << 'EOF' | sudo tee /etc/systemd/system/runner.service > /dev/null
[Unit]
Description=Runner
After=podman.socket
Requires=podman.socket

[Service]
ExecStart=/usr/local/bin/runner -c /home/runner/runner.yml daemon
Type=exec
Restart=always

# Same error if User= and Group= are removed.
User=runner
Group=runner

WorkingDirectory=/home/runner
TimeoutSec=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

echo "runner:1000000:1000000000" | sudo tee -a /etc/subuid
echo "root:1000000:1000000000" | sudo tee -a /etc/subuid
echo "runner:1000000:1000000000" | sudo tee -a /etc/subgid
echo "root:1000000:1000000000" | sudo tee -a /etc/subgid

sudo systemctl daemon-reload

sudo systemctl --no-pager enable --now podman.socket
sudo systemctl --no-pager enable --now runner.service

rm -rf /tmp/setup.d

This works in so far as when I push to Forgejo a CI job is successfully kicked off and the pipeline is run as expected. However, when the CI job tries to invoke podman, the error at the start of this thread occurs.

On the Ubuntu 24.04 machine incus is run as root, and /etc/subuid and /etc/subguid both contain root:1000000:1000000000. This is added to /etc/subuid and /etc/guid in the Debian Trixie vm launced by incus for both the root and runner users. Podman runs as root, and the Forgejo runner runs as runner. I understand there's a security risk when there's an overlap in these values between users. I'm just trying things out right now. There's no difference if I run the runner as the runner user or as root, for example. The security options to incus launch don't seem to matter. Running things in privileged mode result in different errors.

I can't say that I know what's going on here. I've mostly just copied and pasted stuff I've found on the internet. Any help in getting podman in podman to work would be greatly appreciated. Thank you

[carrete]

I have the same problem

[Luap99]

This error happens even before any of the subuid/sugid files come in play. It sounds like you did not allow user namespaces.

[nalcabio-tom]

Thanks. Unshare works on the Ubuntu host, and in the Debian Trixie vm started by incus. Unshare fails inside a podman container run on this vm.

I thought I had disabled apparmor, but I'll double check

[rhatdan]

If the container is being launched by Docker, then the issue is Docker's seccomp rules do not allow unshare. So you need to turn off the secccomp filtering.

[nalcabio-tom]

Thanks @rhatdan. Does the same hold true for podman?

[rhatdan]

yup

[rhatdan]

You can launch the container with

docker run --security-opt seccomp=unconfined and turn off seccomp.

[nalcabio-tom]

Unshare works when apparmor is disabled, i.e. apparmor=0 is added the the kernel boot parameters of the Ubuntu host. No other options are required, like the privileged or nesting security options to incus when the Debian trixie vm is created.

I assume/hope there's a better way to get around this, but the sysctl commands described in https://ubuntu.com/blog/ubuntu-23-10-restricted-unprivileged-user-namespaces didn't work for me

[sbrivio-rh]

With rootless networking, you might be hitting this on top: https://bugs.launchpad.net/ubuntu/+source/apparmor/+bug/2077158

It would be nice if an Ubuntu user could fix it on 24.04, it's rather trivial. It's fixed on Ubuntu 24.10 by the way.

[carrete]

Same. Thanks @nalcabio-tom

[nalcabio-tom]

👍 @carrete

[nalcabio-tom]

This is way too permissive, but at least this works and is something I can experiment with later to tighten up the security restrictions. For any time travelers who might come across this discussion, what I ended up with is:

The Debian Trixie vm created by incus:

    sudo incus create -c security.privileged=true images:debian/trixie forgejo-runner

Then this vm is setup by:

#!/usr/bin/env bash
# -*- coding: utf-8; mode: sh -*-
set -euo pipefail
IFS=$'\n\t'

sudo useradd -m -s /bin/bash -U runner

sudo loginctl enable-linger runner

sudo apt update
sudo apt dist-upgrade

PKGS=(
    curl
    fuse-overlayfs
    podman
    slirp4netns
    uidmap
)

sudo apt install "${PKGS[@]}"

sudo systemctl --no-pager disable --now podman.socket

sudo apt clean
sudo apt autoremove --purge

sudo mkdir -p /etc/systemd/system/podman.socket.d

cat << 'EOF' | sudo tee /etc/systemd/system/podman.socket.d/nalcabio.conf > /dev/null
[Socket]
SocketUser=
SocketUser=runner
SocketGroup=
SocketGroup=runner
EOF

cat << 'EOF' | sudo tee /etc/containers/registries.conf.d/99nalcabio.conf > /dev/null
[[registry]]
location = "${FORGEJO_REGISTRY}"
insecure = true
EOF

cat << 'EOF' | sudo tee /usr/local/bin/podman-login > /dev/null
#!/bin/sh
echo "${FORGEJO_PASSWORD}" | podman login --username "${FORGEJO_USERNAME}" --password-stdin "${FORGEJO_REGISTRY}"
EOF

sudo chmod a+rx /usr/local/bin/podman-login

mkdir -p /tmp/setup.d

curl -sL https://code.forgejo.org/forgejo/runner/releases/download/v4.0.1/forgejo-runner-4.0.1-linux-amd64 \
    -o /tmp/setup.d/runner
sudo install /tmp/setup.d/runner /usr/local/bin/runner

cat << 'EOF' | tee /tmp/setup.d/runner.yml > /dev/null
log:
  level: warn

runner:
  file: /home/runner/.runner
  labels: [
    "debian-trixie:docker://docker.io/library/debian:trixie",
    "forgejo-builder:docker://${FORGEJO_REGISTRY}/nalcabio/containers/forgejo-builder:trixie"
  ]

container:
  docker_host: "-"
  options: --device /dev/fuse:rw --security-opt label=disable
  privileged: true
  network: "host"
EOF

sudo install -o runner -g runner -m 0440 /tmp/setup.d/runner.yml /home/runner

runner -c /home/runner/runner.yml create-runner-file \
    --name runner --instance http://forgejo:3000 --connect --secret "${FORGEJO_SHARED_SECRET}"

sudo chown -R runner:runner /home/runner

cat << 'EOF' | sudo tee /etc/systemd/system/runner.service > /dev/null
[Unit]
Description=Runner
After=podman.socket
Requires=podman.socket

[Service]
ExecStart=/usr/local/bin/runner -c /home/runner/runner.yml daemon
Type=exec
Restart=always

User=runner
Group=runner
WorkingDirectory=/home/runner
TimeoutSec=0
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload

sudo systemctl --no-pager enable --now podman.socket
sudo systemctl --no-pager enable --now runner.service

chmod a+rx /run/podman

rm -rf /tmp/setup.d

And as previously mentioned apparmor is disabled by adding apparmor=0 to the kernel boot parameters.

Once I got past the user namespace problem, I encountered other problems, like dns problems which required the forgejo runner to use a host network, and other problems related to the overlay fs which required the privileged options above.

[carrete]

Fantastic @nalcabio-tom !