Permission error - mount permission denied - during building image or when running buildah info command

[SohamChakraborty]

Issue Description

I am not able to build images in CI (Jenkins). Buildah fails with this error:

buildah build -t foobar/flyway:${GIT_COMMIT} SOME_RANDOM_DIR/.
Error: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied
time="2025-07-10T08:40:05Z" level=warning msg="failed to shutdown storage: \"mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied\""

Also buildah info fails with:

$ buildah info
Error: overlay: failed to make mount private: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied
WARN[0000] failed to shutdown storage: "overlay: failed to make mount private: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied" 

Steps to reproduce the issue

Steps to reproduce the issue

1. We are trying to build container images in CI (Jenkins)
2. We are using self hosted kubernetes cluster (kOps) running on AWS. Jenkins is running on the same k8s cluster.
3. The build goes through if we use privileged: true.
4. Now, I am trying to build the image without privileged: true. I am using this configuration:

    - name: buildah
      image: foobar/buildah:latest
      tty: true
      command:
        - cat
      securityContext:
        runAsUser: 1000
        runAsGroup: 1000
        appArmorProfile:
          type: unconfined
      volumeMounts:
        - name: storage
          mountPath: /storage
        - name: storage-conf
          mountPath: /home/build/.config/containers/
      labels:
        agent: buildah

• The only change from upstream buildah is that we have installed parallel in buildah for parallelizing the builds. But I don't think that's important because buildah info command also complains:

$ buildah info
Error: overlay: failed to make mount private: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied
WARN[0000] failed to shutdown storage: "overlay: failed to make mount private: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied" 

5. This is my storage-conf.yaml:

$ cat storage-conf.yaml 
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: storage-conf
data:
  storage.conf: |
    [storage]
    driver = "overlay"
    runroot = "/storage/run/containers/storage"
    graphroot = "/storage/.local/share/containers/storage"
    rootless_storage_path = "/storage/.local/share/containers/storage"
    [storage.options]
    pull_options = {enable_partial_images = "true", use_hard_links = "false", ostree_repos=""}
    [storage.options.overlay]

Describe the results you received

The build fails with

Error: overlay: failed to make mount private: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied
WARN[0000] failed to shutdown storage: "overlay: failed to make mount private: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied" 

Describe the results you expected

The build to go through or buildah info showing correct output.

buildah version output

$ buildah --version
buildah version 1.40.1 (image-spec 1.1.1, runtime-spec 1.2.1)

buildah info output

$ buildah info
Error: overlay: failed to make mount private: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied
WARN[0000] failed to shutdown storage: "overlay: failed to make mount private: mount /storage/.local/share/containers/storage/overlay:/storage/.local/share/containers/storage/overlay, flags: 0x1000: permission denied"

Provide your storage.conf

$ grep -v "#"  /usr/share/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
"/usr/lib/containers/storage",
]
pull_options = {enable_partial_images = "true", use_hard_links = "false", ostree_repos=""}
[storage.options.overlay]
mountopt = "nodev,metacopy=on"

$ grep -v "^#" /etc/containers/storage.conf
[storage]
driver = "overlay"
runroot = "/run/containers/storage"
graphroot = "/var/lib/containers/storage"
[storage.options]
additionalimagestores = [
"/var/lib/shared",
"/usr/lib/containers/storage",
]
pull_options = {enable_partial_images = "true", use_hard_links = "false", ostree_repos=""}
[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "nodev,fsync=0"

Upstream Latest Release

Yes

Additional environment details

$ uname -r
6.8.0-1030-aws
$ arch
x86_64

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[nalind]

That 0x1000 is the storage library attempting to bind mount the graph root over itself so that it can mark mounts under it as private. Normally, that should be allowed within the user namespace that it's set up for itself inside of the container. Adding skip_mount_home = "true" to the storage configuration might be enough to work around that, or it may simply defer the error until it tries to do something else with that directory. What sort of volume is it?

[nalind]

I'm not yet sure that there's going to be anything to patch here. Converting to a discussion.

[SohamChakraborty]

Hi @nalind, thanks for your response.
I followed your advice and now storage-conf configmap looks like this:

[storage]
driver = "overlay"
runroot = "/storage/run/containers/storage"
graphroot = "/storage/.local/share/containers/storage"
rootless_storage_path = "/storage/.local/share/containers/storage"
[storage.options]
pull_options = {enable_partial_images = "true", use_hard_links = "false", ostree_repos=""}
[storage.options.overlay]
skip_mount_home = "true"

Noticed that after adding skip_mount_home = "true" I am able to run buildah info. So that's an improvement.

But I am not able to build the containers, which is the goal.

Now the error is:

Error: creating build container: internal error: unable to copy from source docker://alpine:latest: copying system image from manifest list: writing blob: adding layer with blob "sha256:fe07684b16b82247c3539ed86a65ff37a76138ec25d380bd80c869a1a4c73236"/""/"sha256:fd2758d7a50e2b78d275ee7d1c218489f2439084449d895fa17eede6c61ab2c4": unpacking failed (error: exit status 1; output: remount /, flags: 0x44000: permission denied)

Another example:

Error: creating build container: internal error: unable to copy from source docker://foobar/openjdk:14: writing blob: adding layer with blob "sha256:bb79b6b2107fea8e8a47133a660b78e3a546998fcf0427be39ac9a0af4a97e90"/""/"sha256:d0fe97fa8b8cefdffcef1d62b65aba51a6c87b6679628a2b50fc6a7a579f764c": unpacking failed (error: exit status 1; output: remount /, flags: 0x44000: permission denied)

[SohamChakraborty]

@nalind Forgot to answer this

What sort of volume is it?
It's an AWS EBS volume.

[SohamChakraborty]

Hi @nalind, how do you think this problem can be tackled?
Thanks,

[nalind]

Ultimately, find out why the mount(2) call is producing an error.
What sort of filesystem is the volume?
Does the user the container runs as have write access to it?
Is there other mandatory access control in place on the host, such as SELinux, and if so, is it logging anything about this?
Check for other limits on what the container is allowed to do, set by the runtime that ran the container, such as seccomp filters and limits on kernel capabilities.
In the container, does unshare -Urm succeed? If the version of unshare is new enough to recognize the --map-auto flag, does unshare -Urm --map-auto succeed? In either case, from inside of the shell that it spawns, does mount --bind --make-private /storage/.local/share/containers/storage succeed?