Default cosa build behavior changed to not produce qemu image

[jmarrero]

Bug Report

cosa build does not build qemu image anymore by default

Environment

What operating system is being used to run coreos-assembler?
Fedora 42

What operating system is being assembled?
CoreOS

Is coreos-assembler running in Podman or Docker?
In a toolbox

If Podman, is coreos-assembler running privileged or unprivileged?

Expected Behavior

By default in the past cosa build created a oci archive and a qemu image, I expect the default continue specially when I continue leveraging cosa run

Actual Behavior

No qemu image by default and running buildextend-qemu requires privileges for some reason.

Reproduction Steps

1. cosa build
2. cosa run

Other Information

[jmarrero]

I am not sure if this was intentional, but docs still say otherwise.

[dustymabe]

Hey Joseph. This was an intentional change because we are now maintaing two separate build paths (one is building via container and the other is legacy directly via rpm-ostree) and limiting the scope of cosa build made it easier to proxy in the new flow. Sorry it broke you.

Can you point us at what docs need to be updated and we'll get those updated?

[dustymabe]

No qemu image by default and running buildextend-qemu requires privileges for some reason.

I'd be interested to know about the required privileges you are talking about. Nothing really changed in #4255 regarding how the qemu is built.

[jmarrero]

Can you point us at what docs need to be updated and we'll get those updated?

https://github.com/coreos/coreos-assembler/blob/main/docs/cosa.md#buildextend-commands

By default, the build command will build an OSTree and a QEMU image as base artifacts. Those commands extend those artifacts to make them functional on other platforms or cloud providers:

[jmarrero]

No qemu image by default and running buildextend-qemu requires privileges for some reason.

I'd be interested to know about the required privileges you are talking about. Nothing really changed in #4255 regarding how the qemu is built.

I had to do sudo let me try do a clean --all and rebuild to see if I can get the error again.

[jmarrero]

jmarrero@toolbx ~/d/cosa-build> cosa buildextend-qemu
Config commit: 15ca2a7a0939c67253ff7c7925e2f86438a9cbc0
Using manifest: /var/home/jmarrero/development/cosa-build/src/config/manifest.yaml
Will build qemu
Estimating disk size...
Disk sizes: metal: 2934M (estimated), cloud: 10240M
Formatting 'cache2.qcow2.tmp', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=48318382080 lazy_refcounts=off refcount_bits=16
++ arch
+ osbuild-mpp -D 'arch="x86_64"' -D 'artifact_name_prefix="fedora-coreos-42.20250909.dev.0"' -D 'build_version="42.20250909.dev.0"' -D 'ociarchive="/var/home/jmarrero/development/cosa-build/builds/42.20250909.dev.0/x86_64/fedora-coreos-42.20250909.dev.0-ostree.x86_64.ociarchive"' -D 'osname="fedora-coreos"' -D 'container_imgref="ostree-remote-registry:fedora:quay.io/fedora/fedora-coreos:testing-devel"' -D 'container_repo=""' -D 'container_tag=""' -D 'extra_kargs="mitigations=auto,nosmt"' -D metal_image_size_mb=2934 -D cloud_image_size_mb=10240 -D rootfs_size_mb=2421 /usr/lib/coreos-assembler/osbuild-manifests/coreos.osbuild.x86_64.mpp.yaml /tmp/osbuild-wnxx.json
+ set +x
+ osbuild --out cache/osbuild/out --store cache/osbuild/store --cache-max-size 20GiB --checkpoint deployed-tree --checkpoint tree --checkpoint raw-image --checkpoint metal --checkpoint metal4k --export=qemu /tmp/osbuild-wnxx.json
starting /tmp/osbuild-wnxx.jsonPipeline source org.osbuild.curl: e0eff42291b44563a8cfdd80e6983e2abdc31842b14709a3b2fcb5694442c9bc
Build
  root: <host>
source/org.osbuild.curl (org.osbuild.curl): Downloaded file:///var/home/jmarrero/development/cosa-build/builds/42.20250909.dev.0/x86_64/fedora-coreos-42.20250909.dev.0-ostree.x86_64.ociarchive

⏱  Duration: 1757442889s
Pipeline oci-archive: dd1ec323e7b4ded9b375d4eaa4791432d1fe7a9b30d700e28bf1cfb393e64810
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.copy: dd1ec323e7b4ded9b375d4eaa4791432d1fe7a9b30d700e28bf1cfb393e64810 {
  "paths": [
    {
      "from": "input://inlinefile/sha256:0443f119537015bc698f3214309f2233a3862b6692c05b7825b82dc20d9195b7",
      "to": "tree:///coreos.ociarchive"
    }
  ]
}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
copying '/run/osbuild/inputs/inlinefile/sha256:0443f119537015bc698f3214309f2233a3862b6692c05b7825b82dc20d9195b7' -> '/run/osbuild/tree/coreos.ociarchive'

⏱  Duration: 0s
Pipeline deployed-tree: bf11969d381d6f5a125a287c9d296e6bc905d12d58c20eb7aa830eb276817000
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.container-deploy: bf11969d381d6f5a125a287c9d296e6bc905d12d58c20eb7aa830eb276817000 {}
input/images (org.osbuild.containers): target /var/home/jmarrero/development/cosa-build/cache/osbuild/store/tmp/buildroot-tmp-yir30cyp/inputs/images
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
time="2025-09-09T18:34:50Z" level=info msg="Not using native diff for overlay, this may cause degraded performance for building images: kernel has CONFIG_OVERLAY_FS_REDIRECT_DIR enabled"
Getting image source signatures
Copying blob sha256:bd71fc15517e880a77d3c6c79cee789736095794cb72f36ee9c630b7ee6ac747
Copying blob sha256:67111ed7b9dd06b7a4a6f0aa951d35b36b33bb95f09cef4815af09c90b3157b7
Copying blob sha256:aab5ce09aecad74a10ca3002622fbde68a1fd068fc08b79bd8035aaaf3a6b980
Copying blob sha256:8bc9c60c25d52413f3a104e9c01f694794ad730de2c975db704f51f4b567377f
Copying blob sha256:e6cc968aa24746382bdc899967de238461af0646667e33523497be574dda64f2
Copying blob sha256:de601d32a114119a6fe7b910c26a386d754bb1199623fa0b3700ea964d6c325c
Copying blob sha256:f28cc6a80a3707aced6a4ab6d37550671cb22c301f00eff640c5a8e64198664f
Copying blob sha256:913e2509940e3515109777972579e2a6f3c439fd6bb8951c3e26a0c3d99a0f2a
Copying blob sha256:75f485213b72f37e91e8e95e2dfb95fc5ef4b6817b29676fcfb3365a99f0069d
Copying blob sha256:314e495f6a8d5434418d0d0e0b999f9aab9828f479bb565318be1c8f6c6bca96
Copying blob sha256:3eedd84a6447f46e15ea69bd5f25042b4c17e235bf092101ab48c6682d2b4ee6
Copying blob sha256:f29802b0738633ca380855b417ce0f9cd7a3e0aca386857acd5c73fd04d79fc9
Copying blob sha256:5083f4f24d4fdf154737aa106f8fc7baf50a584eb7f09679e62f7f9f6502a994
Copying blob sha256:6096e4c005770d22bf099365f653b06881ec8ff38e8cccbca7decd7d7ba41542
Copying blob sha256:2d5a2604bc195fee37f7637949ff02eafe62b64cd2734bafde9d6e57d0760e1b
Copying blob sha256:9662698f238f4700b3982bd44c81370b9ff970111f2bba212d32cc9ee86ee0dc
Copying blob sha256:2a5941eae47fbebdf3b59044c0143ae76012844acee33ae60ccf6a01d6f45582
Copying blob sha256:a000ed027729a237832ce850427c54a59af05e9d3007bf1ea33dcd3f7cd6ed83
Copying blob sha256:71dca203c29e18d9ec5c2c8b170e9981bd5524ae33fd9b63145b0f968b44c636
Copying blob sha256:534b50511e40c15743bea15b5a23a04230a970b243e0a41843b1da6e9ab95c8c
Copying blob sha256:1ebcac6db8afde8d8d45b6c7e67c206effb3a38db873f41dfd05d421ec7e9dd6
Copying blob sha256:6b74e081e2f991b66aed49f42dc75503d0bf2203eceb10611898df4336d24d36
Copying blob sha256:a4796f8760c52ad525056a61adcb0ac32c348948d9d7545ee671a78bf5c2fa1a
Copying blob sha256:ac154b16357e7f6f60624184fc4cd3ad2851626f8a701316992e9c22eae35fb8
Copying blob sha256:bbb70c64dba5f00427a329c22eb27cb57cfdc346d70c278ad864114b5d11b4ed
Copying blob sha256:376c259dde7b7e30066aecc48518c608eef5fc99a474f3e9d6cf2c6db223f752
Copying blob sha256:779e564d94ce56ca002907bee79ee48dbc8862984e39787fe02773b63dec6fb9
Copying blob sha256:5c8180879110b8fa189fa4594bc8cf367073dbe7e0b876be8e531e44dba21fda
Copying blob sha256:5b9a19d4f4822cb9658f7ee069844d222013a8ff35db683dea7094ea82490235
Copying blob sha256:7f7c7a41ca0833da332a7477239f752697221b7eadcddbeaddbbf829249dcb41
Copying blob sha256:136b60e484e9d6618571cb5e4ce24edd556bf367002685a2bb2982f6fe2d08c4
Copying blob sha256:cbea00aed297f72862ec4027e3838976d68ece011cf5ce32438a5dc9274b5aca
Copying blob sha256:d11efc94ccfde4884c5127197768ba0e16b16f466925b3ebaa4ce869dad50423
Copying blob sha256:d3b04dbbafc2045a2f4d671f37e4e5b233bf6e3e641f1665778ab9b11d2d4c97
Copying blob sha256:9928065a2832cbd1856c1df31f378a8e756cc9d962ef554978646b736d3ce089
Copying blob sha256:d716b416c3ebe719c7ca8d56e32f52590d2542690bc019bcd2102a7fe59ee534
Copying blob sha256:289060938eb6ebe91dd1e61172af98dbcba08a1a19b9c0252dbf1fea6843b2a3
Copying blob sha256:af3fdf5ef56909e9aa0c46ce9c8cbd12b2731b06622e88f0b7802e5444c32a4f
Copying blob sha256:6231b3f9bf58dfc64e03c07bd904a1aab76433c380022e3d7102c00ce94558e2
Copying blob sha256:730a494129dd42da1a5e43a0cfd4a8a536ea35de581a9f0f36073200f70f3079
Copying blob sha256:2f590ca57d6313f2e38bf6c185edbac5f818333b3cd50862fe0c27a8801341fc
Copying blob sha256:61a687150603f6552990774bbfebeeca3f67c46d2557dce5ac79f6ff0a7004fd
Copying blob sha256:15b921064267745b77dea303704d2ed54f3706feb538b2c7c42376d90ed0c4b6
Copying blob sha256:050919622b464b877b8455ce519fe695f5d073ace2a176f61a6d5f92d8d1e355
Copying blob sha256:c0a057cb1dafb318a8399ab1d99592674c91bbc77b5b230d93584bc726ade300
Copying blob sha256:84e688a9087fbae0161f581db0b040ed8f4460eb0a93ee50c99a88d093063a45
Copying blob sha256:22e547e151ed4bc82eeec8d8e8603cc4efbb4be3f2224ad6ec797146f8e89ac3
Copying blob sha256:fddf3437c9e3123e409996bf51ab8aec602dc530b00bca19e109283c484282b8
Copying blob sha256:ce22fe1e347571d11b21d121277a71e3af886945727638842ba71f850da72413
Copying blob sha256:ff3b1c206feb1e67c0ae90d4db610c59328891c83d0210d9e9258d6b74f7ee78
Copying blob sha256:9dad063a624b62064bf25dbbc2e802e472d636056f661f2a0be73efd8a4da98b
time="2025-09-09T18:34:51Z" level=fatal msg="trying to reuse blob sha256:bd71fc15517e880a77d3c6c79cee789736095794cb72f36ee9c630b7ee6ac747 at destination: looking for layers with digest \"sha256:bd71fc15517e880a77d3c6c79cee789736095794cb72f36ee9c630b7ee6ac747\": loading additional layer stores: open /usr/lib/containers/storage/overlay-layers/layers.lock: open /usr/lib/containers/storage/overlay-layers/layers.lock: read-only file system"
Error: tmp-container-deploy-38137590811884: image not known
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.container-deploy", line 55, in main
    subprocess.run(cmd, check=True)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['skopeo', 'copy', 'oci-archive:/tmp/tmpap6plug9/image', 'containers-storage:tmp-container-deploy-38137590811884']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.container-deploy", line 65, in <module>
    r = main(args["inputs"], args["tree"], args["options"])
  File "/run/osbuild/bin/org.osbuild.container-deploy", line 48, in main
    with contextlib.ExitStack() as cm:
         ~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib64/python3.13/contextlib.py", line 619, in __exit__
    raise exc
  File "/usr/lib64/python3.13/contextlib.py", line 604, in __exit__
    if cb(*exc_details):
       ~~^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/contextlib.py", line 482, in _exit_wrapper
    callback(*args, **kwds)
    ~~~~~~~~^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.13/subprocess.py", line 577, in run
    raise CalledProcessError(retcode, process.args,
                             output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['podman', 'rmi', 'tmp-container-deploy-38137590811884']' returned non-zero exit status 1.

⏱  Duration: 1s
manifest /tmp/osbuild-wnxx.json failed
Failed

Filesystem      Size  Used Avail Use% Mounted on
/dev/vdb1        45G  1.7G   44G   4% /var/home/jmarrero/development/cosa-build/cache

failed to execute cmd-buildextend-qemu: exit status 1

[jlebon]

Is your cosa up to date? That should be fixed by 55d5540.

[jmarrero]

I built it from source last week (5 days ago according to podman). I wonder if something is off because I build in a Containerfile from source...

 jmarrero@toolbx ~/d/cosa-build> ls -la /usr/lib/containers/storage/overlay-images
ls: cannot open directory '/usr/lib/containers/storage/overlay-images': Permission denied

jmarrero@toolbx ~/d/cosa-build> sudo ls -la /usr/lib/containers/storage/
total 0
drwxr-xr-x. 1 root root 56 Sep  4 15:02 .
drwxr-xr-x. 1 root root 14 Sep  4 15:02 ..
drwx------. 1 root root 22 Sep  4 15:02 overlay-images
drwx------. 1 root root 22 Sep  4 15:02 overlay-layers

jmarrero@toolbx ~/d/cosa-build [1]> sudo chmod 755 /usr/lib/containers/storage/overlay-images
jmarrero@toolbx ~/d/cosa-build> sudo chmod 755 /usr/lib/containers/storage/overlay-layers/
jmarrero@toolbx ~/d/cosa-build> ls -la /usr/lib/containers/storage/
total 0
drwxr-xr-x. 1 root root 56 Sep  4 15:02 ./
drwxr-xr-x. 1 root root 14 Sep  4 15:02 ../
drwxr-xr-x. 1 root root  0 Sep  4 15:02 overlay-images/
drwxr-xr-x. 1 root root  0 Sep  4 15:02 overlay-layers/

I can try rebuilding the toolbox if it's helpful in another moment, but there is a chance I am doing something not much people are doing.

[dustymabe]

after you ran the chmod commands did it work?

[jmarrero]

after you ran the chmod commands did it work?

yeah

[jlebon]

Opened #4322 related to this.