Better systemd-repart logs on failure

[marc-hb]

This is an offshoot from:

• "Disk full" error when building GPT, default diskpart config #3223

While #3223 focuses on fixing the actual issue, I think it would be beneficial to provider better --debug logs for systemd-repart in general. But I'm not sure how. I'd be happy to submit a pull request if anyone has any suggestion.

We use mkosi as a part of something bigger, which means most users do not interact directly with mkosi and are not familiar with it. So when they are affected by the #3223 "disk full" failure, they are completely puzzled by the error below and stuck.

Formatting ESP/XBOOTLDR partitions
‣ + systemd-repart --empty=allow --size=auto --dry-run=no --json=pretty --no-pager --root=/buildroot --offline=yes --seed 24133db4-0b8b-4d5b-9fc7-67fe31430d30 /work/tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/staging/root.img.raw --definitions /work/tmp/qbuild.tmp/mkosi.repart
Automatically determined minimal disk image size as 3.2G, current image size is 3.2G.
File '/work/tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/staging/root.img.raw' already is of requested size or larger, not growing. (3.2G >= 3.2G)
Applying changes to /work/tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/staging/root.img.raw.
Successfully wiped file system signatures from future partition 0.
Successfully discarded data from future partition 0.
Formatting future partition 0.
Preparing to populate vfat filesystem.
Ready to populate vfat filesystem.
mkfs.fat 4.2 (2021-01-31)
Disk full
(mcopy) failed with exit status 1.
‣ "/usr/bin/python3 -SI /tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/tmp/tmprh39irel/mkosi/sandbox.py --proc /proc --unsetenv TMPDIR --setenv SYSTEMD_OFFLINE 0 --unshare-net --ro-bind /usr /usr --symlink usr/bin /bin --symlink usr/bin /sbin --symlink usr/lib /lib --symlink usr/lib /lib64 --dir /var/tmp --dir /var/log --unshare-ipc --ro-bind /tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/tmp/tmprh39irel/mkosi/sandbox.py /sandbox.py --dev /dev --ro-bind /etc/ld.so.cache /etc/ld.so.cache --setenv PATH /usr/bin:/usr/sbin --bind /tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/tmp/mkosi-var-tmp-e4eee4795b064f33 /var/tmp --dir /etc --dir /var --dir /tmp --dir /run --become-root --bind /tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/staging /work/tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/staging --bind /tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/root /buildroot --ro-bind /tmp/qbuild.tmp/mkosi.repart /work/tmp/qbuild.tmp/mkosi.repart --ro-bind /etc/ssl/certs/ca-certificates.crt /proxy.cacert systemd-repart --empty=allow --size=auto --dry-run=no --json=pretty --no-pager --root=/buildroot --offline=yes --seed 24133db4-0b8b-4d5b-9fc7-67fe31430d30 /work/tmp/rq_mkosi_wspaces/mkosi-workspace-y3td3nlz/staging/root.img.raw --definitions /work/tmp/qbuild.tmp/mkosi.repart" returned non-zero exit code 1.

Note they would likely be baffled by any other other systemd-repart failure, this is not so specific to #3223

I think one general issue is the lack of --debug/--verbose option in systemd-repart. The most important thing that is missing is IMHO the partition definitions. mkosi has a very useful logging.debug(json.dumps(output, indent=4)) but there's no output at all when systemd-repart fails.

When mkosi switched to systemd-repart, it made partitioning "configuration-less" which has been both a usability blessing (less manual configuration, yay!) and a curse: what disk just became full and why?!?

Because code speaks louder than words (tm), I'm sharing this debug hack below. It's for v25.3 but I tested with the current HEAD 02bf256 and it works there too. While it's obviously not ready for production, I recommend it as a local hack.

--- a/mkosi/__init__.py
+++ b/mkosi/__init__.py
@@ -3233,14 +3233,16 @@ def make_image(
             "--generate-crypttab=/etc/crypttab",
         ]
 
     for d in definitions:
         cmdline += ["--definitions", workdir(d)]
         opts += ["--ro-bind", d, workdir(d)]
 
+    subprocess.run('grep ^ ' +  str(definitions[0] / '*'), shell=True)
+
     with complete_step(msg):
         output = json.loads(
             run_systemd_sign_tool(
                 context.config,
                 cmdline=cmdline,
                 options=opts,
                 certificate=context.config.verity_certificate if verity else None,

This hack is especially useful in a "configuration-less" use case because it provides sample systemd-repart definitions to the many users affected who are hearing about systemd-repart for the first time in their life. These definitions provide the best starting point to manually workaround #3223 , they can just be copied to mkosi.repart/ and the SizeMinBytes bumped in mkosi.repart/00-esp.conf without trying to understand man systemd-repart

[marc-hb]

A big kernel is one way to trigger #3223. Another way is using mkosi.extra/ for manual kernel installation and ending up with multiple kernels there because of git version changes and the resulting uname changes

[DaanDeMeyer]

I totally agree but this is non-trvial to fix, all we get from filesystem developers is a mkfs CLI tool with only human readable output which we cannot parse for obvious reasons as there's no guarantees it won't change in the future.