Switching to systemd-repart

[DaanDeMeyer]

After all my in progress systemd-repart PRs are merged, it should support everything we need to start using it in mkosi. There's a few different ways to go about it though which I'm opening this issue for to discuss

Backwards Compatible vs Backwards Incompatible

The way I see us using repart is at the end of the image build process. Before that we do everything in a temporary directory on the host (btrfs subvolumes if supported), and at the end we call repart, tell it to look for definition files in mkosi.repart or w.e. directory is configured, use --root <image-root> --exclude-partition esp when we call it the first time, build the UKI and call it again (with --split if requested) to add the EFI partition.

If we do the above, we can get rid of a number of mkosi's options, since they'd just have to be specified in the repart config file. Of course this breaks backwards compat. I'd make the compat break less painful by keeping --root-size intact and generating a default root partition definition if no definitions directory is provided by the user. I figure that should cover 90% of existing use cases. What I'm wondering is if we should go further and go for 100% backwards compat with all the existing options.

The options we'd be dropping:

• Verity=
• Encrypt=
• UsrOnly=
• All the filesystem specific variants of Format=
• OutputSplitRoot=
• OutputSplitVerity=
• OutputSplitVeritySig=
• ReadOnly=
• Minimize=
• Encrypt=
• ESPSize=
• SwapSize=
• HomeSize=
• SrvSize=
• VarSize=
• TmpSize=
• BiosSize=
• XbootldrSize=
• GPTFirstLBA=

Given the amount of options we'd be able to drop. I'm very much in favor of breaking compat here and delegating all of this to repart definition files.

Dropped Features

Currently, when building btrfs GPT images, we create subvolumes instead of partitions. When switching to systemd-repart, we won't be able to do this anymore since we want to be able to create the image without needing root privileges which means creating the filesystem via mkfs.btrfs which doesn't support creating subvolumes. If mkfs.btrfs ever adds support for creating subvolumes, we'll expose that via systemd-repart and users can use subvolumes instead of partitions if they prefer.

Dealing with the systemd-repart dependency

We'll be depending on a very new version of systemd-repart which won't be available in most distributions yet. While we might be able to work (barely) with older versions of systemd-repart, the experience won't be great. Ideally we'd provide users with an up to date version of systemd-repart that they can just download and use, either bundled with mkosi or via --extra-search-paths.

How we'd go about building this up to date version of systemd-repart is still an open question. I'm unsure if we can get away with building a single version that'll work on most distros or whether we'll need to build a custom version for each distro. If we need to build a custom version for each distro, it shouldn't be too hard to use mkosi and github actions to do this for us. We just need to figure out how to get the resulting artifacts somewhere onto the releases page so they become available for users. We could also extend the zipapp stuff to somehow bundle an up to date version of systemd-repart as well.

[behrmann]

I agree with basically all of this, but some thoughts:

• We should definitely cut a release before the switch (so maybe soon?) and advertise this upcoming change to gather some feedback. We should probably pin this issue, too.
• 100% compat could probably be reached if we had some super simple translation command, not hooked into any of our CLI parsing machinery, to just load the config file, look for the keys that are being phased out and write the needed repart config from that. If we encounter an error while parsing during a normal run because we encounter some unknown key we could do a clippy and ask Looks like I can't read the config. Maybe you want to run mkosi migrate?
• The repart dependency is a big one, but I'm not sure how much it will matter.
  • Arch: will always have a current systemd
  • Fedora; Will soon release 37 and I assume it will get 252 once that's released?
  • Debian: @bluca backports systemd and I'm sure 252 will make it into the release next year and I'm not sure how much we want to support oldstable
  • Mageia: I have no idea what their release schedule looks like, but cauldron seems to be rolling. Maybe @tpgxyz has some input there
  • Suse: I have no idea what their release schedule looks like, but tumbleweed is rolling, so it should get a recent systemd in a timely manner
  • RHEL family: This one is probably the biggest and I have no idea how the situation there looks. Will a recent systemd make it into RHEL9 and derivatives? I know there's also this SIG for newer versions of kernel and basic userspace plumbing, but no idea whether people can get newer pieces for their RHEL - though, I guess, this kinda defeats the purpose of running RHEL.
  • If repart could be made to be built standalone, like sysusers and tmpfiles, we could probably ship a sysext for everybody.
  • I know the story for CLI tools in flatpaks is not really good, as that's not what they're meant for, but maybe this could be an option to explore.
• The loss of the btrfs subvolume feature might be the biggest issue, since Suse is so big on those. We should probably ask people there for their opinion.

[DaanDeMeyer]

The repart dependency is a big one, but I'm not sure how much it will matter.

The other thing we could do is just add a build step to mkosi. We could add a meson build script that pulls in systemd as a meson subproject and then have a mkosi target that either depends on the host's systemd-nspawn, repart, and any other executables if they are new enough, or builds those from source otherwise. It should be trivial then to generate bin/mkosi with the added required --extra-search-paths if we build stuff from source.

Thinking further, pulling in meson would also be useful for testing, since we could move the actual image building into meson as described in #1229 and only have the tests boot the image and run the tests.

[netzego]

since you ask for it, here are my two cents. shipping an extra version of repart sounds not appealing to my ears. it is unclear to an user which version of repart is used, and more likely an user assumes the distro version is used. possibly 'build[ing] a custom version for each distro' sounds even more confusing. so basically you want to depend on an feature of repart which no distro ships jet and therefore shipping it with mkosi. but repart is part of systemd and not mkosi. but waiting for distros could took some time, ehh? i totally agree but since mkosi building distro images the lowest nominator approach will always bites.

also dropping btrfs subvolumes in favor of repart and with it dropping distros which uses subvols as default/core now or in future shrinks the overall supported distros. which is one of the major selling points.

resizable partitions/image and user builds comes with fewer distros support and shipping extra versions of systemd-repart. is there no better way?

[DaanDeMeyer]

also dropping btrfs subvolumes in favor of repart and with it dropping distros which uses subvols as default/core now or in future shrinks the overall supported distros. which is one of the major selling points.

We should still be able to do subvolumes, I'll just have to add support for creating subvolumes to systemd-repart, but the caveat is that there's no way to do that without root privileges, and we really want to get to a point where mkosi can run without needing root privileges. But there shouldn't be a problem in requiring that users run mkosi as root if they want to have subvolumes. At least until mkfs.btrfs adds support for creating subvolumes itself so we can do it without needing root privileges in systemd-repart.

[bluca]

I'd definitely recommend against bundling and shipping anything, as then you become responsible for security updates, and ensuring it works everywhere that is supported, and here be dragons. It's just going to get messy. The baseline should just be: document what is needed, and let users/distros figure it out. For extra clarity, detect the version at runtime and error out if it's not good enough also sounds useful, on top of that.

As a convenience, supporting locally built tools with --extra-paths is fine as well, but log clearly about what is being used, so that there are no surprises. If someone wants to maintain integrated machinery to clone and build locally systemd, that's also fine, but it should really be optional, as you can't do that from an isolated build system (which is what should be used for production cases).

[DaanDeMeyer]

If someone wants to maintain integrated machinery to clone and build locally systemd, that's also fine, but it should really be optional, as you can't do that from an isolated build system (which is what should be used for production cases).

I'm leaning towards having optional machinery in mkosi that builds the tools we need from systemd from source and somehow configures mkosi to use those. Especially if we start using mkosi more heavily for integration tests in systemd we need some way for CI systems and developers to be able to run mkosi from git even if we depend on very recent stuff from systemd.

But yeah, bundling doesn't make a ton of sense. Distributions can figure this out themselves, I assume what'll happen is that mkosi won't be updated in distributions until systemd is updated (which makes sense to do). We should document our minimum systemd version somewhere though.

But we do need to do something so that using mkosi from git is still straightforward.

[bluca]

yeah that would be good to have

[netzego]

okey, so ci w/ git is what you after, i see. also love to see that 'point where mkosi can run without needing root privileges' for other reasons. adding 'optional machinery in mkosi that builds' newer parts of sd as needed, i dunno. feels like it should be an extra script/repo that does just that and mkosi having an option like --sd-dir or what ever it will be called. why should mkosi build a recent version of sd? it should be done by the user or the ci which depends on the newer version.

[tpgxyz]

• Mageia: I have no idea what their release schedule looks like, but cauldron seems to be rolling. Maybe @tpgxyz has some input there

Hi @behrmann i do not know eiher, as i'm from OpenMandriva and systemd-repart is here fully supported:)

[behrmann]

Sorry, but you guys are awfully close in naming. Not only do you both start with "Ma", but you have releases Cooker and Cauldron. It's like you want to be mistaken for another. ;)

[keszybz]

document what is needed, and let users/distros figure it out. For extra clarity, detect the version at runtime and error out if it's not good enough also sounds useful, on top of that.

Exactly.

Even in the case of CI, it's up to the user to install those dependencies in a way that makes the most sense. mkosi already has some generic mechanisms to call binaries from specified paths, and that is all that should be needed from this side.

Filed systemd/systemd#25155 to make this possibly a little bit easier.

Fedora; Will soon release 37 and I assume it will get 252 once that's released?

No. F37 is out already (in final freeze, blocked on openssl critical update), and v252 missed the deadline. We won't be upgrading to v252 in F37 or earlier.

• We should definitely cut a release before the switch (so maybe soon?) and advertise this upcoming change to gather some feedback. We should probably pin this issue, too.

Yes please.

But there shouldn't be a problem in requiring that users run mkosi as root if they want to have subvolumes. At least until mkfs.btrfs adds support for creating subvolumes itself so we can do it without needing root privileges in systemd-repart.

Agreed.

Given the amount of options we'd be able to drop. I'm very much in favor of breaking compat here and delegating all of this to repart definition files.

Ultimately we want to switch to repart config, since this removes a lot of complexity and documentation from mkosi. I don't think backwards-compat is terribly important at this point: most users of mkosi are closely following upstream development and care more about the new features than stability.

Buuut, if we want to maintain backwards compat, I'd do in a different way:
When reading configuration, see if any of the "old" options are used, and if yes, read them and generate a runtime repart configuration and use that. Refuse mixing "old" options and "new" repart config. I'd prefer that over adding a separate migrate verb. Maybe also print it out somewhere, so users can use that as basis of migration if they wish.

[malt3]

I am not very familiar with systemd-repart and could not find a lot of information on reproducibility of the outputs.
Let's say I create two disk images from the same filetree by running systemd-repart twice. Would the images be bit-by-bit identical?
Can the initial filesystem metadata be created reproducible and is the copying of the filetree done with a deterministic order?

[DaanDeMeyer]

That all depends on the filesystem itself. repart does nothing more than calling mkfs.<filesystem>. If it's not reproducible but there's options to make it reproducible, we can modify systemd-repart to use those options. Otherwise, you'll have to ask the fs developers to support reproducible output.

[mcassaniti]

What's the latest on this? What's the plan for distributing systemd-repart so that "older" distributions can still work with mkosi? I'm still carrying a patch do to the UUID file naming for now (there's worse things in life).

[DaanDeMeyer]

The latest is that we don't plan to distribute a more up to date version of systemd-repart and it'll be required to build systemd from source and add ExtraSearchPaths=<systemd-build-directory> to the mkosi config in order to be able to use mkosi on older distributions.

The PR that switches to repart can be found here. I'm fixing one more bug in systemd-repart and then CI should be green and the PR will be ready for review.

[mcassaniti]

This should probably cross-link to #1290. Not sure if/when this can be closed.

[DaanDeMeyer]

We can close this, I forgot to link it to the relevant PR #1276 so it wasn't closed automatically