Build system integration

[DaanDeMeyer]

This description assumes #1228 is already implemented and we do our builds inside regular directories on the host filesystem.

I've been thinking quite a bit on how to integrate mkosi built initrds into mkosi itself. My initial idea was to add another build stage that builds the initrd and then uses it in the final image. This is complicated because of several reasons:

• We have even more state to keep track of in mkosi itself, increasing overall complexity
• We have to figure out how to handle configuring the built initrd. If we'd want to keep a single config, we'd have to add extra options to configure the initrd specifically
• Doesn't take into account prebuilt initrds supplied via other means

Given the above points, this doesn't seem like the way to go. Instead, adding an --initrd option that takes a list of initrds to use for the final image is simpler and more generic, since the prebuilt initrd can come from any source instead of just mkosi.

The issue with the --initrd option is that we'd need to run mkosi twice to first generate the initrd first and the final image second. This led me to start thinking about how we could effectively manage multiple invocations of mkosi, with dependencies between them. This is pretty much what build systems are made to do. So I started thinking about what we could do to make mkosi integrate into popular build systems, and I came up with the following:

We should record all configuration used to build the image to the manifest

We already wanted to do this for other reasons but it's important to have a detailed description of the image and the configuration used to build it if we want to be able to build mkosi images in multiple separate invocations of mkosi

We should be able to merge instances of MkosiConfig

When building an image in multiple invocations of mkosi, we need to be able to specify only additional configuration that needs to be applied on top of the base image. We'd read the base MkosiConfig from the base image manifest, and any additional configuration to be applied on top, and merge them into the final configuration.

Also of note is that the base image manifest should also be included in the manifest as part of the config.

Mkosi should be able to use a previously built image as a base to start working from

If we want to be able to build images in multiple invocations of mkosi, we need to be able to take an existing image and extend it. We already support this in a very limited way using --base-image, but we can do much better. First, instead of taking a directory/image as an argument, --base-image should take a mkosi manifest as its argument. The manifest records where the base image was stored and the configuration used to build it. We merge it with any additional configuration for the new image and use that as the final config. The new image built only applies steps that weren't yet done in the base image build.

The base image is not modified, we always use an overlay when building (if there is no base we just use an empty overlay). We keep the existing output formats, but also add a new output format overlay, which just keeps the overlay we used when writing the image. When the base image is itself an overlay, we recurse through the base image manifests until the root or until we find a base image that is not an overlay, create a new overlayfs mount that combines all these overlays and use that as the base image.

This recursion extends to mkosi shell and mkosi boot, where we have systemd-nspawn boot into the overlay mount. For qemu, we'd need an extra step to run repart on the combined overlay to convert it into a disk image.

Note that we won't just build from any base image, the distribution, release, architecture and probably a few more things can't be changed.

We should remove --incremental and --build-script

we can remove --incremental and re-implement these outside of mkosi if we implement the above changes. The best way to explain this is by an example so I'll describe how I would use this in the systemd repo.

We replace --build-script by extending shell or adding a new verb run-script that runs a script in a given image. The verb would take the script, source directory, build directory, and maybe a few other things.

In systemd's meson build scripts, we'd have a series of targets that run mkosi to build a set of images and do things with them. I'll list the targets and their purpose:

• base image: This is the starting image that everything else builds on. It contains the runtime dependencies of systemd. Output is format is overlay.
• build image: This extends the base image and adds the dependencies required to build systemd, so devel packages, compilers and build systems. Output format is overlay
• configure systemd: This target depends on the build image and runs meson in the build image, source dir and mkosi.builddir/ are mounted in as with --build-script
• build systemd: Same as above but runs ninja to build systemd and installs it, destdir is also mounted in here
• lsp: This target runs clangd in the build image and depends on the configure systemd target to make sure compile_commands.json is available. This would use meson's run_target() functionality (and depend on ninja 1.11 so we can silence ninja's output).
• initrd image: This extends the base image and depends on the "build systemd" target. It adds some extra packages specific to the initrd and installs the built version of systemd on top and packages everything up as a cpio.
• dev image: This image extends the base image and depends on the initrd image and the build systemd target. It adds all the extra tooling to make the image useful for testing and developing systemd, copies in the systemd installation directory and uses the prebuilt initrd as its initrd. Output format is overlay.
• build-shell: This target depends on the build image and configure systemd and gives you a shell in the build image with source dir and build dir mounted.
• shell: This target depends on the dev image and gives you a shell in it
• boot: This target depends on the dev image and boots it with systemd-nspawn
• disk image: Depends on the dev image and packages it into a disk image with systemd-repart. Output is a disk image
• qemu: This target depends on the disk image and boots it with qemu

The above would be the basic setup. But we'd be able to go even further with this when we start using mkosi for systemd's integration tests. Since each integration test could be a test in meson that would depend on the dev image/disk image and run the integration test in it. If needed, for some tests we could define custom images that extend the dev/disk image even more.

Implementation wise, the image targets would all use meson's custom_target() and the interactive commands would use meson's run_target(). But this isn't specific to meson and can be adapted to any build system.

Given the above setup, there's no more need for --incremental since it's implemented in the build system. There's also no more need for the notion of build images, since that's moved to the build system as well. mkosi itself would be stripped down to building a single image based on the provided options and writing the manifest after building the image. All features that involve scheduling multiple image builds and running commands in them would be moved to the build system.

In the systemd repo, you could even have the mkosi targets depend on the repart, nspawn, and other targets to first build all these components on the host and then use these directly when building the images.

Conclusion

I think using mkosi with a build system would be a very powerful way to schedule multiple image builds and the interaction between them while simplifying mkosi itself due to not needing to deal itself with scheduling multiple image builds. In terms of breaking backwards compat, we'd be removing --incremental, and replacing --build-script with a similar verb.

We could go slightly further and require --initrd for bootable images remove dracut support and require the initrd to be supplied by the user.. But this would mean building bootable images would no longer be possible without a build system, at least not until distro prebuilt initrds become a thing.

This turned into quite a wall of text. I'm curious as to what others think of this approach.

[behrmann]

A thought I had, was that if we do this, it would maybe be nice to add some template functionality to template out an empty mkosi project directory for the sanctioned build system (I guess meson?) with the necessary boiler plate files, so that people don't have to learn yet another thing / have an easy starting point.

[DaanDeMeyer]

Related to this I opened mesonbuild/meson#10928 in meson

[DaanDeMeyer]

A thought I had, was that if we do this, it would maybe be nice to add some template functionality to template out an empty mkosi project directory for the sanctioned build system (I guess meson?) with the necessary boiler plate files, so that people don't have to learn yet another thing / have an easy starting point.

I'd be wary of adding something like this too early. None of the above implies that you are required to use a build system with mkosi. Until distro shipped initrd packages become a thing, You can just have a bash script that first builds the initrd and then builds the final image with the initrd and it'll work fine, no need for a build system. After distro shipped initrd packages become a thing, default usage will go back to being a single mkosi command.

It's only when you want more fine grained dependency tracking and you're using mkosi in a context such as developing systemd that integrating it into the build system becomes useful. We should definitely ship an example inside mkosi explaining how one can integrate with a build system such as meson but I'd keep it at that for now.

[labichn]

I use mkosi for building standalone images and sysexts. I like the sound of less build/compile-specific machinery inside mkosi, I'd love to build my initrd with mkosi, and I agree that mkosi should be able to use a previously built image as a base to start working from. But, is passing around mkosi-specific manifests/configs the only way to do this?

It'd be nice if the necessary information could be pulled from partition UUIDs or {extension,os}-release files in the usual spots, so you can use the tools that already exist to operate on the sysroots/images. Or is there other information you need that I'm overlooking?

To your point, building extensions using raw base images is currently broken (mounted with the extension's partition table rather than the base image's partition table). But systemd-dissect can mount any conformant base image without issue. Why not use systemd-dissect?

[DaanDeMeyer]

I use mkosi for building standalone images and sysexts. I like the sound of less build/compile-specific machinery inside mkosi, I'd love to build my initrd with mkosi, and I agree that mkosi should be able to use a previously built image as a base to start working from. But, is passing around mkosi-specific manifests/configs the only way to do this?

I thought about this as well, we should probably just support multiple possible inputs. The thing is that if you just output an overlay, you won't have any useful information in just the overlay directory, you need the manifest so you can figure out that's it's just an overlay in the first place.

But that doesn't mean we shouldn't support regular directories and images, we can just support a few different ways of providing the base image.

[labichn]

The thing is that if you just output an overlay, you won't have any useful information in just the overlay directory, you need the manifest so you can figure out that's it's just an overlay in the first place.

Gotcha, you're trying to support creating (non-sysext) root-overlay layers using mkosi that other layers can be built on after? The lack of {extension,os}-release files would distinguish that case, but then I guess it makes sense that you're missing exactly that information on the input side.

I used to build overlays like this too, but between nspawn's root --overlay flakiness and the difficulty building them I ditched that approach in favor of sysexts for RO overlays and --bind in RW state (rather than RO --overlays with a RW top-level). It would be nice to have a structured way of building and building-on overlays.

[keszybz]

Until distro shipped initrd packages become a thing, You can just have a bash script that first builds the initrd and then builds the final image with the initrd and it'll work fine, no need for a build system. After distro shipped initrd packages become a thing, default usage will go back to being a single mkosi command.

Also, even without distro-built initrds, you can just use an initrd from the host system. That's what we do in systemd integration tests.

I agree with the general idea. All that mkosi needs is an --initrd=… option, and where this initrd comes from would be up to the user: built with mkosi in an earlier step, part of some package, or pulled from the host.

instead of taking a directory/image as an argument, --base-image should take a mkosi manifest as its argument. The manifest records where the base image was stored and the configuration used to build it. We merge it with any additional configuration for the new image and use that as the final config.

Hmm, I'm not convinced.

First of all, this might be really really hard to implement, because mkosi would need to figure out how all possible configuration options combine. As an example, let's say that the base config has Packages= and RemovePackages= and the higher-level config has the same. Now we'd need to redo the work of the package manager and figure out what packages to add and what to remove. Or the base config had a number of partitions specified with repart, and now mkosi would need to figure out which ones need to be added because of "additional" config, possibly in the middle. So far we used a different solution: start with the base image, and just apply the new config on top of that. So for Packages this means that the base image must be built with a package database and we call the package manager again with a new set of packages and it figures out what to add. For partitions, this means we call systemd-repart and it just executes the new repart config and probably creates new partitions. Making this DTRT is up to the user who creates the mkosi configs.

Second, I don't think this should use the base config or manifest at all. I think we should treat stuff that is found in the base image as any other input, and just let the higher layers adapt to it. Similarly to the case when some package in the distro puts in additional content and we install it and later steps see that content and somehow react to it.

We should definitely ship an example inside mkosi explaining how one can integrate with a build system such as meson but I'd keep it at that for now.

Ack. We shouldn't integrate with a build system, but instead allow the build system to integrate with mkosi, by calling it with appropriate arguments.

[keszybz]

One more thought: if we switch to repart, and the repart phase consists mostly of repart taking some prepopulated directories and writing an fs image based on that, that earlier population phase and the later image creation phase are essentially separate. Thus is should be possible to for example do the first phase once, and then repeat it second phase a few times with different output formats. We should make this possible, similarly to how a program can be compiled and linked in one step, but in a build system, the steps are separated and controlled by the build system, not the compiler.

[DaanDeMeyer]

Hmm, I'm not convinced.

First of all, this might be really really hard to implement, because mkosi would need to figure out how all possible configuration options combine. As an example, let's say that the base config has Packages= and RemovePackages= and the higher-level config has the same. Now we'd need to redo the work of the package manager and figure out what packages to add and what to remove. Or the base config had a number of partitions specified with repart, and now mkosi would need to figure out which ones need to be added because of "additional" config, possibly in the middle. So far we used a different solution: start with the base image, and just apply the new config on top of that. So for Packages this means that the base image must be built with a package database and we call the package manager again with a new set of packages and it figures out what to add. For partitions, this means we call systemd-repart and it just executes the new repart config and probably creates new partitions. Making this DTRT is up to the user who creates the mkosi configs.

I've changed my mind on this as well over the past weeks. We can use /etc/os-release to figure out distro, release, architecture and we don't need to know the other stuff. The only thing we can't figure out from the image itself is when it's an overlay and we need to recursively find the parent overlays to construct the full image. We could leave that to the user instead by allowing to specify --base-image multiple times and creating an overlay of all the images specified, but that would be more difficult compared to just putting that information in the manifest.

One more thought: if we switch to repart, and the repart phase consists mostly of repart taking some prepopulated directories and writing an fs image based on that, that earlier population phase and the later image creation phase are essentially separate. Thus is should be possible to for example do the first phase once, and then repeat it second phase a few times with different output formats. We should make this possible, similarly to how a program can be compiled and linked in one step, but in a build system, the steps are separated and controlled by the build system, not the compiler.

This is exactly what I had in mind, if you run mkosi on an existing image without any config except output config, it shouldn't change anything to the image and just pack it up in the requested format. I think a model like this could be very powerful.

[DaanDeMeyer]

So after lots of discussion, we should keep support in mkosi itself for running a build script to not force users to use a build system to use mkosi for anything useful. In addition, we can add a simple for loop to mkosi to build multiple images, whose configuration we read from e.g. mkosi.images/ or a user configured directory. We build the images in alphanumerical order and if a later image needs an earlier built image output, it can simply refer to it by relative path in its configuration.

In systemd, we'd have the following images configured:

• base: This has the runtime packages required to run our systemd built from source in Packages and the packages to build systemd from source in BuildPackages. The output is an image with the runtime dependencies + systemd built from source installed into it. We also install the distro systemd package explicitly so that the package manager thinks its installed and doesn't try to overwrite our systemd built from source with the distro package later.
• initrd: This builds on base and adds the necessary packages required for the initrd in Packages. Output is a compressed cpio to be used as the initrd in the final image
• final: This builds on base and uses the initrd we built earlier. The final image adds a kernel and other packages required in the final image and produces the final image, which is packaged as a disk image with systemd-repart.

Given that build system integration isn't really required anymore with this model, let's close this issue.