Add OSTree support

[Kekun]

GNOME OS lost its OSTree backend in early 2025.

For Endless OS 7 to be based on GNOME OS and OSTree, we need to restore OSTree support, either upstream or downstream.

[zeenix]

I'll start this task by first trying to revert Jordon's commit that removes ostree images and see how that goes.

[ssssam]

I've been testing the freedesktop SDK vm/minimal-ostree/ stuff with eos-image-builder to see if we can reuse that for eos-build-meta.

Notes so far:

• In freedesktop-sdk, use make ostree-serve to serve repo at http://localhost:8000
  • Export the GPG key with: gpg --homedir ./ostree-gpg --export --armor > fdsdk-test.key
• For eos-image-builder:
  • I'm running it in a Debian 12 container as it seems to assume root
  • Copy fdsdk-test.key to ./data/keys
  • Configure ostree settings in config/default.ini to pull from the local repo

So far, eos-image-builder pulls the ostree commit ok, but something in the container then fails:

+ 16:27:01 INFO run-build: Checking out OSTree ref eos:freedesktop-sdk/minimal/25.08beta/x86_64 to /var/tmp/eos-image-builder
error: Copy checkout of 389846c2702216e1367c8dfb68326a6b93ccf5703c89c93979052a9bf359608e to bin: lsetxattr(security.selinux): Operation not supported

SELinux is disabled on the host. The workaround maybe to use ostree checkout --no-xattrs in eos-image-builder (or maybe that'll break something further down); or fix my container / use a vm

[ssssam]

+ 16:27:01 INFO run-build: Checking out OSTree ref eos:freedesktop-sdk/minimal/25.08beta/x86_64 to /var/tmp/eos-image-builder
error: Copy checkout of 389846c2702216e1367c8dfb68326a6b93ccf5703c89c93979052a9bf359608e to bin: lsetxattr(security.selinux): Operation not supported

I got past this issue by using a volume mount for /var/tmp, but then hit more issues, the image builder expects /run/udev to be available which is tricky in a container; bypassing that hits further issues mounting /dev inside the buildroot.

It seems like a VM will be the easier option, I'll set one up once back from GUADEC and pick this up again ;-)

[dsd]

I don't have a strong preference if you want to use eos-image-builder and dive into that aspect right away, but unless there are strong reasons for that, I would suggest starting with:

1. Get the ostree built and pushed to a new branch the server (it builds, ship it!)
2. Make the ostree usable through installing a current (Debian-based) EOS development image, and then using ostree switch to jump to the new branch.

I expect this to be a more incremental approach than diving into the image builder right away. The image builder will have requirements around how the ostree is shaped and formed, which will be more fiddly, and that's why I had proposed putting that work in milestone 3 (#13), although it will need to be done at some point either way!

It should be a bit less work to just have a branch that we can switch to as a starting point, without having to worry about the image builder (yet). However it won't be without a handful of complications, I imagine. One that I can forsee is that we'll have to make the system level uids/gids used by the GNOME OS build precisely match the ones used by current EOS installations. Otherwise you'll probably boot and find that permissions on certain things are wonky.

cc @wjt (currently on break)

[ssssam]

Did that using ostree admin switch and it successfully booted the minimal-ostree system, without apparently breaking eos6.

Steps to reproduce:

1. Download EOS6 (Basic - ISO Image) from "Endless OS direct download" page. Create a VM, run the installer and create a user.
2. In a clone of freedesktop-sdk.git, run make ostree-repo then make ostree-serve. You need my changes from !27582 "Fixes for minimal-ostree" so it builds.
3. Get the address of the local ostree server. This will be your machine's IP, port 8000. For example: http://1.2.3.4:8000/
4. Export the GPG publickey from the local ostree repo. You can run this in the freedesktop.git clone: gpg --homedir ./ostree-gpg --export --armor
5. Open a terminal in the EOS6 machine and set up the ostree remote:

• sudo ostree remote add local
• sudo ostree remote gpg-import local (paste in the publickey, hit CTRL+D)

6. Deploy the minimal-system:

• sudo ostree admin switch local:freedesktop-sdk/minimal/25.08beta/x86_64

7. Reboot, and login as the existing user you created in step 1.
8. Rejoice

The next step should be to copy or reference the vm/minimal-ostree elements in eos-build-meta to get a GNOME OS ostree, then we can repeat the process.

Screencast.From.2025-07-29.17-53-25.mp4

[wjt]

Actually I would expect that if upgrades work, and if you disable most/all hooks in the image builder, turn off ISO generation (will need to be fixed presumably), BIOS/CSM bootloader support (are we dropping this more widely?), and dual-boot .boot.zip file generation (I guess we're dropping dual boot in EOS 7?) the only tricky piece in the image builder will be populating the ESP, which currently boils down to copying /usr/lib/efi_binaries/EFI from the OS into the ESP.

Looking forward to trying this out!

[ssssam]

How Freedesktop SDK minimal-ostree system builds (as of 24.08.22).

First, Make targets on the host configure a local-only OSTree server:

1. ostree-gpg: creates ./ostree-gpg with a new GPG keychain.
2. files/vm/ostree-config/fdsdk.gpg: exports the GPG public key, to be included in the BuildStream build
3. ostree-config.yml: create ostree-config.yml config file with branch name and the build machine's IP address

Then, the update-ostree make target (via a helper script) triggers a build of vm/minimal-ostree/repo.bst.
Relevant elements here are:

• vm/config/ostree.bst: Output artifact contains the system skeleton and ostree config, including:
  • Toplevel directories: /boot, /efi, /etc and so on
  • Toplevel ostree directories: /sysroot and /sysroot/ostree
  • Toplevel symlinks: /home -> /var/home, /root -> /var/roothome and /media -> /run/media
  • fdsdk.gpg, the local-only server's public key, installed to /etc/pki/ostree
  • freedesktop-sdk.conf, which defines an OSTree remote named FreedesktopSDK using the build machine's IP address
• vm/config/{resolved,journald,networkd}.bst: Configuration for systemd resolved, journald and networkd.
• vm/minimal-ostree/deps.bst: Lists the contents of the minimal-ostree system, including the above config.
• vm/minimal-ostree/initial-scripts.bst: Collects "initial-script" commands from dependencies and
  saves them to /etc/fdsdk/initial_scripts in its artifact.
• vm/minimal-ostree/filesystem.bst: Applies split-rules to vm/minimal-ostree/deps.bst to remove
  development headers and so on, producing a rootfs as its output artifact.
• vm/minimal-ostree/repo.bst:
  • Uses prepare-image.sh script to post-process the rootfs, running everything in /etc/fdsdk/initial_scripts, and more.
  • Creates an empty OSTree repo and commits the rootfs (/sysroot) as a new commit. This is the output artifact.

Finally, utils/update-repo.sh exports the repo.bst artifact, and
imports the commit into a new or existing branch on the host at ./ostree-repo.

For local development we can now pull the new OStree from the local OSTree server, as I did above.

Image builds are done in vm/miminal-ostree/image.bst, using ostree admin init-fs, ostree admin os-init and ostree admin deploy, then using
pengutronix genimage to create a disk image. There's no support for signed
boot here.

How EOS6 builds

First, OBS builds some Debian packages.

Then, eos-ostree-builder
runs through these stages:

• OS stage:
  • assembles .deb packages into a filesystem tree
  • Runs os hooks
• ostree stage
  • Various important bits of rootfs preparation, similar to vm/config/ostree.bst
• Publish stage
  • Runs publish hooks, which pushes the new ostree to the public server for use
    with eos-updater.

Then, eos-image-builder runs through these stages:

• ostree stage: pulls the latest filesystem tree onto the image-builder machine, does ref tweaks
• image stage:
  • Prepare a rootfs using ostree admin init-fs, ostree admin os-init and ostree admin deploy
  • Various bits of bootloader setup
  • Runs content hooks
  • Sets up terrifying bind mounts and runs image hooks
  • Disk image creation (using rocks and bones basic Linux tools)
• manifest stage:
  • Create manifest.json, including basic info and manifest hooks
• publish stage:
  • Runs publish hooks which push the images, sha256sums and build logs

Then, eos-sb-signer does some stuff that we
will not think about until #8.

Next steps for this issue

Notes:

• Assuming that we keep using eos-image-builder, the freedesktop-sdk
  vm/minimal-ostree/image.bst element is redundant, as is the
  gnome-build-meta gnomeos/image.bst element and the signed-boot
  infrastructure it uses.
• There is overlap between eos-ostree-builder and vm/config/ostree.bst
• EOS doesn't have any "local OSTree server" tooling, while FDSDK does have some,
  and it would be nice-to-have for EOS.

Next steps:

• Add eos/repo.bst to produce an OStree repo built from gnome-build-meta:gnomeos/filesystem.bst,
  deploy that locally as I did with the FDSDK minimal system. (Will require some "local OStree server"
  tooling for the testing).
• Try eos-ostree-builder on the result and see if that is a feasible way to finish the issue.

[dsd]

Try eos-ostree-builder on the result and see if that is a feasible way to finish the issue.

This project is replacing eos-ostree-builder so you don't need to run that.

Once you have the ostree building the next step is to push it onto our server in a new branch. That work could perhaps be considered part of #6.

[ssssam]

Ok thanks, so the work is to go through each of the eos-ostree-builder os hooks and see which of those need reimplementing as BuildStream elements.

Makes sense to do the push as part of #6 so I avoid having any keys locally.

[dsd]

Yeah there is probably some stuff from the hooks that should be reimplemented.
But at a glance, I see in there some finer details that can be addressed later in the project, and some things that will fall into PAYG and image builder support, which have their respective separate milestones/issues.

So in terms of priorization, I think after having the basic ostree build going from buildstream land (this issue), pushed to a branch on our ostree server (#6), the next thing I would encourage you to help us with is #22. Once that is done you will have myself and a couple of others as daily testers!