Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Turning on composefs breaks pivot from old bootimages #1678

Open
Prashanth684 opened this issue Dec 9, 2024 · 5 comments
Open

Turning on composefs breaks pivot from old bootimages #1678

Prashanth684 opened this issue Dec 9, 2024 · 5 comments

Comments

@Prashanth684
Copy link
Contributor

Attached the full secondboot journal.
journal.txt

The specific error is:

Dec 09 17:13:23 localhost systemd[1]: Mounted /sysroot.
Dec 09 17:13:23 localhost systemd[1]: Starting OSTree Prepare OS/...
Dec 09 17:13:23 localhost ostree-prepare-root[675]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora-coreos/deploy/8bd789c8d973d9910e8cc1fc477dd20ac5e975e0663085eeb06a95bfcdba5199.0
Dec 09 17:13:23 localhost ostree-prepare-root[675]: sysroot.readonly configuration value: 1 (fs writable: 1)
Dec 09 17:13:23 localhost ostree-prepare-root[675]: ostree-prepare-root: composefs: failed to mount: No such file or directory
Dec 09 17:13:23 localhost systemd[1]: ostree-prepare-root.service: Main process exited, code=exited, status=1/FAILURE
Dec 09 17:13:23 localhost systemd[1]: ostree-prepare-root.service: Failed with result 'exit-code'.
Dec 09 17:13:23 localhost systemd[1]: Failed to start OSTree Prepare OS/.
@jlebon
Copy link
Member

jlebon commented Dec 10, 2024

This doesn't look like the first boot of the machine. Which AMI is this BTW? Is this from the CoreOS pipeline, or somewhere else? Let's try to reproduce it outside of OpenShift (presumably by booting the AMI, doing an rpm-ostree rebase or bootc switch and rebooting?).

@Prashanth684
Copy link
Contributor Author

Prashanth684 commented Dec 10, 2024 •
edited
Loading

Yes this is not the first boot. This log is from the e2e test that is run on OKD release nightlies, where the Openshift installer boots the machine with fcos AMis and then on second boot, pivots to scos.

@Prashanth684
Copy link
Contributor Author

When i tried booting with the AMI that the Openshift installer uses - 39.20231101.3.0, I see this issue. But when booting with a newer AMI - 40.20240602.3.0, I don't see the issue anymore and the rebase works. openshift/installer#8640 updates the AMI for fcos and also tests the okd scos e2e.

@jlebon jlebon changed the title ostree-prepare-root: composefs fails to boot latest scos image Turning on composefs breaks pivot from old bootimages Dec 16, 2024
@jlebon
Copy link
Member

jlebon commented Dec 16, 2024

So the issue here is that any bootimage with a libostree older than v2023.4 will be too old to know how to create composefs images when upgrading. And so once we upgrade, ostree-prepare-root in the new initramfs fails because it sees that composefs should be enabled and yet no composefs image exists.

This should be fixed by ostreedev/ostree#3353, which would allow ratcheting in composefs by letting ostree-prepare-root tolerate missing composefs images.

But clearly, longer-term we want to be able to rely on composefs just working the first time. This would definitely be fixed as we keep rolling out the bootimages update effort (openshift/enhancements#1496), but we may decide to do a shorter-term tactical fix for it if we're blocked on it.

jlebon added a commit to jlebon/os that referenced this issue Dec 16, 2024
@jlebon
common.yaml: disable composefs on all arches
d9ae116 
There are issues currently on ppc64le and kernel-64k in el9 that break
with composefs. So it's not sufficient to do an architecture check
anyway. See: https://issues.redhat.com/browse/RHEL-70199.

But additionally, there are issues also to resolve when dealing with
old bootimages. This should be fixed by turning on composefs image
generation by default in libostree and downgrading to `maybe` here. See:
openshift#1678.

Once we fix the latter issue, we should be able to turn it on at least
in the el10 variants, where the first issue doesn't exist. (Though
technically, even the latter issue is "fixed" in el10 by virtue of new
enough libostree in the bootimages, but it would still break to go from
an el9 bootimage to el10, which we'll likely have to support.)
@jlebon jlebon mentioned this issue Dec 16, 2024
Merged
@jlebon
Copy link
Member

jlebon commented Dec 16, 2024

For now, let's back out composefs: #1700

jlebon added a commit to jlebon/os that referenced this issue Dec 16, 2024
@jlebon
common.yaml: disable composefs on all arches
2a61323 
There are issues currently on ppc64le and kernel-64k in el9 that break
with composefs. So it's not sufficient to do an architecture check
anyway. See: https://issues.redhat.com/browse/RHEL-31991.

But additionally, there are issues also to resolve when dealing with
old bootimages. This should be fixed by turning on composefs image
generation by default in libostree and downgrading to `maybe` here. See:
openshift#1678.

Once we fix the latter issue, we should be able to turn it on at least
in the el10 variants, where the first issue doesn't exist. (Though
technically, even the latter issue is "fixed" in el10 by virtue of new
enough libostree in the bootimages, but it would still break to go from
an el9 bootimage to el10, which we'll likely have to support.)
@jlebon jlebon mentioned this issue Jan 9, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Prashanth684 @jlebon