Need to think about an update strategy

[probonopd]

pkg upgrade seems to overwrite our configuration.

For example, our changed rc scripts are seemingly replaced with those from the pacakges, and our carefully configured slim.conf is moved to slim.conf.pkgsave.

How does e.g, the Mac work when "over-installing" a newer version of the OS?

[grahamperrin]

Was the slim.conf.pkgsave effect reproducible?

… How does e.g, the Mac work when "over-installing" a newer version of the OS?

IIRC it's quite blunt. Something like, writing all affected files to a temporary path then a move into place – typically without considering the before/after state of things (because things are so neatly separated).

[probonopd]

The OS in a read-only image + user modifications in a read-write unionfs overlay would solve this very neatly...

[grahamperrin]

helloSystem/hello#37 (comment) recently I might have misunderstood what's to be expected within the constraints of 'live' media.

If the OS will be read-only: that's fine, if:

1. user data is saved to persistent storage on the medium; and
2. user data will be transferable, fairly easily, from an old medium (i.e. outdated OS) to a new medium (up-to-date OS).

Alternatively: some way to periodically download then 'slot in', to the medium, an updated OS?

[probonopd]

What I mean is that the OS should be conained in a read-only image file (in the worst case: ZFS dataset) but overlayed with read-writeable space so that any changes by the user get redirected into the read-writeable area. This way, the system appears entirely r/w to the user.

When an OS update comes along, the read-only file is switched to another one, but the user-generated content "on top" still is there.

NomadBSD kinda goes into this direction, but the underlying OS is not an image file but a partition, which makes updating it harder.

Why do I want an image file rather than a partition for the system: So that it can be most easily exchanged with another one, and users can easily keep multiple ones around.

The more I think about it, we should have a closer look at what NomadBSD is doing there. Using a file rather than a partition shouldn't be that hard to change... but then, as far as I remember, not even NomadBSD achieves that all parts of the system appear r/w to the user.

[probonopd]

Why work against the natural desire to install, or update, an application?

Because you can end up with a half-outdated (hello components), untested (in this combination), messed up (regarding configuration files), and potentially no longer working (as you have shown) system.
helloSystem is (also) for non-technical users who should never have to see a terminal.

Note that it says "not yet". We still "need to think about an update strategy" and it will probably be more like Android/iOS.

[grahamperrin]

end up

Are boot environments forgotten again?

helloSystem/Utilities#33 (comment)

non-technical users who should never have to see a terminal.

If you stop crippling pkg, these users will be blissfully unaware of the improvement.

[probonopd]

Boot Environments just allow to go back once disaster has struck. I want to avoid the disaster in the first place. (As I am sure you find out you can use pkg.real if you really know what you are doing.)

Note that pkg is not mentioned in any helloSystem documentation at all.

[grahamperrin]

pkg is not mentioned

Maybe not mentioned at https://hellosystem.github.io/docs/index.html but pkg is integral to FreeBSD.

From helloSystem/hello#110 (comment):

… we want to stay "true FreeBSD" wherever possible. So at least for what comes as part of the Live ISO, I think the existing system wih ports and packages serves us well …

From main page https://github.com/helloSystem/hello#what:

Based on FreeBSD. Less, but better!

… Without lockdown. Without Big Brother. The user in full control.

Crippling pkg, for any length of time, does not make helloSystem better than FreeBSD.

Boot Environments just allow to go back once disaster has struck.

That's a somewhat narrow vision.

Less dramatically: boot environments allow you to progress¸ with the option of going back.

If there's a fear of boot environments, then why is Boot Environments amongst the preferences for helloSystem?

[probonopd]

There is no fear of BEs, but

• We need a way to select BEs from the bootloader, which means we need to work on the bootloader to be completely silent unless a key combo (e.g., Command-B-E for Boot Environments) is held down
• In order to be able to update the system (not just some apps) using pkg, someone would need to package all of helloSystem

[kettle-7]

In order to be able to update the system (not just some apps) using pkg, someone would need to package all of helloSystem

If you use SystemImageKit and have a pure FreeBSD image with an early init script to copy over hello-specific files from a second image with geom_rowr or something, then you can use rsync to update the two.

Does this simplify things or complicate them?

[probonopd]

So your idea is to have one image that contains an unmodified FreeBSD, and another one that contains the helloSystem modifications? Would work but would require a working union filesystem, because geom_rowr doesn't allow you to use the second image without the original first image.
Also, I think even updating an unmodified FreeBSD is too complicated. I'd just replace the whole image with a new one. Similar to how OpenWrt works.

[kettle-7]

Yeah, pretty much. Even if you mount /var from a persistent partition, and symlink /etc, /Applications, and /Users to subfolders of /var, like Silverblue does, would that work?

[grahamperrin]

up.bsd.lv Binary Upgrades Report

From https://up.bsd.lv/:

2021-07-26 - This Proof of Concept was extremely informative and is taking a break to build a Proof of Concept replacement for freebsd-update. Thank you everyone who helped out with it!

[grahamperrin]

pkg is integral to FreeBSD.

More accurately:

• the normally used pkg is a port – ports-mgmt/pkg
• the pkg that's installed with FreeBSD is for bootstrapping – when run, it invites installation of ports-mgmt/pkg.

https://cgit.freebsd.org/src/tree/usr.sbin/pkg

[probonopd]

https://github.com/ChimeraOS/frzr

frzr is a deployment and automatic update mechanism for operating systems. It deploys pre-built systems via read-only btrfs subvolumes, thus ensuring safe and atomic updates that never interrupt the user.

Updated system images are downloaded at boot time and deployed to an entirely separate subvolume so as not to interfere with the currently running system. Upon next boot, the newly installed system is started and the old one is deleted, completely seamlessly and invisibly.

Persistence is handled by making /home and /var separate subvolumes mounted into the read-only root. /etc is made read-write via overlayfs.

Goes in the direction I mean. But I'd like to use filesystem image files rather than btrfs subvolumes.

[kettle-7]

Yes, that might be just what we need. As for the btrfs thing, we might be able to fork it...

[probonopd]

No btrfs for me, so much is sure ;-)

[kettle-7]

Yes, and that's why we could fork it. ZFS is welcome, right? And that naturally uses a similar but more flexible subvolume system. So, could we not fork it and add ZFS support?

[kettle-7]

Or yeah, there's the option of having ufs2 images that can be downloaded, that may be better.

[kettle-7]

Also I want my Firefox updates ;~)

I tried bundling Firefox, but it's one of those apps that's damn near impossible to compile on a relatively slow computer, on FreeBSD. I might have a look at the port, see how they do it, I might be missing a compiler option.

[probonopd]

As for Firefox, it shouldn't be too hard to make Application Bundles for each version. It shouldn't even be neccessary to re-compile it, it should be doable by using the pkgs as ingredients.

[grahamperrin]

I want my Firefox updates

What's the issue?

[kettle-7]

Oh, I posted it on a separate issue.

Basically, updating Firefox updates Inkscape, python and Thunderbird, uninstalls krita and removes Qt, which explains why krita went but hey, now the system doesn't boot because my display manager crashes cause of no qt libraries.

[grahamperrin]

a separate issue

Link please. Thanks.

[kettle-7]

#267 (comment)

[grahamperrin]

#267 (comment)

Thanks, that's somewhat lost (with unanswered questions) in the midst of an issue that may be non-related. I suggest raising a separate issue moving to a discussion so that it can be actionable and closable.

[kettle-7]

I feel it is an issue with pkg, not helloSystem.

[grahamperrin]

I feel it is an issue with pkg,

I doubt it.

not helloSystem.

Have you raised the issue elsewhere?

Installation and Maintenance of Ports or Packages | The FreeBSD Forums

https://github.com/freebsd/pkg/

https://lists.freebsd.org/subscription/freebsd-pkg

[grahamperrin]

helloSystem/Filer#3 (comment) (2020-08-31)

Check Cirrus CI …

helloSystem/hello#37 (comment) (2020-11-18)

… clone the project and build their own ISOs using public services (like GitHub and Cirrus CI) …

helloSystem/Utilities#33 (2021-01-17)

System Update utility

#141 (comment) (2021-02-20)

… a continuous build pipeline without persistent storage.

#135 (comment) (2021-03-21)

https://cirrus-ci.com/github/helloSystem/ … interesting for helloSystem developers, testers, and power users ;-)

helloSystem/hello#161 (comment) (2021-09-23)

• …
• … easy for anyone to fork. By not needing any infrastructure to build helloSystem (other than what GitHub and Cirrus CI provide), anyone can simply fork the repository and create their own modified ISOs. …

… built by a shell script from the official FreeBSD binary ingredients plus official FreeBSD ports, …

▼

https://github.com/helloSystem/ISO/blob/experimental/build.sh

helloSystem/Utilities#33 (comment) (2021-10-18)

… if we really want end users to update the system using packages. …

For FreeBSD-based systems, use of things such as pkg-install(8) should be thought of as inevitable in most cases.

Addition of a package might require an upgrade to a previously-installed package, and so on.

---

Rewind: #135 (comment) (2021-02-17)

freebsd-12-2-release-p3-amd64 does not exist as an instance name on Cirrus CI …

Fast forward: freebsd/freebsd-src#543 (comment) there's /usr/sbin/freebsd-update in the context of Cirrus CI:

• does this help to shape thoughts on how the helloSystem build routine(s) might produce an ISO with a suitably patched (updated) base operating system?

(Compare with, for example, yesterday's experimental non-patched 12.2-RELEASE, pictured under helloSystem/hello#219 (reply in thread).)

[probonopd]

During initial installation, the installer uses bsdintall zfsboot to create and mount ZFS partitions:

zroot/ROOT/default on /mnt (zfs, local, noatime, nfsv4acls)
zroot/tmp on /mnt/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /mnt/usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /mnt/usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /mnt/usr/src (zfs, local, noatime, nfsv4acls)
zroot/var/audit on /mnt/var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/crash on /mnt/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/log on /mnt/var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /mnt/var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /mnt/var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot on /mnt/zroot (zfs, local, noatime, nfsv4acls)

On the running system, it will end up as

% zfs list
NAME                 USED  AVAIL     REFER  MOUNTPOINT
zroot               1.40G   214G       96K  /zroot
zroot/ROOT          1.36G   214G       96K  none
zroot/ROOT/default  1.36G   214G     1.36G  /
zroot/tmp            264K   214G      264K  /tmp
zroot/usr           41.6M   214G       96K  /usr
zroot/usr/home      41.3M   214G     41.3M  /usr/home
zroot/usr/ports       96K   214G       96K  /usr/ports
zroot/usr/src         96K   214G       96K  /usr/src
zroot/var            692K   214G       96K  /var
zroot/var/audit       96K   214G       96K  /var/audit
zroot/var/crash       96K   214G       96K  /var/crash
zroot/var/log        212K   214G      212K  /var/log
zroot/var/mail        96K   214G       96K  /var/mail
zroot/var/tmp         96K   214G       96K  /var/tmp

Currently there is no way for the installer to install a newer version of the operating system "over" an existing installation, because the installer always runs bsdintall zfsboot first.

We would like to change this so that users can install a newer version of the operating system over an existing one. For this we would like to:

1. Snapshot the whole zroot
2. Then clone it to make it writable and mount the clone at mnt and its subdirectories in the same manner as zfsboot install does?`
3. Install the operating system over the clone
4. Change whatever needs to be changed so that the next boot (only the next boot) will be from the clone
5. Once the cloned system is fully booted all the way into the desktop, change whatever needs to be changed so that all subsequent boots will be into the snapshot. Possibly ask the user for confirmation.

What are the exact commands needed to do this from the Live ISO?

Actually, since we would be using a clone, we might not even need to boot into the Live ISO to install the new OS over the clone but could do it from the running system. At least that is how I understand it - that you can snapshot and clone the running system without bad side effects.

Is creating a "recursive snapshot" what we need here? sudo zfs snapshot -r zroot@forupdate.

We now can do ls /.zfs/snapshot/forupdate/.

Now we can make a writable clone:

sudo zfs clone zroot@forupdate zroot/cloneforupdate.

Then we get sudo ls /zroot/cloneforupdate but there is nothing in there.

There are no less than 11(!) zfs "mountpoints" listed. How do we get clones of all of them mounted in the correct nested/recursive way under /mnt?

[probonopd]

Using Boot Environments would probably be the correct way to do this, but Boot Environments do not cover all zfs datasets, at least not by default. So none of these are covered by Boot Environments:

zroot/tmp on /tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /usr/home (zfs, local, noatime, nfsv4acls)
zroot/var/log on /var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot on /zroot (zfs, local, noatime, nfsv4acls)
zroot/var/audit on /var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/crash on /var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /var/mail (zfs, local, nfsv4acls)
zroot/var/tmp on /var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/ports on /usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/src on /usr/src (zfs, local, noatime, nfsv4acls)

Can this be changed?

[probonopd]

Just using cpdup -o overwrites critical files n /etc like the user and password files. We would need to find a way to spare them out, which means that we would need to know which ones should and should not be updated when installing a newer OS version over an existing one.

Possibly freebsd-update has clever mechaninsms for this, but it asks the user questions (by default), which we don't want (and have found a way around).

Its config file has

# Paths which start with anything matching an entry in an UpdateIfUnmodified
# statement will only be updated if the contents of the file have not been
# modified by the user (unless changes are merged; see below).
UpdateIfUnmodified /etc/ /var/ /root/ /.cshrc /.profile

and it looks like our update strategy will need something along those lines.
Ideally without having to reinvent the wheel.

So the question is, can freebsd-update use a local directory instead of a remote server as its source?

[probonopd]

vermaden says:

Its pointless to have /home or /usr/ports on ZFS Boot Environment
but if you must, then just move these needed datasets
into the BE like that:

zfs rename -u zroot/usr/src zroot/ROOT/default/usr/src

[probonopd]

Some more thoughts on "over-installing" (installing over an existing OS):

How can we come up with a list of files in /etc that must not be touched by over-installing a new OS? /etc/passwd is an obvious one but which others are like it?

Conceptually, i think what we'd want is a diff from the old to the new version, and apply that diff to /etc during an update. So that changes made locally by the user would persist, but changes made by the new OS version would also get applied.

And it is completely unclear to me how the package database would have to be treated. Because the update would clearly install newer versions of the packages, while user-installed packages would still be on the system.

I think this is roughly what freebsd-update can do.

Probably sysadmins in large corporations have quite some tricks on their sleeves for this sort of thing...

[probonopd]

Assume we want to implement this:

When running from the Live ISO, can we get the whole zpool with all of its datasets mounted under /tmp so that the installer could do its thing there without having to format the whole disk?

Apparently yes but not quite because we are missing, e.g., /mnt/etc and /mnt/boot:

# mount -t tmpfs tmpfs /mnt
# zpool import -f -R /mnt zroot

# mount
(...)
tmpfs on /mnt (tmpfs, local)
zroot on /mnt/zroot (zfs, local, noatime, nfsv4acls)
zroot/tmp on /mnt/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/audit on /mnt/var/audit (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/tmp on /mnt/var/tmp (zfs, local, noatime, nosuid, nfsv4acls)
zroot/usr/home on /mnt/usr/home (zfs, local, noatime, nfsv4acls)
zroot/usr/ports on /mnt/usr/ports (zfs, local, noatime, nosuid, nfsv4acls)
zroot/var/crash on /mnt/var/crash (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/var/mail on /mnt/var/mail (zfs, local, nfsv4acls)
zroot/var/log on /mnt/var/log (zfs, local, noatime, noexec, nosuid, nfsv4acls)
zroot/usr/src on /mnt/usr/src (zfs, local, noatime, nfsv4acls)

This is all way too complicated.

And how can we make a Boot Environment/writable clone of all of the above?

[probonopd]

So, trying the folloing approach:

1. Install helloSystem to disk
2. Create a Boot Environment from the running system using bectl create overinstalled and mount it using bectl mount overinstalled
3. Mount the new helloSystem ISO and mount its /boot/rootfs.uzip (e.g., by double-clicking it)
4. Install the contents of rootfs.uzip over the Boot Environment, excluding certain files (below) using cpdup -I -j0 -o -X /path/to/cpignore /media/ufs /tmp/be_mount.xxxx
5. Set the Boot Environment active and boot into it

How can we come up with a list of files in /etc that must not be touched by over-installing a new OS?

Maybe using a cpignore along the lines of

# These files should not be overwritten when using
#
# bectl create overinstalled
# bectl mount overinstalled
# cpdup -I -j0 -o -X /path/to/cpignore /media/ufs /tmp/be_mount.xxxx
# to install the operating system over an existing installation
# bectl activate overinstalled

*/etc/resolv.conf
*/etc/fstab
*/etc/passwd
*/etc/spwd.db
*/etc/pwd.db
*/etc/master.passwd
*/etc/group
*/etc/wpa_supplicant.conf
*/etc/login.conf.db
*/etc/localtime
*/etc/opiekeys
*/etc/hostid
# The next file contains zfs_enable="YES" which is CRUCIAL
*/etc/rc.conf
*/etc/rc.conf.local
*/etc/ssh/*

# Essential files for booting into a graphical desktop are even in /usr/local
*/usr/local/etc/sudo*
*/usr/local/etc/pam.d/*

# Contains hardcoded user account for autologin
*/usr/local/etc/slim.conf*

Allan Jude talks about this exact problem in https://youtu.be/YcdFln0vO4U?t=582.

(Side rant: This whole user accounts/home directories/passwords/pam/sudo/Xauthority/cookies thing is a bit... unwieldy. All of this appears to be scattered around in various files at different locations. One single database would be much neater. This whole thing seems to be very finicky.)

With this, we can reboot into the Boot Environment and we can log in as a user on the console but slim refuses to work, saying "Waiting for X server to begin accepting connections" in /var/log/slim.log.

xauth list says timeout in locking authority file /.Xauthority.

Why is it searching for that file in / rather than in $HOME? echo $HOME returns /. Why? Turns out /usr/home is not mounted at all. Why? Edit: Turns out that zfs_enable="YES" was missing in /etc/rc.conf. When we exclude this file from being overwritten, it works.

Relevant video: ZFS Powered Magic Upgrades Using boot environments for atomic in-place upgrades by Allan Jude

zfs_get_all.txt

[probonopd]

According to https://youtu.be/YcdFln0vO4U?t=1000 this can be used to boot the new Boot Environment once:

zfsbootcfg zfs:zroot/ROOT/overinstalled:

(TODO: Understand and potentially make use of kenv zfs-bootonce.)

[grahamperrin]

@jsm222 helloDesktop https://wiki.freebsd.org/helloDesktop seems like a very good idea, if it can be in ports for FreeBSD helloSystem/hello#161 and maintained.

[jsm222]

@jsm222 helloDesktop https://wiki.freebsd.org/helloDesktop seems like a very good idea, if it can be in ports for FreeBSD helloSystem/hello#161 and maintained.

Would be a nice thing indeed. I looked a little into it again today, but got caught up with qterminal crashing, when the menu is running.. When all packages needed for helloSystem are in the ports tree freebsd-upgrade and pkg upgrade would indeed not break anything.

[jsm222]

For the record

• Global dbus menus results in segfault when windowstate is restored. lxqt/qterminal#992

[grahamperrin]

@jsm222

• Consider building "proper" packages #51

[grahamperrin]

• security/openconnect-gui and editors/libreoffice-bn hello#377 (reply in thread)

…

• pkg check --dependencies --dry-run

pkg-check(8)

---

Maybe we should have Update.app run this command always as its first thing? …

If you allow pkg-upgrade(8) to complete its work (for your current case, you should probably run repeatedly), then pkg check --dependencies --dry-run should find nothing missing.

[m4lin]

Hello,
i hope i am not in the wrong post/issue, but before finding this post i decided to update the BSD system below hellosystem following those steps
https://www.freebsd.org/releases/13.2R/installation/#upgrade
now the system is updated, but i believes something wrong happened ... the video driver is not found anymore (standard i915) and quite a lot of error (e.g. runnig falkon or dock)
No more logout, no more reboot or halt from the menu (still working if i do sudo reboot / sudo halt from terminal but halt is hanging forever) and other stuff probably.

not a big issue is a test machine i am playing with so i can re-install the whole shebang

but i think i learned to not run the update of the system as i did.

which one is the right way? should i wait for the next ISO? just out of curiosity

thanks

M

[probonopd]

Hello @m4lin. Currently we don't have an update system in place. So you need to do a clean reinstallation of helloSystem every time. We know that we need to think about an update stratefy, but so far we haven't found a good way of doing it. One way would be to package all helloSystem components as proper FreeBSD packages, but so far no one has succeeded in doing this.

[louies0623]

#55 (comment)
This is what I expected too.

[probonopd]

Point in case:

helloSystem/hello#523

User updated all the packages using pkg, now the system is broken and there is no more Xorg.