zfs-linux-lts-poscat sd-zfs does not support zfs native encryption

[darkbasic]

问题类型 / Type of issues

• 其它 / other

受影响的软件包 / Affected packages

• zfs-linux-lts-poscat

---

Hi, I'm trying to boot Arch Linux with sd-zfs (from the zfs-linux-lts-poscat package) + zfsbootmenu (zfs set org.zfsbootmenu:commandline="rw" zroot/ROOT) + zfs-import-scan + zfs-mount-generator but I'm getting stuck at "Failing to start Switch Root":

HOOKS=(systemd sd-zfs autodetect microcode modconf kms keyboard sd-vconsole block filesystems fsck)

/etc/zfs/zfs-list.cache/zroot has been populated and /etc/zfs/zpool.cache does not exist.

If I switch to the zfs hook (HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block zfs filesystems fsck)) it works:

zfsbootmenu ignores any root entry and forcefully provides a zfs=zroot/ROOT/arch parameter.
I don't know if that might the problem, because if you don't provide a root parameter it should default to the bootfs which has been correctly set (zpool set bootfs=zroot/ROOT/arch zroot).

Any idea how to debug/fix this issue?

Please note that I'm not using the whole zfs-linux-lts-poscat package (I want to use zfs-dkms-staging-git/zfs-utils-staging-git along with the latest stable kernel) but instead a trimmed down version which provides just the sd-zfs hook:

package_sd-zfs-poscat() {
    pkgdesc='ZFS mkinitcpio hook for systemd based initrds'

    install -Dvm644 "${srcdir}"/sd-zfs.initcpio.install "${pkgdir}"/usr/lib/initcpio/install/sd-zfs

    install -Dvm755 -t "${pkgdir}"/usr/lib/zfs/initcpio/ "${srcdir}"/{parse-cmdline,zfs-set-env,zfs-root-generator}
    install -Dvm755 -t "${pkgdir}"/usr/lib/systemd/system-shutdown/ "${srcdir}"/zfs.shutdown
}

If I manage to get it working with zfsbootmenu I would like to provide an sd-zfs-poscat AUR package, with your permission.

[darkbasic]

I've just discovered org.zfsbootmenu:rootprefix (https://docs.zfsbootmenu.org/en/v2.3.x/man/zfsbootmenu.7.html) and thus I have been able to pass root=zfs:zroot/ROOT/arch instead of zfs=zroot/ROOT/arch

Now I get Failed to mount /sysroot instead.

At this point I think that I should also clarify that I'm using native encryption, but it shouldn't be an issue given that no passphrase should be asked:

 -O encryption=aes-256-gcm \
 -O keylocation=file:///etc/zfs/zroot.key \
 -O keyformat=passphrase \

At this point it would be useful to see the echo commands from the sysytemd services in the console: do you know how to achieve it?

[darkbasic]

I've been able to unlock the root account by adding add_file /etc/shadow /etc/shadow to /usr/lib/initcpio/install/sd-zfs, so with the busybox recovery shell available I've been finally able to run journalctl -xb:

zfs_mount_at() failed: encryption key not loaded

This is clearly an encryption issue. Running zfs load-key -L /etc/zfs/zroot.key zroot should be enough to load the key I guess, but I'm not sure why the standard zfs hook doesn't have this issue.

[darkbasic]

It looks there is a runtime hook for the busybox init which takes care of decrypting and which obviously don't get run with sd-zfs: https://aur.archlinux.org/cgit/aur.git/tree/zfs.initcpio.hook?h=zfs-utils-staging-git#n29

[darkbasic]

I got it working by adding ExecStartPost=/usr/bin/zfs load-key zroot to zfs-import-root-pool.service here: https://github.com/archlinuxcn/repo/blob/master/archlinuxcn/zfs-linux-lts-poscat/zfs-root-generator#L110
But it's just an hack: sd-zfs needs to implement proper encryption support to be a viable alternative.

[poscat0x04]

Yep, currently it does not support zfs native encryption. I currently do not have plan to implement support (as the functionality largely overlaps with LUKS and it seems to be abandoned with various bugs not fixed), but that shouldn't be hard to implement given that zfs-mount-generator already exists and supports generating all the systemd services required to decrypt datasets.

[darkbasic]

as the functionality largely overlaps with LUKS and it seems to be abandoned with various bugs not fixed

I don't agree.

First native encryption and LUKS don't overlap: I challenge you to send/receive encrypted backups without knowing the source encryption key with LUKS. I use it all the time to send incremental backups/snapshots, sometimes even to external USB disks.

Also the fact that Tom Caputi left the project doesn't mean that native encryption is abandoned, on the contrary he is still available to review PRs: https://discourse.practicalzfs.com/t/the-future-of-the-zfs-native-encryption/1037
Native encryption is being used in production with multi petabytes storage.

Do we want to talk about LUKS related issues? Several users of the zfs-dkms-staging-git package I co-maintain lost their data due to this zfs bug with LUKS: openzfs/zfs#16631

I might give this a shot if you're willing to review the code, but first I would like to know if it would be possible to host sd-zfs.initcpio.install, parse-cmdline, zfs-set-env, zfs-root-generator and zfs.shutdown in their own git repo somewhere. That would come in handy for both development and packaging.

[poscat0x04]

as the functionality largely overlaps with LUKS and it seems to be abandoned with various bugs not fixed

I don't agree.

First native encryption and LUKS don't overlap: I challenge you to send/receive encrypted backups without knowing the source encryption key with LUKS. I use it all the time to send incremental backups/snapshots, sometimes even to external USB disks.

Also the fact that Tom Caputi left the project doesn't mean that native encryption is abandoned, on the contrary he is still available to review PRs: https://discourse.practicalzfs.com/t/the-future-of-the-zfs-native-encryption/1037 Native encryption is being used in production with multi petabytes storage.

I see.

I might give this a shot if you're willing to review the code, but first I would like to know if it would be possible to host sd-zfs.initcpio.install, parse-cmdline, zfs-set-env, zfs-root-generator and zfs.shutdown in their own git repo somewhere. That would come in handy for both development and packaging.

Yes, I'm willing to review and maintain the code. I will make a separate repo soon.

[poscat0x04]

@darkbasic Apologies for the delay.

All the necessary code are now placed here. Some dev notes are also supplied. If you have any questions about the code just open an issue.