Zfs promote results in immediate and total system freeze

[olidal]

System information

Type	Version/Name
Distribution Name	Proxmox
Distribution Version	7.4.1
Kernel Version	5.15.158-2
Architecture	x86_64
OpenZFS Version	2.1.15

Describe the problem you're observing

zfs promote operation results in total freeze of system.

Describe how to reproduce the problem

• This pool receives (incremental) replication streams from multiple sources (backups)
• Replication streams are compressed (lz4) and cyphered (sha256) but not deduped
• The cypher key is not available, all operation apply to cyphered, unmounted datasets
• Replication streams consist of "cascades" of snapshots and clones, eg:
  dsv0 -> dsv0@ro -> dsv1 -> dsv1@ro -> dsv2 -> ... dsvn@ro
• Notice all snapshots are named @ro

At some point I start pruning the cascade to get rid of old versions, eg. to get rid of everything before a given version.
For example, to get rid of everything before dsv4, I would apply iteratively the following sequence of 4 operations to dsv3, dsv2 and dsv1:

1. zfs rename dsv3@ro dsv3@ro_old
2. zfs promote dsv4
3. zfs destroy -R dsv4@ro_old
   4. zfs destroy -r dsv3 (example edited: this destroy step is useless)
   (And repeat with dsv2, and dsv1)

The issue happens in a total reproducible way, when I reach step2 for the last iteration, ie when the origin of dsv4 is dsv0@r0, after the intermediate iterations completed without error (dsv1 and dsv2 successfully deleted).

However, the pool hosts many such "cascades", but the issue does NOT happen with all of them.

The pool has no error, and last scrub completed without any error.
The pool has not log or cache device.
I also had quite a few errors using bookmarks in the past and totally gave up using them.
The datasets on which the error occurs have no bookmarks.

The pool was not created using full devices but with partitions, with the following command:

root@stor4:~# zpool history zpool | head
History for 'zpool':
2022-04-13.23:34:30 zpool create -o ashift=12 -o feature@async_destroy=enabled -o feature@bookmarks=enabled -o feature@embedded_data=enabled -o feature@empty_bpobj=enabled -o feature@enabled_txg=enabled -o feature@extensible_dataset=enabled -o feature@filesystem_limits=enabled -o feature@hole_birth=enabled -o feature@large_blocks=enabled -o feature@lz4_compress=enabled -o feature@spacemap_histogram=enabled -o feature@zpool_checkpoint=enabled -O acltype=posixacl -O compression=lz4 -O normalization=formD -O relatime=on -O xattr=sa zpool raidz1 /dev/disk/by-id/wwn-0x5000c50084fa59f3-part4 /dev/disk/by-id/wwn-0x5000c500855c7907-part4 /dev/disk/by-id/wwn-0x5000c50097ce49bb-part4 /dev/disk/by-id/wwn-0x5000c50097cea04f-part4 /dev/disk/by-id/wwn-0x5000c50097cf5663-part4 /dev/disk/by-id/wwn-0x5000c50097cf59ab-part4

I have the same setting on multiple backup storage systems and the issue seems to happen randomly on some of them.
I also tried to replicate the whole "cascade" to another server with latest promox kerenl and zfs version 2.2.7 and the problem happens exactly the same.

Include any warning/errors/backtraces from the system logs

No error log, no console message, no disk activity, network connexions are frozen.
NIC leds are still blinking though.

Please advise on how to increase kernel module and/or kernel log level.
If needed I might be able to recompile ZFS module, I had the tool-chain working some time ago.
I would love to be able to activate a higher level of log to trace operations just before starting the evil promote.

[GregorKopka]

Unrelated to the freeze, which should not happen:
Is there a particular reason why you do this with chains of clones, instead of simply creating an additional snapshot on the source and receiving the incremental to the last common one into one target dataset (and, if needed to r/w access a snapshot, create a clone, that you can destroy any time, since nothing else depends on it)?

[olidal]

Hi Gregor, The reason is simple: I am reusing an OSS backup software (urbackup) that does it that way. I always had the same questioning as you but never found the time to dig into Urbackup's algos and I was happy with this approach ... until now. I suspect one reason that motivated this approach could be that the author wanted to emulate read-write snapshots with ZFS RO snaps and clones, like in BtrFS, such that switching from one backend to the other can be transparent for upper algorithmic layers, Ie, the reasoning could be : I need a "logical" RW snapshot with ZFS, so I take a native ZFS RO snap, and clone it. Then iterate on the sequence snap + clone to produce each subsequent "logical RW snap", which produces the resulting "cascades" I described.

[GregorKopka]

Regarding your demo-code:
Step 4 should give an error message, as step 3 already destroyed the (after promote) clone dsv3 with origin dsv4@ro_old.

On Topic:
Looking at Urbackup I think I found a kind-of of your issue at https://forums.urbackup.org/t/incremental-image-bug-causing-kernel-panic/7695 and the respective OpenZFS issue (#3959) - first discovered from 2019 - where it bombed with a stack overflow when recursively destroying too long chains of dependent clones... not while promoting a clone, as in your case.

@behlendorf, do you have an idea what could happen here ?

Sidenote:
In case the goal is replicating ZFS datasets to other ZFS capable machines (= no Windows / file based backups)... I suggest looking for a snapshot based tool dedicated to this use case.
That would not only give less convoluted results by skipping the chain of clones (making cleanup of old versions and staggered deletes -keeping a certain amount of hourly, daily, weekly, ... snapshots as a time machine- easily manageable and, when using timestamps for the snapshot names, also understandable) and also would allow to backup native ZFS encrypted datasets onto '"untrusted" backup servers (by simply not giving them the key for decryption), while maintaining the data integrity guarantee of ZFS both in flight with send|recv and at rest on-disk (everything is protected/verified by a strong checksum).

[olidal]

@GregorKopka Yes, you are right, destroy -r is useless in the example I gave, I think it is only useful when dealing with corner case (eg broken string of snap and clones after a full backup rather than incremental).

Thanks for digging the problem, I do indeed remember having other issues with zfs 0.8.x, but I think most of them got resolved with zfs 2.x or by changing my habits (I used to jail zfs in proxmox/LXC containers, and sometimes zfs binaries ended up not matching the running kernel, which often oopsed. I am very careful with that mismatch issue now).

About sidenote: I do use urbackup to produce bloc level incremental snaps of a windows volume (aka "image" mode backup). This first level of backup is sent by Urbackup on my first level ZFS storage where Urbackup builds the "cascading" backup snaps. Then I replicate this first level ZFS storage to another site using ZFS replication streams (actually using a slightly modified version syncoid).