Incremental backup for RAW storage backend volumes

[jneuhauser]

Great project, but haven't looked at the code base for now...

When I read the documentation and realized that LVM thin provisioned volumes are not supported, I wondered how much work it would be to add support for incremental backups of LVM thin provisioned volumes using the thin pool metadata and thin snapshots.

I have seen incremental backups of LVM thin provisioned volumes on wyng-backup, which could serve as a first starting point.

So initially the main questions before doing any coding are:

• Does it make sense to support LVM thin provisioned volumes for VM backups?
• Does this feature fit into the current code base or do we need huge preparations/reworks?
• How would be the first starting point?

[abbbi]

hi,

i created virtnbdbackup to use the latest features available in qemu, means dirty bitmaps. The current state is that dirty bitmaps make most sense if they are persistent. Persistent bitmaps are saved in the qcow image and are kept even if the virtual machine is powered off, or if the hypervisor restarts.

If a virtual machine uses non qcow image for storage (lvm, lvm-thin, ceph+RDB, whatever RAW device) the bitmaps for these disks are lost once the virtual machine is powered off. This is a limitation in Qemu and i dont think this will be fixed anytime in the future (The argument is that it basically requires a new metadata format to be invented, which qcow2 basically is), see:

https://lists.gnu.org/archive/html/qemu-devel/2021-03/msg07448.html

and:

https://github.com/abbbi/virtnbdbackup/blob/master/libvirtnbdbackup/virt/client.py#L426

So for Virtual machines that use non-qcow backed storage there is only one option currently:

1. Create a full backup for these disks only without creating a checkpoint

Basically every raw storage backend such as lvm+thin, ceph/rdb, ZFS has a way to create incrementals/differentials based on storage volume snapshots. For LVM thin the device mapper backend can be used and for other storage backends there are special functions to get the changed blocks, most of the time this requires to keep an reference snapshot around to compare against.

Currently the code base in virtnbdbackup is not designed to handle multiple storage backends correctly, and the feature you want to see would basically mean to rip off 60% of the wyng-backup code and port it into virtnbdbackup.

If someone would want to implement storage based incremental backups for virtnbdbackup, that would mean:

1. re-design the complete code base to be modular and provide different storage plugins (qcow2, lvm, lvm+thin, ceph+rdb,zfs)
2. handle all the nitty gritty edge cases of those different storage backends
3. handle remote backup of those storage backends well.
4. Create testcases for all those setups to verify restore functions work correctly.

This is lots of work and personally i dont have the time to implement all this, nor the will to maintain.

[jneuhauser]

I almost thought so, but I had a tiny ray of hope and that's why I asked.

Thank you for your statement and your work on virtnbdbackup.

[abbbi]

Well, qemu allows to create non-persistent bitmaps, so you can do incremental backups for raw backed volumes. As long as the virtual machine isnt powered off, it will work. If the bitmaps are lost, a new full backup must be created.

See:

https://www.qemu.org/docs/master/interop/bitmaps.html#supported-image-formats

From a qemu standpoint, it seems OK, but libvirt explicitly denies the creation of checkpoints for non-qcow backed images,
see:

https://github.com/libvirt/libvirt/blob/master/src/qemu/qemu_checkpoint.c#L363

So, if in future, libvirt allows to create non-persistent checkpoints for RAW volume, virtnbdbackup could at least create incrementals as long as VM stays online (no matter what the real storage backend is, it would work for ceph/zfs/etc).

That would be something i would be willing to implement, but everything else is just too much work.

[abbbi]

short discussion on #virt irc.

10:27 <abi> hm.. qemu allows to define non-persistent bitmaps (for example for raw devices), libvirts prevents this (qemu_checkpoint.c:365:>src->format != VIR_STORAGE_FILE_QCOW2), why?
13:50 <pkrempa> well because they don't make much sense
13:51 <pkrempa> if you turn off your VM you lose the state
19:52 <abi> pkrempa: yes, thats true, but i think its up to the backup software to handle these situations gracefully then..
19:54 <abi> just like it has to handle situations where the bitmap is lost due to sudden power outage ..
20:11 <pkrempa> the thing is that it doesn't tie in well with the idea of checkpoint metadata
20:11 <pkrempa> e.g. it'd have to be done such that the checkpoint vanishes as well
20:11 <pkrempa> basically, too much hassle; too little utility
20:12 <pkrempa> for the common user it's better to mandate qcow2 and be done