AutoRotate

[artem-bm]

Hello,

The auto mode lacks the function of the maximum number of incremental backups.
In the number of days or copies created.

[abbbi]

i dont think i will work on this, you can easily work around that by scripting via bash or alike.
See also:

https://github.com/abbbi/virtnbdbackup/blob/master/scripts/vmbackup.sh
or
https://github.com/slackdaystudio/vm-snapshot

[sentry0]

Hey, I'm the author of those scripts. I caution you against using the prune option in those scripts.

The prune option leaves the backups inconsistent because the script doesn't manage dirty bitmaps properly. I think it's fixable, it's just a pain to do.

I'm re-writing the whole thing to be a node script this weekend. I plan on making a more robust backup system that includes managing bitmaps. I may even look for a node file rotation system for better backup rotations.

[abbbi]

Should i remove the Script from the repo?

[sentry0]

If you don't mind, you may want to pull it. The rotation looks broken, and I would not want people to rely on it.

Sorry for the trouble @abbbi My node script is working, I need to drop in the logic for rotations and cleaning up bitmaps next. If you would consider a PR I will send one your way.

[sentry0]

Here is what I'm currently using to rotate my backups - https://github.com/slackdaystudio/vmsnap

This NodeJS application is identical to the original bash script but also scrubs bitmaps and checkpoints off VMs when rotating.

I'm in the middle of testing it now. My tests are coming back green, but I want to wait until the end of the month to see if the new checkpoint will be cut properly.

[rmk177]

Let me reopen discussion. As far as I understand, the proper way is to merge backups is using some daemon on host, that monitor autorotate periods, but for now we have two standalone applications for backups and restore, that don't suits for this. Neverless we can use some commandline options that instructs on next backup make autorotate. For example, --groupBy monthly --rotate-depth 2 on next daily virtnbdbackup merge incremental backups for previous periods to next full backup and cleans not needed backups for specified periods. As far as I understand these operations makes image on restore for backup chain, so we can reuse them.

[sentry0]

Let me reopen discussion. As far as I understand, the proper way is to merge backups is using some daemon on host, that monitor autorotate periods, but for now we have two standalone applications for backups and restore, that don't suits for this. Neverless we can use some commandline options that instructs on next backup make autorotate. For example, --groupBy monthly --rotate-depth 2 on next daily virtnbdbackup merge incremental backups for previous periods to next full backup and cleans not needed backups for specified periods. As far as I understand these operations makes image on restore for backup chain, so we can reuse them.

The vmsnap Node app can produce snapshots based on a configurable time period (using the --groupBy flag). This will lay out the backups in directories named after the period in a specific format. If you ran a new monthly snapshot today (2025-02-26) you would get a folder named DOMAIN_NAME/vmsnap-backup-monthly-2025-02 in your backup dir root.

This means at the start of every new period (month, quarter, bi-annual, year) vmsnap creates a new full backup and erases the old checkpoints from the domain. You can track down the correct period folder if you need to roll back to a point in time.

The side effect is that all periods start with a full backup and incremental backups afterward. This is not great for space if you run daily backups with --groupBy=month on large domains, as it produces one full backup every month.

I set my backups to be --groupBy=bi-annual. That will produce one full backup every 6 months and is suitable for storage for my needs. I also use the --prune flag to clean up old directories automatically around the 90-day mark of the current period.

This is the full command I run to backup my VMs:

vmsnap --backup --domains=DOMAINS --output=BACKUP_ROOT --groupBy=bi-annual --prune

This produces the following directory structure in my BACKUP_ROOT location:

/BACKUP_ROOT/
├── [   3]  Domain_1
│   └── [  14]  vmsnap-backup-bi-annually-2025-p1
├── [   3]  Domain_2
│   └── [  35]  vmsnap-backup-bi-annually-2025-p1
├── [   3]  Domain_3
│   └── [  76]  vmsnap-backup-bi-annually-2025-p1
├── [   3]  Domain_4
│   └── [   9]  vmsnap-backup-bi-annually-2025-p1
└── [   3]  Domain_5
    └── [  32]  vmsnap-backup-bi-annually-2025-p1

Does this help explain the scheduling part, or have I added confusion?

[abbbi]

see also:

https://mizu.reisen/posts/2025-01-08_kvm-backup/