Does rustic handle filesystem snapshots?

[kapitainsky]

Good practice for backups is to use modern filesystem (BTRFS, ZFS, APFS or LVM) snapshots .

Does rustic handles it or does it have restic inherited problem (restic/restic#3041)?

In a nutshell restic relies on device ID (which is not stable for snapshots) and it causes breaking snapshots inheritance - deduplication of course works but all content metadata is treated as changed. It leads to extra space usage and performance degradation - maybe not critical for small datasets but very painful for big and frequent backups.

I see that there is an option available:

      --ignore-devid
          Don't save device ID for files and directories

which sounds like maybe the option to use. However I could not find anything more specific, e.g. by using this option does rustic is still able to handle hard links?

[kapitainsky]

I actually tried to answer some of my questions empirically. I have used macOS APFS filesystem and Time Machine snapshots. I can share details if anybody is interested.

My findings:

1. rustic backup /path/to/mounted/snapshot/mydata --as-path /mydata --ignore-devid seems to do the trick for backup. Subsequent runs with different APFS snapshots but with no data changes result with identical backup snapshots - no extra metadata uploaded.
2. When restoring such backup using rustic any pre existing hard links are gone (replaced by individual files), everything else seems to be ok. Not ideal - this is probably what missing devID leads too.
3. When restoring using restic hard links are preserved but files/dirs ownership and permissions are gone completely - do --as-path and --ignore-devid make repo not fully compatible with restic?

And now loudly thinking - wouldn't it make sense instead (or maybe in addition) of --ignore-devid to have --replace-devid which would use devID from --as-path

Let's say I have /Users/me/data folder to backup. I run full filesystem snapshot and mount it as /mnt/rustic-snapshot. Now I could run:

rustic backup /mnt/rustic-snapshot/Users/me/data --as-path /Users/me/data --replace-devid

And backup would use my physical device ID (derived from --as-path) instead of temporary filesystem snapshot mount one which effectively would work like backup up physical /Users/me/data directory but with data coming from /mnt/rustic-snapshot/Users/me/data

Limitation of --replace-devid I can see is that it would only allow backing up single filesystem but it can be explained in the documentation - BTW I am happy to document filesystem snapshots usage with examples for e.g. APFS and ZFS.

Advantage would be that as I understand restic/rustic internals it would nicely fit into existing logic - no need to create completely new dev path how to nicely handle filesystem snapshots. And resulting repo would be 100% compatible with existing restic/rustic format.

Overall I think that it is problem worth to tackle at last. I can see on restic forum more and more people trying to use filesystem snapshots. I do not think that restic can have it sorted anytime soon (given that restic/restic#3041 is many years old). Maybe rustic can come for the rescue here?

[aawsome]

@kapitainsky Thanks for opening this topic and thanks for sharing your results.

About file system snapshots - you already found what rustic currently supports 👍

About the hard links: This is not yet supported in rustic. The point is that the repository format doesn't support hardlinks and even restic just uses a heuristic during restore which tries to identify hard links. But actually, hard links are not easy to handle. Before doing something half-working (as the restic solution probably would be) I decided to not support it at all in the first run.
I am however very open to a discussion how to support it and how to treat all kinds of edge cases.

About

3. When restoring using restic hard links are preserved but files/dirs ownership and permissions are gone completely - do --as-path and --ignore-devid make repo not fully compatible with restic?

This behavior is actually not expected. Can you try to dig deeper into this and help with debugging the problem? For instance, if you have a dir which is not correctly restored by restic, can you run rustic cat tree <SNAPSHOT>:<PATH> on it and give the output together with what restic restore produces? Thanks a lot in advance!

About the replace-devid idea: In principle something like this is possible. As you mentioned, it poses problems, though, if you backup a dir containing multiple devices. Also, the device id is not really needed for a backup, especially as there are cases where it is expected that it changes and moreover there is no way to really set/restore it during restore. The only reason for backing it up with restic is the hard-link heuristic during restore. So I would propose to discuss the hard-link topic and seek a solution of it (being open to a solution including devids) instead.

[kapitainsky]

About file system snapshots - you already found what rustic currently supports 👍

And many more things: --glob, --git-ignore, cold storage support and rustic restore can be interrupted and resumed. These are few things I think are no-brainer and make backup process so much more efficient and easy. Very impressive all project.

I have been keeping an eye on this project for some time but since started playing with it I can see myself leaving restic soon probably:) In general I am more than happy that there are two projects supporting the same repo format - it gives massive reassurance.

About the hard links: This is not yet supported in rustic. The point is that the repository format doesn't support hardlinks and even restic just uses a heuristic during restore which tries to identify hard links. But actually, hard links are not easy to handle. Before doing something half-working (as the restic solution probably would be) I decided to not support it at all in the first run.
I am however very open to a discussion how to support it and how to treat all kinds of edge cases.

Very clear now re hard links. Another reason why two different programs are great to have:) So far I am happy with my tests and restic hard links heuristic. I do not mind to mix usage of both programs - whichever suits my needs better in given scenario.

3. When restoring using restic hard links are preserved but files/dirs ownership and permissions are gone completely - do --as-path and --ignore-devid make repo not fully compatible with restic?

This behavior is actually not expected. Can you try to dig deeper into this and help with debugging the problem? For instance, if you have a dir which is not correctly restored by restic, can you run rustic cat tree <SNAPSHOT>:<PATH> on it and give the output together with what restic restore produces? Thanks a lot in advance!

Will do. No single object is restored correctly by restic.... Content is OK but if I backup

rustic backup /tmp/rustic-snapshot/Users/kptsky/Downloads --as-path /Users/kptsky/Downloads--ignore-devid

and restore using restic then Users, kptsky and Downloads directories have permissions/ownership:

d---------@ 99 _unknown _unknown

All content of Downloads have correct permissions but missing ownership, e.g. (but hard links are here):

-rw-r--r--@ 5 _unknown  _unknown  7 30 Dec 14:10 a
-rw-r--r--@ 5 _unknown  _unknown  7 30 Dec 14:10 b
-rw-r--r--@ 5 _unknown  _unknown  7 30 Dec 14:10 c
-rw-r--r--@ 5 _unknown  _unknown  7 30 Dec 14:10 test.txt

when I do the same restore using restic all is OK:

drwxr-xr-x@ 3 admin  staff    96B 30 Dec 20:45 Users

no hard links but all permissions are OK

-rw-r--r--@ 1 admin  staff     7B 30 Dec 14:10 a
-rw-r--r--@ 1 admin  staff     7B 30 Dec 14:10 b
-rw-r--r--@ 1 admin  staff     7B 30 Dec 14:10 c
-rw-r--r--@ 1 admin  staff     7B 30 Dec 14:10 test.txt

About the replace-devid idea: In principle something like this is possible. As you mentioned, it poses problems, though, if you backup a dir containing multiple devices. Also, the device id is not really needed for a backup, especially as there are cases where it is expected that it changes and moreover there is no way to really set/restore it during restore. The only reason for backing it up with restic is the hard-link heuristic during restore. So I would propose to discuss the hard-link topic and seek a solution of it (being open to a solution including devids) instead.

Ok - it is also clear now. If hard-link topic is cleared than maybe --replace-devid is not needed at all - at least from my perspective. I could always use restic and their heuristics.

[kapitainsky]

This behavior is actually not expected. Can you try to dig deeper into this and help with debugging the problem? For instance, if you have a dir which is not correctly restored by restic, can you run rustic cat tree <SNAPSHOT>:<PATH> on it and give the output together with what restic restore produces? Thanks a lot in advance!

# backing up from mounted APFS snapshots on the latest macOS
$ rustic backup /tmp/rustic-snapshot/Users/kptsky/Downloads --as-path /Users/kptsky/Downloads --ignore-devid

$ rustic cat tree 431ba858:/Users/kptsky/Downloads/hardlinks
{"nodes":[{"name":"a","type":"file","mode":420,"mtime":"2023-12-30T14:10:12.988185968Z","atime":"2023-12-30T14:10:12.988185968Z","ctime":"2023-12-30T14:11:27.753605328Z","uid":501,"gid":20,"user":"kptsky","group":"staff","inode":314775716,"size":7,"links":5,"extended_attributes":[{"name":"com.apple.provenance","value":"AQAAgNkrPpOkGrc="}],"content":["61b4c705859f4158d38090c1e38e8fdc4f3d29db007f012766276aa498835cf6"]},{"name":"b","type":"file","mode":420,"mtime":"2023-12-30T14:10:12.988185968Z","atime":"2023-12-30T14:10:12.988185968Z","ctime":"2023-12-30T14:11:27.753605328Z","uid":501,"gid":20,"user":"kptsky","group":"staff","inode":314775716,"size":7,"links":5,"extended_attributes":[{"name":"com.apple.provenance","value":"AQAAgNkrPpOkGrc="}],"content":["61b4c705859f4158d38090c1e38e8fdc4f3d29db007f012766276aa498835cf6"]},{"name":"c","type":"file","mode":420,"mtime":"2023-12-30T14:10:12.988185968Z","atime":"2023-12-30T14:10:12.988185968Z","ctime":"2023-12-30T14:11:27.753605328Z","uid":501,"gid":20,"user":"kptsky","group":"staff","inode":314775716,"size":7,"links":5,"extended_attributes":[{"name":"com.apple.provenance","value":"AQAAgNkrPpOkGrc="}],"content":["61b4c705859f4158d38090c1e38e8fdc4f3d29db007f012766276aa498835cf6"]},{"name":"test.txt","type":"file","mode":420,"mtime":"2023-12-30T14:10:12.988185968Z","atime":"2023-12-30T14:10:12.988185968Z","ctime":"2023-12-30T14:11:27.753605328Z","uid":501,"gid":20,"user":"kptsky","group":"staff","inode":314775716,"size":7,"links":5,"extended_attributes":[{"name":"com.apple.provenance","value":"AQAAgNkrPpOkGrc="}],"content":["61b4c705859f4158d38090c1e38e8fdc4f3d29db007f012766276aa498835cf6"]}]}

# restoring to external disk
$ restic restore 431ba858 --target /Volumes/SSD-500-Temp/restore

# have to use sudo as top dirs have very "restricted permissions":)

$ sudo ls -l /Volumes/SSD-500-Temp/restore/
total 0
d---------  3 admin  staff    96B 30 Aug  1754 Users

# on this level permissions are ok - only ownership is not
$ sudo ls -l /Volumes/SSD-500-Temp/restore/Users/kptsky/Downloads/
...
drwxr-xr-x@  6 _unknown  _unknown       192 30 Dec 14:11 hardlinks
...


$ sudo ls -l /Volumes/SSD-500-Temp/restore/Users/kptsky/Downloads/hardlinks
total 32
-rw-r--r--@ 5 _unknown  _unknown  7 30 Dec 14:10 a
-rw-r--r--@ 5 _unknown  _unknown  7 30 Dec 14:10 b
-rw-r--r--@ 5 _unknown  _unknown  7 30 Dec 14:10 c
-rw-r--r--@ 5 _unknown  _unknown  7 30 Dec 14:10 test.txt

[kapitainsky]

The same as above but using rustic

$ rustic restore 431ba858 /Volumes/SSD-500-Temp/restore/

$ ls -l /Volumes/SSD-500-Temp/restore/
total 0
drwxr-xr-x@ 3 admin  staff    96B 30 Dec 21:12 Users

$ ls -l /Volumes/SSD-500-Temp/restore/Users/kptsky/Downloads/hardlinks
total 32
-rw-r--r--@ 1 admin  staff     7B 30 Dec 14:10 a
-rw-r--r--@ 1 admin  staff     7B 30 Dec 14:10 b
-rw-r--r--@ 1 admin  staff     7B 30 Dec 14:10 c
-rw-r--r--@ 1 admin  staff     7B 30 Dec 14:10 test.txt

[kapitainsky]

Actually I think there is no issue with ownership/permissions on /Volumes/SSD-500-Temp/restore/Users/kptsky/Downloads/ level.

_unknown _unknown ownership is due to me using sudo on external disk. The real issue and reason I have to use sudo are permissions of Users, kptsky and Downloads:

d--------- 3 admin staff 96B 30 Aug 1754 Users

$ rustic cat tree 431ba858:
{"nodes":[{"name":"Users","type":"dir","content":null,"subtree":"4468d65002770655452b697d1d0eb445c356c3a8e329f23f043716f6ec95bb22"}]}

$ rustic cat tree 431ba858:/Users
{"nodes":[{"name":"kptsky","type":"dir","content":null,"subtree":"38b7fa5eca3d21d3c1aa2c2fe4d83890aeab9ecbffe6aa80078c6695d6857d47"}]}

$ rustic cat tree 431ba858:/Users/kptsky
{"nodes":[{"name":"Downloads","type":"dir","content":null,"subtree":"8009a895f391f0c19e7c1d711c24b33fb08b8efdb785e9bb2c08a9893ad28349"}]}

# only fragment here - `hardlinks` directory
$ rustic cat tree 431ba858:/Users/kptsky/Downloads
...
"content":["5492849dc3eaf777214ea463dcb77f0d66f1b14c4fa63bca5d4bd6c42e74a03c"]},{"name":"hardlinks","type":"dir","mode":2147484141,"mtime":"2023-12-30T14:11:01.439874229Z","atime":"2023-12-30T14:11:01.439874229Z","ctime":"2023-12-30T14:11:01.439874229Z","uid":501,"gid":20,"user":"kptsky","group":"staff","inode":314775693,"extended_attributes":[{"name":"com.apple.provenance","value":"AQAAgNkrPpOkGrc="}]
...

when using sudo I copy content of /Volumes/SSD-500-Temp/restore/Users/kptsky/Downloads/ somewhere else then all is OK. So the issue is with top level directories.. very weird.

[aawsome]

the issue is with top level directories.. very weird.

Thanks for digging into it! The point is that these top level dirs are soly virtual dirs which are set by --as-path but in fact don't (need to) exist on the filesystem. Therefore, rustic isn't able to save permissions/ownership for them.

When restoring using rustic, rustic restore 431ba858 /Volumes/SSD-500-Temp/restore/ in fact does rustic restore 431ba858:/Volumes/SSD-500-Temp/restore/ /Volumes/SSD-500-Temp/restore/, i.e. it doesn't restore these top-level dirs from the snapshot but instead just creates /Volumes/SSD-500-Temp/restore/ using mkdirall - using the ownership of the user running the command.

To do the same restore with restic, use restic restore 431ba858:/Volumes/SSD-500-Temp/restore/ /Volumes/SSD-500-Temp/restore/ - I however don't know if restic creats the destination dir for you or if you have to self-create it in advance.

[kapitainsky]

Yes it is clear now... but I keep finding more weird issues, e.g. I create new repo, backup stuff. Then I remove from source big folder. Create second snapshot and forget the first one.

I would expect now rustic prune to delete tones of repo files but nope:

$ rustic prune
using config ./rustic.toml
[INFO] repository local:/Volumes/SSD-500-Temp/rustic-test: password is correct.
[INFO] using cache at /Users/kptsky/Library/Caches/rustic/c923f1a79e436c96db128c6a510a6e062e6d3ad03821f095bb4ca72546c480d8
[00:00:00] reading index...               ████████████████████████████████████████          1/1
[00:00:00] reading snapshots...           ████████████████████████████████████████          1/1
[00:00:00] finding used blobs...          ████████████████████████████████████████          1/1
[00:00:00] getting packs from repository...                                                                                                                                               to repack:          0 packs,          0 blobs,        0 B
this removes:                         0 blobs,        0 B
to delete:          0 packs,          0 blobs,        0 B
unindexed:          0 packs,         ?? blobs,        0 B
total prune:                          0 blobs,        0 B
remaining:                          634 blobs,  513.6 MiB
unused size after prune:        0 B (0.00% of remaining size)

packs marked for deletion:         89,    2.9 GiB
 - complete deletion:               0,        0 B
 - keep marked:                    89,    2.9 GiB
 - recover:                         0,        0 B
[INFO] nothing to do!

It shows packs marked for deletion and does nothing... I've tried --max-unused 0 - the same. The repo size remains unchanged.
And strange line unindexed: 0 packs, ?? blobs, 0 B

But then when I run old good restic:

$ restic prune
repository c923f1a7 opened (version 2, compression level auto)
created new cache in /Users/kptsky/Library/Caches/restic
loading indexes...
[0:00] 100.00%  1 / 1 index files loaded
loading all snapshots...
finding data that is still in use for 1 snapshots
[0:00] 100.00%  1 / 1 snapshots
searching used packs...
collecting packs for deletion and repacking
[0:00] 100.00%  17 / 17 packs processed

to repack:             0 blobs / 0 B
this removes:          0 blobs / 0 B
to delete:             0 blobs / 2.856 GiB
total prune:           0 blobs / 2.856 GiB
remaining:           634 blobs / 513.640 MiB
unused size after prune: 0 B (0.00% of remaining size)

deleting unreferenced packs
[0:00] 100.00%  89 / 89 files deleted
done

All is done and and repo size is now what I would expect.

It is even worst.

When I repeat all the process but before running rustic prune I again include my big folder and create new snapshot I would expect rustic to use already existing repo data (deduplication). But nope... rustic creates new snapshot but by saving new data:

$ rustic prune
using config ./rustic.toml
[INFO] repository local:/Volumes/SSD-500-Temp/rustic-test: password is correct.
[INFO] using cache at /Users/kptsky/Library/Caches/rustic/c923f1a79e436c96db128c6a510a6e062e6d3ad03821f095bb4ca72546c480d8
[00:00:00] reading index...               ████████████████████████████████████████          2/2
[00:00:00] reading snapshots...           ████████████████████████████████████████          2/2
[00:00:00] finding used blobs...          ████████████████████████████████████████          2/2
[00:00:00] getting packs from repository...                                                                                                                                               to repack:          0 packs,          0 blobs,        0 B
this removes:                         0 blobs,        0 B
to delete:          0 packs,          0 blobs,        0 B
unindexed:          0 packs,         ?? blobs,        0 B
total prune:                          0 blobs,        0 B
remaining:                        13237 blobs,    3.3 GiB
unused size after prune:    5.4 MiB (0.16% of remaining size)

packs marked for deletion:        178,    5.7 GiB
 - complete deletion:               0,        0 B
 - keep marked:                   178,    5.7 GiB
 - recover:                         0,        0 B
[INFO] nothing to do!

repo size on disk is 6GB - it should be 3GB.... And again to clean the mess it is enough to run restic prune...

It is not what I ever experienced with restic - now it is possible that I miss some rustic logic here. But it would be very different to what one expects... or forget/prune/deduplication is broken when using as-path and ignore-devid

[kapitainsky]

I am sorry for spamming:) But I keep testing:)

I think I got now why rustic prune behaves this way - it is due to delayed delete action. Now I understand it.

But now I have strong suspicion that packs marked for deletion are not reused if needed. Like in my test - before they are physically deleted by rclone prune if another snapshot uses the same blobs they are uploaded again to the repo instead of reusing already existing and clearing their "to be deleted" marker.

[kapitainsky]

Thanks for digging into it! The point is that these top level dirs are soly virtual dirs which are set by --as-path but in fact don't (need to) exist on the filesystem. Therefore, rustic isn't able to save permissions/ownership for them.

Still to keep compatibility we restic restore these directories should be stored with some usable permissions - not just simple directory:

d---------

rustic should put them in the repo as for example with default more sane permissions:

drwxr-xr-x

then restic restore would work in much more user friendly way.

[aawsome]

I think I got now why rustic prune behaves this way - it is due to delayed delete action. Now I understand it.

Exactly. rustic implements the two-phase pruning in order to make prune work without locks. The drawback is that packs are only deleted after the delay time (by default 23h) - or better: At the next prune run after the delay time.
There is --instant-delete which makes pruning work like restics prune - but you have to ensure yourself that nothing runs in parallel.

But now I have strong suspicion that packs marked for deletion are not reused if needed. Like in my test - before they are physically deleted by rclone prune if another snapshot uses the same blobs they are uploaded again to the repo instead of reusing already existing and clearing their "to be deleted" marker.

No, they are not reused once they are marked. Reusing them would require unmarking them and this is a "change" operation which violates the principle that backup should be append-only.
They are only unmarked by prune if prune detects that they are anyway used - this is the case case if a backup was run parallel to the marking.

[aawsome]

Another issue about the unmarking: This could also lead to inconsistent repository. Think about a backup run which unmarks some packs and a parallel running prune which saw the mark and now is about to remove that pack...

[aawsome]

rustic should put them in the repo as for example with default more sane permissions:

drwxr-xr-x

Fully agree! I'll change this.

[kapitainsky]

Thank you for your explanations. I think I am happy to start using rustic for my file system snapshots.

It would be nice if these things are mentioned at least in docs - e.g. now https://rustic.cli.rs/docs/commands/forget/customize_pruning.html looks like only covering the same options as in restic. I know all takes time:) Definitely rustic is fantastic achievement. Hopefully it will only get better,

[kapitainsky]

BTW - I am more than happy to help with whatever possible. I think rustic is worth it:)

Definitely would like to add section in docu how to use filesystem snapshots - especially now when I have it working and understand more. Also to correct some obviously wrong or missing parts in other sections.

However so far I failed with basics - rustic-rs/docs#49 (comment)

I am not a guru but fairly familiar with github so a bit surprised that it is not straightforward. I will try to figure it out next year:)

Thank you for everything!

[aawsome]

Help is very much appreciated! Especially the docs are currently on of the weekest points of rustic, IMO.
But making good docs is hard work and I am personally very short on time, currently (too many small kids at home demanding time with their dad 😉 )

[aawsome]

About hard links: I think rustic-rs/rustic_core#16 is the better place for disussing solutions.

[kapitainsky]

For anybody interested this is how I handle backing up from filesystem snapshots (BTRFS, ZFS, APFS, LVM) using rustic.

This is my config:

[repository]
repository = "/Volumes/external_disk/rustic-repo"
password = "secret_password"

[backup]
# we use glob file to specify what goes to our backup
glob-file = ["./rustic.glob"]

# this will strip the snapshot mountpoint path and our backup will look as run against `/`
# This only works if source contains of a single path.
as-path = "/"

# as devid changes every time we mount new snapshot we have to ignore it
ignore-devid = true 

[[backup.sources]]
# our source is mounted snapshot
source = "/mnt/rustic-snapshotbackup/"

Now I run rustic backup from a shell script (pseudo code example below) taking care of snapshots:

1. take `/` filesystem snapshot
2. mount snapshot at `/mnt/rustic-snapshotbackup`
3. run `rustic backup`
4. unmount snapshot

What goes to my backup is determined by rustic.glob file:

# exclude all
!*

# include specific directories and their content - all individual dirs in the path have to be allowed
# to backup `/Users/kptsky/` directory we need the following rules:
/mnt/rustic-snapshotbackup/Users
/mnt/rustic-snapshotbackup/Users/kptsky
/mnt/rustic-snapshotbackup/Users/kptsky/**/*

# and to backup `/etc/config files/`
/mnt/rustic-snapshotbackup/etc
/mnt/rustic-snapshotbackup/etc/config files
/mnt/rustic-snapshotbackup/etc/config files/**/*

# exclude any log and tmp files
!*.log
!*.tmp

Please note that rustic restore does not handle hard links as of today. If we need them then we have to use restic restore. All tested and works like a charm. This is good example how rustic and restic can compliment each other in solving real life backup problems.

[simonsan]

Might be worth putting that into the FAQ or somewhere in the backup/ directory.

[kapitainsky]

Yeap. This is only here for reference for people reading this thread.

I will create more complete guide for docs including real scripts examples

[aawsome]

@kapitainsky Thanks a lot for sharing!
I think your setup is also a good example why pre-run and post-run hooks to be included in the config file are a good idea.
See #902

[kapitainsky]

Hooks would be nice thing to have indeed but are not essential.

BTW - I like the way how kopia handles it.