Why new repo requires pruning to get 0% unused? and why 0% soemtimes is not possible

[kapitainsky]

I have to setup multiple new repos recently and I have noticed something strange (or not?).

Freshly created repo is not "perfect"... It requires quite substantial (20% of all size) pruning to get truly 0% unused.

For example:

$ rustic prune --instant-delete --max-unused 0% --max-repack=unlimited
[INFO] using config ./rustic.toml
[INFO] repository rclone:ULOZ01-ENC:rustic_kptsky_backup: password is correct.
[INFO] using cache at /Users/kptsky/Library/Caches/rustic/112f2d92ff1757122ac7616f7a513f163212e568634f5e7dfa960ed98ec4a1ec
[00:00:01] reading index...               ████████████████████████████████████████         18/18
[00:00:01] reading snapshots...           ████████████████████████████████████████          1/1
[00:00:13] finding used blobs...          ████████████████████████████████████████          1/1
[00:05:08] getting packs from repository...                 
to repack:        200 packs,     114536 blobs,    3.2 GiB
this removes:                      7245 blobs,  249.0 MiB
to delete:          0 packs,          0 blobs,        0 B
unindexed:          0 packs,         ?? blobs,        0 B
total prune:                       7245 blobs,  249.0 MiB
remaining:                       284383 blobs,   14.4 GiB
unused size after prune:        0 B (0.00% of remaining size)

packs marked for deletion:          0,        0 B
 - complete deletion:               0,        0 B
 - keep marked:                     0,        0 B
 - recover:                         0,        0 B
[00:00:04] repacking // rebuilding index... ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   3.10 MiB/2.94 GiB   89.06 KiB/s  (ETA 15m)

It happens 100% of times. Is it normal?

And another weirdo. I tried to prune another repo to 0% unused but got only to this stage:

to repack:         36 packs,      17485 blobs,  114.5 MiB
this removes:                         0 blobs,        0 B
to delete:          0 packs,          0 blobs,        0 B
unindexed:          0 packs,         ?? blobs,        0 B
total prune:                          0 blobs,        0 B
remaining:                       394714 blobs,   58.4 GiB
unused size after prune:        0 B (0.00% of remaining size)

I could run another rustic prune --instant-delete --max-unused 0% --max-repack=unlimited and it was always the same. Prunning 114MB every time. This is not right IMO...

[aawsome]

Hi @kapitainsky
What do you mean by "freshly created repo"? Can you please give the steps you are performing to get that repository in that state?

About your second example: Note that there are no unused blobs in this example. It is just that prune repacks pack files, but without actually removing anything. prune does not only repack in order to remove data, but also in order to resize pack files.
And this can happen quite surprisingly as the "optimal" pack size is calculated by the repository size, so this is a dynamic process. E.g. directly after a backup, prune may want to repack just-generated pack files. This is because during the backup, rustic can not know how many data will be coming while backing up the first data. So the first generated pack files may be too small w.r.t the repository size after the backup finishes.

There are two fields where this prune and backup behavior can be customized:

• Using rustic config and the various *packisize* options. This is a one-time configuration. Note this also allows you to unset this dynamic pack sizing.
• Using --no-resize in prune does no repacking which only resizes packs.

But I think, we may want to get more information about pack sizes, targeted (i.e. "optimal") pack size and unused blobs in the repoinfo command. This would then allow users to transparently see why prune is doing a repack.
And, yes, there might be some bugs present about this size-detection and repacking, so I am curious and appreciate getting more information about your case...

[kapitainsky]

What do you mean by "freshly created repo"? Can you please give the steps you are performing to get that repository in that state?

1. rustic init --set-compression 14
2. rustic backup (one snapshot created)
3. rustic prune as in original post.

The "issue" happens with all default settings but encryption set to 14. Have not tested with default encryption - will do it too next time.

[kapitainsky]

About your second example: Note that there are no unused blobs in this example. It is just that prune repacks pack files, but without actually removing anything.

It is perfectly fine and I do understand. Problem is that it can keep doing it until the end of the world and not finish. The issue is that without doing anything to the repo but only pruning it does the same amount of work every time. It was odd ball case as usually things end with "Nothing to do" run. Then subsequent prune runs also have "Nothing to do". IMO it shows that pruning algo has some rough edges still.

Good thing is that all repos worked perfectly fine and no data was corrupted. And yes for production I can set --max-unused to something hogher than 0% and not even notice it but still would like to have confidence in the algorithm.

[kapitainsky]

I will do few more tests. Now in case I do not want any fancy packs adaptive size (the same behaviour as restic) is it enough to set:

$ rustic config --set-datapack-size 16MiB  --set-datapack-growfactor 0 --set-datapack-size-limit 16MiB --set-min-packsize-tolerate-percent 30 --set-max-packsize-tolerate-percent 0 --set-treepack-size 4MiB --set-treepack-size-limit 4MiB --set-treepack-growfactor 0

The reason why I do it is that I can plan my repo size in advance and choose most appropriate packs sizes taking into account storage features etc.. It is stored often on slow and high latency destinations I do not need unnecessary pruning operations.

[kapitainsky]

And here example of repository where prune never ends:

$ rustic prune --instant-delete --max-unused 0% --max-repack=unlimited
[INFO] using config ./rustic.toml
[INFO] repository rclone:ULOZ01-ENC:rustic_kptsky_backup: password is correct.
[INFO] using cache at /Users/kptsky/Library/Caches/rustic/112f2d92ff1757122ac7616f7a513f163212e568634f5e7dfa960ed98ec4a1ec
[00:00:02] reading index...               ████████████████████████████████████████         16/16
[00:00:01] reading snapshots...           ████████████████████████████████████████          3/3
[00:00:21] finding used blobs...          ████████████████████████████████████████          3/3
[00:05:09] getting packs from repository...                                                                                                                                    
to repack:         14 packs,       6728 blobs,   42.9 MiB
this removes:                         0 blobs,        0 B
to delete:          0 packs,          0 blobs,        0 B
unindexed:          0 packs,         ?? blobs,        0 B
total prune:                          0 blobs,        0 B
remaining:                       399030 blobs,   59.4 GiB
unused size after prune:        0 B (0.00% of remaining size)

packs marked for deletion:          0,        0 B
 - complete deletion:               0,        0 B
 - keep marked:                     0,        0 B
 - recover:                         0,        0 B
[01:07:55] repacking // rebuilding index... ████████████████████████████████████████  42.93 MiB/42.93 MiB  10.79 KiB/s  (ETA -)
[00:00:01] removing old index files...    ████████████████████████████████████████          8/8
[00:00:02] removing old data packs...     ████████████████████████████████████████         14/14

does not matter how many times I run rustic prune --instant-delete --max-unused 0% --max-repack=unlimited it is always repacking 14 packs, 6728 blobs, 42.9 Mi and removing 8 old indexes and 14 packs....

It is clearly problem with algo optimising tree packs???

Running prune with --no-resize flag indeed finishes quick with nothing to do. But obviously it would be preferable if rustic can optimize tree packs resizing them as needed without entering never ending loop.

[aawsome]

I agree that this is a strange behavior of the algorithm which may need be be a bit more relaxed about resizing..
Can you run it with -log-level debug (a rustic prune -n is sufficient) - it should print out something like this:

[DEBUG] (1) rustic_rs::commands::prune: DebugStats(
    {
        DebugStatsKey {
            todo: Keep,
            blob_type: Data,
            status: EnumSet(TooSmall | HasUsedBlobs),
        }: DebugDetailedStats {
            packs: 4,
            unused_blobs: 0,
            unused_size: 0,
            used_blobs: 82,
            used_size: 127945,
        },

(and so on...)

That information is the input for the prune algo...

[kapitainsky]

Thank you. I have been testing it more recently trying to uncover what is happening. Based on my tests I am convinced that there is an issue with how tree packs are handled by rustic - data packs seem to be OK. Good news is that there is no issue with data integrity - all is related to packs size limits and pruning running circles without achieving anything. I will provide all details I gathered soon + debug.