Borg2 incremental backup is really slow on big folders

[Oygron]

Hi,

I’m testing Borg2 to benchmark how it performs compared to Borg1.4 on my Nexcloud server, containing 1TB of data.

In Borg 1.4.1, it takes about 2 minutes to update a backup (with very limited changes in the files).

In Borg 2.0.0b18 and 2.0.0b17 (did not have the time to test on the previous ones), it takes about 1h30 to update a backup (with very limited changes in the files).

Here are some infos that might be relevant for this:

• I’m backuping from a virtual machine (debian bookworm), to an external USB drive, plugged on the host machine (proxmox, debian bookworm), all on the same physical machine.
• The virtual machine has an ext4 file system, and the external drive plugged on the host machine has a btrfs file system.
• The initial data size is 1TB, and the final archive weights 800GB, same size for all three tests. The update does not change the size of the final archive (as expected).
• There is a total of 900k files in the initial folder that is backuped
• Backuped data is mostly pictures (between 1 and 10MB by files), lots of really small files (previews, code, config…), and a couple of big files (~10GB each).

Here are the commands and exact durations for each version of Borg:

• Borg 1.4.1 (borg-linux-glibc231)
  • borg init --encryption repokey-blake2 ssh://_user_@_host-ip_:22/media/backup/borg
  • time borg create ssh://_user_@_host-ip_:22/media/backup/borg::init /var/www
    • real 173m19.091s
    • user 109m37.745s
    • sys 29m43.105s
    • final size: 803GB
  • time borg create ssh://_user_@_host-ip_:22/media/backup/borg::update /var/www
    • real 2m42.333s
    • user 2m12.602s
    • sys 0m21.980s
• Borg 2.0.0b18 (borg-linux-glibc236)
  • borg -r ssh://_user_@_host-ip_:22//media/backup/borg repo-create --encryption=repokey-aes-ocb
  • time borg -r ssh://_user_@_host-ip_:22//media/backup/borg create init /var/www
    • real 159m14.203s
    • user 124m6.674s
    • sys 31m30.736s
    • final size: 802GB
  • time borg -r ssh://_user_@_host-ip_:22//media/backup/borg create update /var/www
    • real 87m35.450s
    • user 60m13.850s
    • sys 6m40.052s
  • I tried another update, approximately the same durations.
• Borg 2.0.0b17
  • borg -r ssh://_user_@_host-ip_:22//media/backup/borg repo-create --encryption=repokey-aes-ocb
  • time borg -r ssh://_user_@_host-ip_:22//media/backup/borg create init /var/www
    • real 143m13.891s
    • user 110m46.665s
    • sys 30m48.363s
    • final size: 804GB
  • time borg -r ssh://_user_@_host-ip_:22//media/backup/borg create update /var/www
    • real 80m36.358s
    • user 50m45.848s
    • sys 7m20.617s

The VM has 2CPU cores and 10GB of RAM

The Host has 8CPU cores (i7-7700) and 32GB of RAM

Feel free to ask for any precision that might be interesting.

For more runs (including creating relevant logs), that will have to wait until the 21st of July as I’ll be on holidays next week

Thanks a lot for your incredibly useful work!

[Jorge-Fabregas]

Hi,

The problem is that you're using a different name for the archive in each run. This is one of the major new features in Borg 2: the concept of archive series.

Archives do not need to have unique names anymore, they may have the same name and we strongly recommend using this when appropriate as it simplifies and optimizes working with archives (caching, pruning, repo-list output).

An archives series is now simply all the archives in a repository that have the same name. OTOH, the unique identifier for a single archive is now its ID (the hash, it can be shortened as long as it is unique).

To fix this, try using the same archive name across multiple runs. For example, run:

borg create website for the initial seeding, and then
borg create website again for subsequent backups.

This will reuse the existing cache for this series of archives called "website." If you change the name, Borg won’t reuse the existing file cache, which forces it to read, chunk, and hash all your files again. Of course, it won't transmit known chunks, but it will still be slower due to the extra processing.

Check the documentation about it for more details.

HTH,
Jorge

[ThomasWaldmann]

@Jorge-Fabregas Very good analysis of the issue!

@Oygron Try what he said. And thanks for testing borg2 betas!

Your (next) benchmark results will represent what's possible with borg2 beta18, which uses 1 separate file per chunk in the repository. This is a very simple approach, but it has very different performance properties to what borg 1.x does (storing multiple chunks into 1 file in the repository, the files are assembled repo-side). Thus, in borg2 b18, some stuff is slower (e.g. listing all objects in the repo), some stuff is faster (e.g. compact).

But, borg2 might switch to using client-side assembled packs (storing multiple chunks in 1 pack file), this is something I am currently working on. And that would change the performance again...

[jdchristensen]

The problem is that you're using a different name for the archive in each run. This is one of the major new features in Borg 2: the concept of archive series.

Is the files cache stored separately for each archive series? I couldn't find this mentioned in the documentation, and I don't see why it would be useful to do it this way. If the file was seen previously in any archive series, then there is no need to re-chunk it when making a new backup.

[Jorge-Fabregas]

Is the files cache stored separately for each archive series? I couldn't find this mentioned in the documentation, and I don't see why it would be useful to do it this way. If the file was seen previously in any archive series, then there is no need to re-chunk it when making a new backup.

Yes, in Borg 2, the cached files are stored separately for each archive series. One of the benefits of this design is that you can delete your cache directory, and on the next run, Borg will quickly rebuild the cache for a known archive series.

As for why it needs to be from the same series — even if it might have had the data from another series — I’m not entirely sure. I assume it's part of the new design.

See Thomas’s response to my recent question on the mailing list:

https://mail.python.org/archives/list/borgbackup@python.org/thread/6MY5X4PWVHNIVV4Z5OANJ24KDO4ZBXUH/

[jdchristensen]

@Jorge-Fabregas Thanks for pointing out that link, it clarifies things.

@ThomasWaldmann The borg2 borg create documentation page should be updated to mention this. It discusses the files cache, but not this new behaviour. Also, it mentions placeholders like the date/time that can be used in the archive name, but based on this behaviour those should definitely be avoided and it should be emphasized that it's important to not vary the archive name within a series.

[Oygron]

Hi all,

Thanks for your prompt answers!

I confirm that keeping the same archive name solves the problem. It now takes 5min30s to make an incremental backup on my 1TB of data. It is slower than Borg1.4, but honestly it is still super fast for the amount of data.

But as @jdchristensen said, I think that the doc should put a big warning for admins that already know Borg1.x that will skim the doc just to find what changes they have to make to the command for it to keep working, and not read it attentively (the error I did…)