`borg extract` should retry failed files (up to X times) instead of failing the entire job

[shssoichiro]

When a file fails checksum verification during borg extract many times this is simply due to a network error, and can be fixed by retrying the file. When I'm restoring 25 TB of files, I don't want one random failure to force me to restart the entire job, especially since there is no deduplication mechanism (ala rsync) in borg extract.

To me, this is a critical issue, because borg is unable to perform the primary job that I expect of it: restoring my files from backup. Instead I have to do a tedious workaround of getting a list of all files that need restored, and doing them one directory at a time, when there are hundreds of directories.

[ThomasWaldmann]

Please post the traceback and the borg version(s), so we can exactly see what failed.

Also I don't quite understand how you come from "when a file fails checksum verification MANY times" to "can be fixed by retrying the file".

To me that rather sounds like: if it fails many times, it likely fails always and retrying is pointless.

Also, it is not really clear from the given facts that this is a network error, it could be also corruption in the repository due to a fs or media error.

But of course I see that failing the whole extraction isn't good, especially if the extraction volume is huge. To see how / where exactly it fails, we need the traceback.

About "no deduplication mechanism": this isn't easy for the general case, but borg2 / master branch now at least skips re-extracting files if a local file is present and the metadata matches perfectly (size and mtime, IIRC), which is already quite good for continuing an interrupted restore run.

If it is precisely one or a few files that have issues, guess you could also try to exclude them (try that first with a smaller archive, so you'll see how to get a successful exclusion).

Also find out and fix the root cause for that issue.

[shssoichiro]

I suppose I phrased that poorly. The same file is not failing each time. It seems to randomly fail after some large amount of data (about 100GB on average), but it's a different file each time. Any given file will succeed if retried.

This is on borg 1.4.1, both the client and server, on Linux. Neither machine is overclocked in any way, though I suppose that doesn't rule out some sort of memory issue causing intermittent read/write corruption (both disk arrays are on mdraid RAID5, and show no issues on mdstat or smartctl). I will post a full traceback next time I encounter it. I've been having better success doing the restore in smaller batches (one directory at a time), though it's tedious.

[shssoichiro]

Sure enough, memory issues do appear to be responsible for the checksum failures. That being said, I do still think a feature such as this would be very helpful for large restore jobs.

Edit: I'll also note that I found a good workaround using existing features, which is to use borg mount to mount the archive, then use rsync for resumable restoring.

Edit 2: That didn't work. It turns out there are files in the repo that are actually corrupted, and when rsync reaches these files, borg's mount disconnects. I suppose the next step is attempting to borg check --repair and make note of which files were corrupted so I can manually restore them. 🤞 This whole process is quite unfriendly to the user though. In my opinion it would be much less frustrating if borg extract would first retry a corrupted file (maybe 3 times by default), and if it continues failing, skip it, continue on, and report all failures at the end of the extract.