Deleting ~/.cache breaks all future backups with no apparant fix

[john01dav]

Some time ago I deleted my ~/.cache directory since my drive was full and a cache, by definition, should just slow things down if it's gone as it's regenerated. If it is the only place that some vital data is stored or this is otherwise not the case, then it isn't a cache and thus does not belong in ~/.cache. Despite this, after deleting it, borg won't backup anymore. Instead it says the following. I tried mkdir -p ~/.cache/borg, which had no effect. Instead, Borg should notice that the cache has been cleared and regenerate whatever it needs.

Creating Backup <censored>
time borg create --progress --compression zstd,10 <censored>:<censored>::<censored> <censored>
Local Exceptionunks cache. Processing archive 2024-06-05-12:00-AM                                                                                                                                                   Traceback (most recent call last):                                     
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 773, in write_archive_index                                                      
    with DetachedIntegrityCheckedFile(path=fn_tmp, write=True,                                                                               
  File "/usr/lib/python3/dist-packages/borg/crypto/file_integrity.py", line 211, in __init__                                                                                                                                                                                               
    super().__init__(path, write, filename, override_fd)                                                                                     
  File "/usr/lib/python3/dist-packages/borg/crypto/file_integrity.py", line 129, in __init__                                                 
    self.file_fd = override_fd or open(path, mode)                                                                                           
FileNotFoundError: [Errno 2] No such file or directory: '/home/<censored>/.cache/borg/<censored>/chunks.archive.d/<censored>.tmp'

# more of similiar erors

During handling of the above exception, another exception occurred:                                       

Traceback (most recent call last):                   
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5089, in main                              
    exit_code = archiver.run(args)                   
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5020, in run                               
    return set_ec(func(args))                        
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 183, in wrapper                            
    return method(self, args, repository=repository, **kwargs)                                            
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 649, in do_create                                                                                                                                    
    with Cache(repository, key, manifest, progress=args.progress,                                         
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 383, in __new__                               
    return local()                                   
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 374, in local                                 
    return LocalCache(repository=repository, key=key, manifest=manifest, path=path, sync=sync,                                                                                                                      
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 496, in __init__                              
    self.close()                                     
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 545, in close                                 
    self.cache_config.close()                        
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 321, in close                                 
    self.lock.release()                              
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 417, in release                             
    self._roster.modify(EXCLUSIVE, REMOVE)                                                                
  File "/usr/lib/python3/dist-packages/borg/locking.py", line 316, in modify                              
    elements.remove(self.id)                         
KeyError: ('<censored>', 2013457, 0)

[infectormp]

Which version of borg are you using?

[john01dav]

This: ``` $ borg --version borg 1.2.0 ``` This is the latest that my Ubuntu version packages.

[ThomasWaldmann]

Before we start debugging a potentially long-fixed bug, you could first try to reproduce this issue with something more recent (like 1.4.x or 1.2.x) from one of these sources:

• https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/borgbackup (from borg ubuntu/debian maintainer)
• https://github.com/borgbackup/borg/releases (fat binaries from here, you could start with 1.2.4 from here)

Please note that when crossing the 1.2.4 / 1.2.5 version boundary, you MUST read and follow the CVE-related instructions at top of the changelog.

When you upgrade to Ubuntu 24.04 at some time, you would also get a more recent borg version (1.2.8).

[ThomasWaldmann]

@github Why is the code block formatting there not rendering correctly?: #8937 (comment)

[john01dav]

How risky is the CVE update procedure when my current borg instance is broken like this?

Also, does it store any data outside of the archive and the ~/.cache directory? If so, deleting that may make sense. It would seem to think that the ~/.cache directory exists, so there should be some easy way to convince it otherwise (like if I was backing up after reinstalling my OS).

[ThomasWaldmann]

At first, you would just use the more recent release to reproduce the bug (or see that it is fixed).

If it is not fixed, you could just delete the cache: borg delete --cache-only REPO

The cache can be rebuilt, it just will need some time.

You can do some harmless operations like borg list or so, but do NOT run borg check --repair with a 1.2.5+ release before doing the upgrade steps from the changelog.

[ThomasWaldmann]

I did a quick try with borg 1.4.1 and removing the cache dir was no problem, it was automatically rebuilt:

(borg-env) tw@MacBook-Pro test % borg -V
borg 1.4.1

(borg-env) tw@MacBook-Pro test % borg init -e none repo

(borg-env) tw@MacBook-Pro test % borg info repo
Repository ID: 71ee8013698c6c6e4af931a454d95fc9f3286250194b4176d711ca6df221f9ec
Location: /Users/tw/w/borg/test/repo
Encrypted: No
Cache: /Users/tw/.cache/borg/71ee8013698c6c6e4af931a454d95fc9f3286250194b4176d711ca6df221f9ec
Security dir: /Users/tw/.config/borg/security/71ee8013698c6c6e4af931a454d95fc9f3286250194b4176d711ca6df221f9ec
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                    0 B                  0 B                  0 B

                       Unique chunks         Total chunks
Chunk index:                       0                    0

(borg-env) tw@MacBook-Pro test % borg create repo::docs ../docs
(borg-env) tw@MacBook-Pro test % borg list repo
docs                                 Sun, 2025-07-06 17:31:20 [5c673383558a5fa523bd9feb6b3c2de62619958589fb547c7ef1e8eaec1e2df2]

(borg-env) tw@MacBook-Pro test % ls -l /Users/tw/.cache/borg/71ee8013698c6c6e4af931a454d95fc9f3286250194b4176d711ca6df221f9ec
total 208
-rw-------  1 tw  staff     68 Jul  6 17:30 README
-rw-------  1 tw  staff  45382 Jul  6 17:31 chunks
drwx------  2 tw  staff     64 Jul  6 17:30 chunks.archive.d
-rw-------  1 tw  staff    586 Jul  6 17:31 config
-rw-------  1 tw  staff  41207 Jul  6 17:31 files
-rw-------  1 tw  staff      2 Jul  6 17:31 pre12-meta

(borg-env) tw@MacBook-Pro test % rm -rf /Users/tw/.cache/borg/71ee8013698c6c6e4af931a454d95fc9f3286250194b4176d711ca6df221f9ec

(borg-env) tw@MacBook-Pro test % borg list repo                                                                               
docs                                 Sun, 2025-07-06 17:31:20 [5c673383558a5fa523bd9feb6b3c2de62619958589fb547c7ef1e8eaec1e2df2]

(borg-env) tw@MacBook-Pro test % borg create repo::docs2 ../docs                                                              

(borg-env) tw@MacBook-Pro test % ls -l /Users/tw/.cache/borg/71ee8013698c6c6e4af931a454d95fc9f3286250194b4176d711ca6df221f9ec 
total 208
-rw-------  1 tw  staff     68 Jul  6 17:32 README
-rw-------  1 tw  staff  45382 Jul  6 17:32 chunks
drwx------  4 tw  staff    128 Jul  6 17:32 chunks.archive.d
-rw-------  1 tw  staff    586 Jul  6 17:32 config
-rw-------  1 tw  staff  41298 Jul  6 17:32 files
-rw-------  1 tw  staff      2 Jul  6 17:32 pre12-meta

[john01dav]

Where can I get further support with this, as my backups are still broken? I tried with Borg 1.2.4. When I run borg check on it, it passes. I now think that there's still a bug, but it is not reproducible by simply deleting ~/.cache as I initially thought. It seems like only borg create is broken, as borg list and borg check both work.

[ThomasWaldmann]

I tried with borg 1.2.4 and it also worked for me:

(borg-env) tw@MacBook-Pro test % borg -V
borg 1.2.4

(borg-env) tw@MacBook-Pro test % borg init -e none repo

(borg-env) tw@MacBook-Pro test % borg info repo
Repository ID: 27cdf2c7e67ded77d0ebe4b14266ab82f60babfbabce86819e5d85013ba05b61
Location: /Users/tw/w/borg/test/repo
Encrypted: No
Cache: /Users/tw/.cache/borg/27cdf2c7e67ded77d0ebe4b14266ab82f60babfbabce86819e5d85013ba05b61
Security dir: /Users/tw/.config/borg/security/27cdf2c7e67ded77d0ebe4b14266ab82f60babfbabce86819e5d85013ba05b61
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
All archives:                    0 B                  0 B                  0 B

                       Unique chunks         Total chunks
Chunk index:                       0                    0

(borg-env) tw@MacBook-Pro test % borg create repo::docs ../docs

(borg-env) tw@MacBook-Pro test % borg list repo
docs                                 Tue, 2025-07-08 11:30:21 [b7cc114605617ea3fd4830f1d13fc27090e4765b66411215183a325abb8119c7]

(borg-env) tw@MacBook-Pro test % ls -l /Users/tw/.cache/borg/27cdf2c7e67ded77d0ebe4b14266ab82f60babfbabce86819e5d85013ba05b61
total 200
-rw-------  1 tw  staff     68 Jul  8 11:30 README
-rw-------  1 tw  staff  45382 Jul  8 11:30 chunks
drwx------  2 tw  staff     64 Jul  8 11:30 chunks.archive.d
-rw-------  1 tw  staff    586 Jul  8 11:30 config
-rw-------  1 tw  staff  40935 Jul  8 11:30 files
-rw-------  1 tw  staff      2 Jul  8 11:30 pre12-meta

(borg-env) tw@MacBook-Pro test % rm -rf /Users/tw/.cache/borg/27cdf2c7e67ded77d0ebe4b14266ab82f60babfbabce86819e5d85013ba05b61

(borg-env) tw@MacBook-Pro test % borg list repo
docs                                 Tue, 2025-07-08 11:30:21 [b7cc114605617ea3fd4830f1d13fc27090e4765b66411215183a325abb8119c7]

(borg-env) tw@MacBook-Pro test % ls -l /Users/tw/.cache/borg/27cdf2c7e67ded77d0ebe4b14266ab82f60babfbabce86819e5d85013ba05b61 
ls: /Users/tw/.cache/borg/27cdf2c7e67ded77d0ebe4b14266ab82f60babfbabce86819e5d85013ba05b61: No such file or directory

(borg-env) tw@MacBook-Pro test % borg create repo::docs2 ../docs                                                              

(borg-env) tw@MacBook-Pro test % ls -l /Users/tw/.cache/borg/27cdf2c7e67ded77d0ebe4b14266ab82f60babfbabce86819e5d85013ba05b61
total 208
-rw-------  1 tw  staff     68 Jul  8 11:31 README
-rw-------  1 tw  staff  45382 Jul  8 11:31 chunks
drwx------  4 tw  staff    128 Jul  8 11:31 chunks.archive.d
-rw-------  1 tw  staff    586 Jul  8 11:31 config
-rw-------  1 tw  staff  41026 Jul  8 11:31 files
-rw-------  1 tw  staff      2 Jul  8 11:31 pre12-meta

[john01dav]

That makes me think that something is corrupted in my backups, but I have no idea how to go about fixing that. These backups are offsite and multiple terabytes in size — it would take months to re-create the archive. I had to physically travel to initially set it up.

[ThomasWaldmann]

What you can try (there seems to be a locking issue, not sure if it is relevant):

• make sure there is no borg running and working with that REPO
• run borg break-lock REPO
• retry borg create

[ThomasWaldmann]

Also, check the permissions of the cache directory, especially if you are switching between root and an unprivileged account.