Corrupted certificate data hobbles certboto-docker

[dav3r]

🐛 Summary

An unknown issue can somehow occur where the data for a particular certificate in the S3 bucket gets corrupted. When that happens, normal certboto-docker commands will fail due to the warning emitted by rebuild-symlinks.py.

To reproduce

$ docker compose run certboto certificates
Syncing certbot configs from my-certificates-bucket
Rebuilding symlinks in /etc/letsencrypt
2022-05-31 18:26:32,008 WARNING Could not find a matching entry in the archive!

Note that the same warning will occur with any certbot command (even if the --shell flag is provided), since the error occurs in the step before actually executing certbot commands.

Expected behavior

It would be nice to do two things here:

1. Identify how the certificate data is getting corrupted in the first place.
2. If this situation is encountered, provide a mechanism for correcting it, or at least working around it until it can be corrected.

Notes

When I manually sorted this issue out earlier today, I took the following steps to correct it:

1. Start up a container:

   $ docker compose run --entrypoint /bin/sh certboto

2. Sync data from my certificates bucket:

   # AWS_PROFILE=mycertificatesbucketfullaccess aws s3 sync "s3://my-certificates-bucket" /etc/letsencrypt

3. Determine corrupted certificate:

   /opt/certbot # ./rebuild-symlinks.py --log-level=info /etc/letsencrypt 2>&1 | grep -B2 WARNING
   2022-05-31 18:32:17,008 INFO Replaced file chain.pem with symlink to ../../archive/messed-up-cert.org/chain1.pem
   2022-05-31 18:32:17,008 INFO Re-linking privkey.pem
   2022-05-31 18:32:17,009 WARNING Could not find a matching entry in the archive!

4. Delete the corrupted certificate:

   # certbot delete --cert-name=messed-up-cert.org
   Saving debug log to /var/log/letsencrypt/letsencrypt.log
   
   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   The following certificate(s) are selected for deletion:
   
     * messed-up-cert.org
      
   WARNING: Before continuing, ensure that the listed certificates are not being
   used by any installed server software (e.g. Apache, nginx, mail servers).
   Deleting a certificate that is still being used will cause the server software
   to stop working. See https://certbot.org/deleting-certs for information on
   deleting certificates safely.
   
   Are you sure you want to delete the above certificate(s)?
   - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
   (Y)es/(N)o: Y
   Deleted all files relating to certificate messed-up-cert.org.

5. Sync cleaned-up data back to my certificates bucket:

   # AWS_PROFILE=mycertificatesbucketfullaccess aws s3 sync --delete /etc/letsencrypt "s3://my-certificates-bucket"

After that, certboto-docker commands worked normally and a fresh certificate was able to be generated.

[dav3r]

Note that this situation occurred again today, except now there were 2 corrupted certificates (out of the 5 that were renewed since I deleted the corrupted certificate yesterday). I repeated the cleanup process described above and certificate creation/renewal was able to continue.

I confirmed that I have the same versions of certboto-docker and certbot as @mkreckel, who has been renewing these certs:

$ docker compose run certboto --version
0.1.0
certbot 1.22.0

We are running slightly different versions of Docker engine and compose and although I don't think that is the problem here, @mkreckel is going to update so that we can try to rule that out.

[dav3r]

This issue has struck again and now there are 17 certificates that appear to be corrupt. I'm working with our Ops folks to delete and recreate these certs, but I have yet to find any rhyme or reason why this corruption occurs.

For the record:

❱ docker compose run certboto --version
0.1.2
certbot 1.32.0