Letsencrypt Certificates Notice

[klizhentas]

If you are a Let's Encrypt user you may have recently received the following notification.

At 16:48 UTC on Tuesday Jan 25, 2022, a third party informed Let’s Encrypt / ISRG that, while examining the Boulder codebase, 
they had noticed two instances of specification non-compliance in our implementation of the “TLS Using ALPN” validation method (BRs 3.2.2.4.20, RFC 8737). As a result, we have made two changes to the way that our TLS-ALPN-01 challenge validation works.

All active certificates that were issued and validated with the TLS-ALPN-01 challenge before 00:48 UTC on 26 January 2022 when our fix was deployed are considered mis-issued. In compliance with the Let’s Encrypt CP, we have 5-days to revoke and will begin to revoke certificates at 16:00 UTC on 28 January 2022. We estimate <1% of active certificates are affected. Subscribers affected by revocations will receive e-mail notifications if their ACME account contains a valid e-mail address. If you are affected by this revocation and need help renewing your certificate please ask questions in this thread.

We will be providing more details about this incident in the next few days.

If you are using Let’s Encrypt with Teleport and have certificates issued before January 26, 2022, we’ve outlined the steps you can take below to mitigate a potential outage due to the revocation of these certificates.

Self-hosted Teleport with builtin ACME support

If you are using Teleport’s builtin ACME support, you will need to delete certificates that were automatically fetched by Teleport.

To determine whether you’re using Teleport’s ACME integration, check if it’s enabled in the Teleport Proxy Service configuration, for example:

proxy_service:
    acme:
      enabled: yes
      email: [email protected]

Let’s Encrypt certificates fetched by Teleport are stored in the “acme” subdirectory of the Teleport data directory (/var/lib/teleport by default).

To clear the ACME cache and make it refetch the certificates, delete all files from /var/lib/teleport/acme except for acme_account+key file.

# Check the ACME cache directory.
$ sudo ls -l /var/lib/teleport/acme
total 12
-rw------- 1 root root  227 Jan 19 01:33 acme_account+key
-rw------- 1 root root 5558 Jan 19 01:33 teleport.example.com

# Backup existing ACME certificate cache.
cp -r /var/lib/teleport/acme /var/lib/teleport/acme.backup

# Remove all files except account key, this will prevent you from hitting
# Let’s Encrypt rate limits.
ls /var/lib/teleport/acme | grep -v acme_account+key | xargs rm

Restart Teleport Proxy Services after clearing out the ACME certificates cache. For example, for systemd users, you can run systemctl restart teleport.

Self-hosted Teleport with external Let’s Encrypt certificates

If you are using an external tool to obtain Let's Encrypt certificates for Teleport, you may need to remove your old certificates, fetch new certificates, and restart the Teleport Proxy Service.

This can be done by first running cat /etc/teleport.yaml on your proxies to find out where Teleport loads it’s certificates from. Look for a block named https_keypairs and remove all Let’s Encrypt certificates that were issued with the TLS-ALPN-01 challenge before January 26, 2022 and rerun whatever external tool you use to obtain Let’s Encrypt certificates.

Restart Teleport Proxy Services after removing and refetching new certificates. For example, for systemd users, you can run systemctl restart teleport.

proxy:
    https_keypairs:
    - key_file: /var/lib/teleport/webproxy_key.pem
      cert_file: /var/lib/teleport/webproxy_cert.pem
    - key_file: /etc/letsencrypt/live/*.teleport.example.com/privkey.pem
      cert_file: /etc/letsencrypt/live/*.teleport.example.com/fullchain.pem

Teleport Cloud

Teleport Cloud customers are not affected by the issue.

Further reading

For more details on the Let’s Encrypt issue, see the following links.

https://community.letsencrypt.org/t/170449
https://community.letsencrypt.org/t/2022-01-25-issue-with-tls-alpn-01-validation-method/170450
https://groups.google.com/g/golang-announce/c/NCaUuUkdUIk