Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

need automatic retry for LE server busy response. #281

Open
chrcoluk opened this issue May 11, 2023 · 5 comments
Open

need automatic retry for LE server busy response. #281

chrcoluk opened this issue May 11, 2023 · 5 comments

Comments

@chrcoluk
Copy link

Seems LE's cert server is now prone to rejecting the request with the following response.

Response Code: 503
Response: {'type': 'urn:ietf:params:acme:error:rateLimited', 'detail': 'Service busy; retry later.'}

After a bit of searching it seems its starting to become more common, and its a case of just retrying, and indeed it is quite random, so one domain could get the error, then the one immediately after is signed successfully.

Is it possible to add automatic retrying for this response?

@chrcoluk
Copy link
Author

More information here. Staff confirm here in this announcement, expected behaviour is to retry.

https://community.letsencrypt.org/t/new-service-busy-responses-beginning-during-high-load/184174

@chrcoluk chrcoluk changed the title need automatic retry for LE infrastructure problems. need automatic retry for LE server busy response. May 11, 2023
@stevemeier
Copy link

According to the LE forum post, rateLimited really means just that: too many requests in a short time period.

I solved this by adding time.sleep(5) on line 156, so there is a small pause after each domain verificiation.

@mjechow
Copy link

mjechow commented Apr 22, 2024

another way is by introducing this yourself in a calling script. I have a cron job calling an update script monthly for that.

[blahblah]

# obtaining certificate by login with account key using former csr and doing the challenge
renew_cert() {
    python3 /home/cert/acme-tiny/acme_tiny.py --account-key $CERT_DIR/letsencrypt.acct.key.pem --csr $CERT_DIR/$REQ_NAME --acme-dir /var/www/html/challenges/ > $CERT_DIR/$CERT_NAME

    if [ $? -ne 0 ]; then
        return 1
    else 
        return 0
    fi
}

renew_cert

if [ $? -ne 0 ]; then
    sleep 120
    printf "\n\nSecond try...\n\n"
    renew_cert
    if [ $? -ne 0 ]; then
        printf "\n\nError renewing certificate! Fallback to old cert.\n"
        cp $CERT_DIR/$CERT_NAME.old $CERT_NAME
        exit 1
    fi
fi

Quick & dirty and working since the beginning of Let's Encrypt flawlessly.

@stevemeier
Copy link

That workaround doesn't work in my case, as my certificate has many SANs, which need to go through in one attempt.
The number of SANs is what triggers the rate-limit, so re-running the script just triggers it again.

@mjechow
Copy link

mjechow commented Apr 22, 2024

Ok, I understand. I have 8 SANs, that works for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants