[Feature Request]: Honor Retry-After header from APIs

[jdmansour]

What needs to be done?

When your quota is exhausted, some APIs use the Retry-After header to signal when it is safe to send another request. Currently, crawl4ai applies basic exponential backoff for requests, but doesn't take this header into account.

What problem does this solve?

This lets Crawl4AI better deal with rate limiting

Target users/beneficiaries

No response

Current alternatives/workarounds

No response

Proposed approach

I've made a very small patch to implement this. If you'd like I can make a pull request.

This works fine for me (using an OpenAI compatible API) but needs testing and could be made more robust (e.g. catching errors if the header is invalid).

In utils.py, perform_completion_with_backoff around line 1794, change it to:

        # Check if we have exhausted our max attempts
        if attempt < max_attempts - 1:
            # Honor Retry-After header if present
            if retry_after := e.litellm_response_headers.get("Retry-After"):
                delay = int(retry_after)
            else:
                # Calculate the delay and wait
                delay = base_delay * (2**attempt)  # Exponential backoff formula
            print(f"Waiting for {delay} seconds before retrying...")
            time.sleep(delay)

[aravindkarnam]

wonderful suggestion @jdmansour. We'll look into this!! Thanks for sharing.