fix: completion backoff ratelimit

[stevenh]

Re-raise RateLimitError if retries are exhausted in perform_completion_with_backoff, which fixes issues with callers requesting usage information when the model is rate limited.

Also eliminate unnecessary catch of Exception and clean up the flow of the function.

Fixes: #989

How Has This Been Tested?

Added a new test and ran against full suite pending on #891

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[lance6716]

Hi @aravindkarnam @unclecode I also trapped by #989 and found there's a fixing PR. Can you take a look? Thanks.

[olawale-york]

Hi @aravindkarnam @unclecode I think the current process of retry the rateLimitError and not being properly processed is still a problem most especially when using LLMExtraction, i need to be able to get the logs relating to it so that am able to stop the scraper when the RateLimitError is caused by RESOURCE_EXHAUSTION and the returned error is obscure and more in the lines of: Error in extracted data: [{'index': 0, 'error': True, 'tags': ['error'], 'content': "'list' object has no attribute 'usage'"}]. Is it possible to address this

[stevenh]

Closing as never got any traction, so we've moved away from crawl4ai.

If someone wants to pick up the branch and reuse, feel free.