Retry on transient error in download workflow

[Wauplin]

Originally from @Wauplin on Slack (internal link):

I'm guessing huggingface_hub doesn't have retry for 429. Should be a good thing to add?

Well, I have no idea why we haven't spotted that before 🫠
http_get relies on _request_wrapper which itself has been refactored in Oct. 2023 (!!) by ... hmm... myself. I wrote in comments Perform request and return if status_code is not in the retry list. . But the implementation is a simple response = get_session().request(method=method, url=url, **params) without any retry mechanism 🙈 See #1766.

So basically this PR adds a retry mechanism when downloading a file (GET /resolve) or when fetching metadata for a file (HEAD /resolve) if rate limit is reached. Retry after 1s, 2s, 4s, 8s and 8s.

---

Comments:

• I'm not 100% sure about that PR. Opening it for discussion but I feel that I'm missing something - I was pretty sure until now that we had already a retry mechanism in place for 429...
• After some checks, I remembered that HfFileSystem uses http_backoff with retries on 5xx errors but not 429 🤔

@julien-c @LysandreJik @hanouticelina any obvious drawback that I missed from retrying on 429 for now?

EDIT: actually HTTP 429 are rare in downloads, that's why we never had one. Problem arose today because of Xet bridge that rate-limited @Vaibhavs10

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hanouticelina]

I was also pretty sure we had the retry-mechanism on 429 errors but I guess i was dreaming too 😄 I don't see any drawback for now, I think it's safe to add a retry mechanism on only 429 errors 👍

[Wauplin]

After some thoughts, let's do it :) Retrying on 429 only is a safe bet. Worth case, we adapt later. @hanouticelina mind reviewing? (and merging if I'm off 🤗)

[hanouticelina]

looks good, thank you!
let's merge it!