fix: add retry to getting GCS client config

[kevinzwang]

Also contains a tiny refactor to centralize logic for exponential backoff retry

[codecov]

Codecov Report

Attention: Patch coverage is 46.83544% with 42 lines in your changes missing coverage. Please review.

Project coverage is 78.52%. Comparing base (5618f21) to head (4166da1).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
src/daft-io/src/google_cloud.rs	0.00%	15 Missing ⚠️
src/daft-io/src/s3_like.rs	42.30%	15 Missing ⚠️
src/daft-io/src/retry.rs	68.42%	12 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3930      +/-   ##
==========================================
- Coverage   78.52%   78.52%   -0.01%     
==========================================
  Files         772      773       +1     
  Lines       98313    98366      +53     
==========================================
+ Hits        77204    77237      +33     
- Misses      21109    21129      +20     

Files with missing lines	Coverage Δ
src/daft-io/src/lib.rs	75.94% <ø> (+1.65%)	⬆️
src/daft-io/src/retry.rs	68.42% <68.42%> (ø)
src/daft-io/src/google_cloud.rs	0.00% <0.00%> (ø)
src/daft-io/src/s3_like.rs	67.06% <42.30%> (+0.10%)	⬆️

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[samster25]

So my understanding is that the 401 actually comes from when this fails, the client goes into anon mode and then does a request to GCS without credentials. That error is the 401.

Based off the user logs, the credential error can actually be some other http error (pretty sure its not 401)

(ScanWithTask-FanoutEvenSlices [Stage:2] pid=2781, ip=10.68.5.21) Google Cloud Storage Credentials not provided or found when making client. Reverting to Anonymous mode.
(ScanWithTask-FanoutEvenSlices [Stage:2] pid=2781, ip=10.68.5.21) Details
(ScanWithTask-FanoutEvenSlices [Stage:2] pid=2781, ip=10.68.5.21) Unable to load Credentials: http error: error sending request for url (https://oauth2.googleapis.com/token)

[kevinzwang]

Hm should we just retry on any HTTP error then?

[samster25]

Okay so I think we want to retry when the reqwest error type is Kind::Request or if the error code is the standard retry able stuff like 429 or 5XX

[samster25]

Something like this should work

                        HttpError(reqwest_err)
                            if reqwest_err.is_request() || reqwest_err.is_timeout() || matches!(reqwest_err.status().map(|s| s.as_u16()), Some(429) | Some(500..599))  =>