Do not retry refreshing Credentials created from "raw" token

[slimsyMeteor]

Hello,

I have the following scenario:

1. Generate a raw token (downscoped token) for a bucket/blob using a third-party service.
2. Pass this token to gcsfs.
3. gcsfs creates new credentials from this token.

Everything works fine until the token expires. At that point, it begins trying to refresh the token:

• https://github.com/fsspec/gcsfs/blob/main/gcsfs/retry.py#L76
• https://github.com/fsspec/gcsfs/blob/main/gcsfs/credentials.py#L205

Since the token is acquired from an external service, this refresh action keeps retrying and failing.

The gcsfs version is >=2024.12.0.

[martindurant]

Eventually you run out of retries and get an exception, right?
What do you think the code should do in your case?

[slimsyMeteor]

Eventually you run out of retries and get an exception, right? What do you think the code should do in your case?

It would be nice to immediately throw a 401 error without attempting to refresh credentials if a raw token is provided.
The six default retry attempts take at least 63 seconds to complete.

Anyway, if the token is raw, there is no point in attempting to refresh the credentials created from it.

[martindurant]

Can you say what distinguishes a raw token from other types?

[martindurant]

Ping - @slimsyMeteor, do you have a fix in mind, how we can decide not to renew in the case of a raw token

[alessio-locatelli]

By the way, refreshing credentials also unnecessary when STORAGE_EMULATOR_HOST is set. Testing without Google credentials spends up to 28 seconds just on useless retries.

google.auth.exceptions.TransportError: Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Compute Engine Metadata server unavailable due to HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NameResolutionError("<urllib3.connection.HTTPConnection object at 0x7f0f785b3360>: Failed to resolve 'metadata.google.internal' ([Errno -2] Name or service not known)"))

ValueError: Invalid gcloud credentials

Perhaps the original problem and this one can be fixed in the same file in the same PR.

[martindurant]

Yes you are right, I was aware of unnecessary runtime in the CI. I will look into it this week, but I can't promise it would help; in the interim, could you please list the keys in a "raw" token, so that I know what we're talking about?