Considerations for caching tokens

[fiadliel]

As I understand, caching tokens is on the roadmap. I have some ideas about what factors might be considered here. No worries if none of this is implemented, just some thoughts on the area, as it's easiest suggested before anything is written 😄 So, below are my 2c.

Caching granularity

Caching should be at the same level as how users can customize the token information. E.g. if the only knob is "scope", then that should be part of the lookup key. If other parameters are configurable, they should be part of the key.

Validate token after fetching from cache

When the token is fetched from the cache, it can be checked for validity (expiry time, in particular), to make sure (within reason) that the caller receives a valid token. If it's not valid, the token should be refetched. Why? Because it's a nice and simple property to have for callers.

Even if there are other mechanisms than at-fetch-time for updating the token, there are scenarios like Cloud Run instances with throttled CPU where the instance potentially has not had a chance to run until the token is needed.

Block all cache waiters on a single request

When a particular cache entry is being updated, it should result in one request to the token provider. This gets rid of thundering herd problems etc. with fetching new tokens; though it does add some complexity to the caching implementation.

Update token in background before expiry

This is an area with a lot more possible questions about design properties (do you refresh all tokens in cache? a subset? how does it interact with removing entries from the cache? how to deal with failures?, etc.). It's also best-effort behavior in the face of scenarios like the aforementioned CPU-throttled Cloud Run. As an alternative to a background thread, one could have a refresh_tokens function that can be invoked on a schedule (this also potentially deals with how to update tokens in a throttled Cloud Run since it could be invoked from Scheduler).

This is a nice to have, not a requirement. The nice property here is that getting a new token can be a runtime latency hit, it'd be nice to always have a valid token ready to go.

In theory one could just exercise codepaths that renew tokens, where the latency doesn't matter. It doesn't need to be an explicit refresh_tokens... except: these calls need to be differentiated so that entries can fall out of the cache when no longer used.