Add support for dynamic postgres connection config

[titanous]

This adds support for providing a dynamic configuration for postgres pools that can provide a unique configuration for each new connection.

The motivating use case for this is AWS RDS IAM authentication, which uses a token that has a 15 minute expiry as the password. I have implemented this algorithm with the new hook provided in this PR and am currently testing it out.

This PR also depends on a small change in tokio_postgres to allow passing errors through cleanly. Once the approach in this PR has been reviewed, I'll submit that PR for review upstream.

I'm pretty new to Rust, so please let me know if I'm approaching this wrong.

[bikeshedder]

Interesting idea. It flips the responsibilities around: Rather than changing the configuration a callback can be configured for providing the manager configuration. I do like the idea of that "configuration provider" but at the same time I wonder if it's really the right approach.

e.g. let's say the database is being migrated, the connection URL has changed and connections using the old URL should be dropped asap. With that approach only new connections get to know about that configuration change and existing connections will stay pooled and reused over and over again. Unless the recycle method also calls get_config and checks for equality with the last known configuration dropping objects with a different configuration the existing connections will stay connected.

I have to play with this a bit. I've toyed with replacing the entire manager but that would be a rather heavy change as it would mean that manager implementations tracking the created objects (such as the postgres one) would either need to reattach connections from other manager instances or upon changing the manager all existing connections need to be dropped when returned to the pool. I'm undecided if I'd rather make the manager non-replaceable and leave it to the manager implementation to support config changes.

See also

• Change pool configuration at runtime #166

[titanous]

Unless the recycle method also calls get_config and checks for equality with the last known configuration dropping objects with a different configuration the existing connections will stay connected.

Hmm, for that use case the first thing that comes to mind is to put an optional "generation number" (u64) in the config and then have a recycle method that recycles everything before a specific generation. That way the config infrastructure can manage things however it wants and the Pool doesn't need to be aware of the implementation details.

[fiadliel]

I was looking for this kind of change (with Google Cloud SQL in my case). I personally don't think there's a single policy that works for whether a config update should cause existing connections to be removed; this should be signalled separately, either here or by another mechanism.

I was considering the possibility of just making the DB config be a Mutex, and clone it before connecting. The advantage of the approach in this PR (over the mutex approach) seems to be that there's never a case where you might make a connection with a stale configuration, as you always "read through" what the current value should be.

[bikeshedder]

Great insights. There doesn't seam to be one "right way" that should be hard coded.

I think the best approach would be to let the object reference the configuration and/or manager that was used to create it. The pre_recycle hook could then compare the current configuration and the configuration that was used to create the object and discard the objects if needed. Deadpool currently doesn't come with any premade hooks. It probably makes sense to write some hooks to cover the most common use cases. e.g. max_age, max_use_count, drop_on_manager_change, etc.

I'm still intrigued by the idea of replacing the manager rather than just replacing the config. This would mean that managers like the one from deadpool-postgres need to support some kind of handover mechanism as it keeps track of the objects it has created in order to access the statement caches of the objects. Alternatively the logic of tracking objects could be moved to the deadpool core. Even though it's currently only needed by deadpool-postgres it could allow features like accessing object metrics of handed out objects for debugging and introspection purposes.

[bikeshedder]

Related to:

• Keep track of all objects #336

If this lands the Manager could just be swapped making configuration changes for all deadpool-* crates available as part of the core crate.

[cormacrelf]

Couple of thoughts on this

• From the perspective of using the code elsewhere, I do think a pull-based config source is a good approach. It is very easy to use + exactly maps conceptually to dynamic authentication use cases, which is what a lot of people seem to want. It also means stuff like token expiry and re-fetch can be implemented with very standard methods like "just use any in-memory cache for the credentials from AWS and re-fetch if key expired" which any backend programmer can do in their sleep.
• Wiping out all existing connections in order to e.g. change the connection endpoint to some other database is something you can do with API purely devoted to that, in concert with pull-based config. These things are composable and will work together. In HikariCP, they have 3 methods suspendPool, softEvictConnections and resumePool that can be used for failover. In many cases including RDS, failover happens elsewhere (e.g. DNS now returns a different IP address) and the only things you need to do are detect when it occurs, and drain existing connections. There's no material difference between DNS changing and configuring a new endpoint, so this one set of API would work for all of it -- pool.suspend(), then pool.soft_evict_connections() and configure your dynamic config to return a new endpoint, then pool.resume(). (Because this is Rust, you could use a PoolSuspensionGuard to auto-resume.)

Motivating use case

• Motivating use case is that postgres has PAM module support, so with something like https://github.com/outerbounds/pam-jwt ¹, this could be used to turn short-lived Kubernetes service account JWTs into database users implicitly by service account name or a particular claim field. Doesn't exist yet but would be great, with the quality k8s operators we are really not far off full service RDS replacement using only open source stuff.
• This use case is fully satisfied by just generating the config on demand whenever it's needed. k8s mounts such things as file volumes and then you basically read the JWT from a file every time you want a new connection. But also there's no need to kill existing connections, they don't expire, the role isn't deleted, you're all good to keep using it.

Footnotes

1. Some mods required, in particular for production use it should be able to cache (and re-fetch on expiry) the OIDC provider details from the .well-known/openid-configuration auto-config endpoint, presumably saving them in a file somewhere, instead of fetching them every time it wants the keys. ↩