Haproxy keeps sending queries to non-ready nodes on shutdown

[wonko]

Report

Haproxy has a checkscript which is run every 10 seconds, and basically checks for non-maintenance mode, and a state of "4" (joined/synced). There is a situation where, with bigger databases, the following happens:

• node is initiating shutdown
• wsrep is leaving the cluster
• haproxy is still sending queries to the cluster
• response is "wsrep has not yet prepared the node for application use"

More about the problem

Query response is "WSRep has not yet prepared the node for application use" when kubernetes reschedules a DB node in the cluster (due to node cycling).

Steps to reproduce

Create a big enough database so the shutdown might take a couple of seconds.

Set up a loop which will do a query on a new connection. When this is running, force pxc pods to terminate cleanly.

You will hit this response.

Versions

1. Kubernetes 1.31
2. Operator 1.17.0
3. Database 1.17.0

Anything else?

The solution seems reasonably simple: the haproxy check script is run every 10 seconds, and has a "fall" of "2", so it means that it will take 20 seconds at max before a haproxy node has marked a node as unavailable.

I assume that issuing a "maintenance mode", and then wait for at least 20 seconds before starting the shutdown, should fix this.