Shutting down active peer QQ members after force shrink to 1 node

[Ayanda-D]

Community Support Policy

• I have read RabbitMQ's Community Support Policy

RabbitMQ version used

3.13.7

How is RabbitMQ deployed?

Debian package

Steps to reproduce the behavior in question

Hi Rabbit Folks 👋

I have a question on QQ force-shrink to 1 node commands (seems to be a design thing). Is there a reason why peer QQ members (the specific ra_server_proc FSMs) aren't shutdown on other-nodes connected to the node on which QQs have been shrunk to? After force-shrinking QQs to a single node, we'd excepted any connected peer QQ procs to be stopped and only the current node to have an active ra_server_proc process.

What shutting down peer connected FSMs allows us to do (from our tests), is after a disaster scenario, we can re-grow the shrunk QQs to the available nodes (which may not necessarily need to be wiped out and rebooted). As it currently stands, after force-shrink QQs to 1 node, any other nodes which werent in a bad state can't be re-grown to.

[michaelklishin]

Shrinking of quorum queues to a single node is a very rare thing to do. So no one has noticed. I don't think this behavior will be considered important to fix.

[Ayanda-D]

Ok cool. We're just preparing for some (extreme) emergency disaster rescue routines - & i agree these should be very rare. Thought this behaviour was intentional with a purpose we might be missing. Thanks for the answer!

[michaelklishin]

It's not intentional and I assume that if someone comes up with a PR that would safely shut down those local replicas, it would be considered. You can use rabbitmq-upgrade drain for inspiration, it has been recently corrected to truly select only local replicas.

That said, my gut feeling tells me this is one of those changes that seem trivial but end up being a a decent can of worms.

[Ayanda-D]

Ok cool. Thanks for the info, I'll leave a PR of what we're testing out for you folks with better insight here to see, and I agree - a possible decent can of worms can't be ruled out. Also at that point when we're now shrinking queues to 1-node, we'd already be at a point of ultimate disaster anyway (worms already spewing out 🪱) and seeking to rescue whatever we can. If we can re-grow back to any few surviving cluster nodes, it could be useful for some scenarios where we don't want to necessarily after the shrink-to-1, migrate and shovel across to a fresh cluster.

Thanks, I've also checked rabbitmq-upgrade drain, we can definitely make use of it and what transfer_leadership_of_quorum_queues/1 will give us - safely pulling out a node, & having leadership safely go to stable nodes while we fix things. Thanks for the answer & suggestions.