Can I safely resize a Redis Cluster with BullMQ?

[Steffen911]

Hey all,

We're using the Redis cluster (https://docs.bullmq.io/bull/patterns/redis-cluster) mode with multiple prefixes for the "same" queue. Within our application we're sharding the single "ingestion-queue" into ingestion-queue-{1 to 6} as prefixes and assign each new job randomly to one of the prefixes. This is working well and helped us scaling beyond a couple of thousands events per second.

My understanding is that each prefix is mapped to a "Slot" in Redis and Slots are distributed across the shards within the cluster, i.e. ingestion-queue-1 may have slot 8000 which maps to shard 1. This will mean that all bullmq internal keys like the sorted sets etc. are created within this shard.

What I'm currently wondering is how this behaviour would work if we resize the redis cluster, e.g. if we go from 2 shards to 3 shards. In this case the slots are redistributed between the shards. How would bullmq behave if the slot for ingestion-queue-1 moves from shard 1 to shard 2?

The options I see:

• Redis rebalancing is performing some magic to move keys to the new shard and also merges the sets correctly.
• New incoming events are being lost if the rebalancing moves data and overwrites keys naively.
• Existing events in the queue are lost if new sets are started on the new shard without any migration happening.

Does anyone have experience with this flow and recommendations on how to handle the resizing?

[manast]

As Redis will migrate key by key, it is likely that keys that are part of one queue may be partially migrated, and thus operating with a queue in such a status will most likely produce unexpected errors. I would advise to pause the queues and stop the producers to add new jobs to a queue during the migration. It may help to migrate one slot at a time, so that only one queue will be stopped at a time.

[Steffen911]

@manast Would you expect that there will be actual errors in case that happens or would you also see "silently" dropped jobs as a possibility here?