Use multiples prefixes within same Queue in Redis Cluster Mode

[Steffen911]

Problem Description

We're having a single queue that receives approximately 4000 events per second and we anticipate this number to increase. We observe that at this scale the Redis Engine CPU utilization reaches 100% and vertical scaling does not mitigate it due to the single-threadedness of Redis.

Question

We want to use bullmq with the Redis Cluster mode to separate the processing across multiple instances. For this purpose our idea is to set different prefixes based on some identifier-hash from the event. E.g. we'd have prefix queue-name-1 to queue-name-8 and based on the tenantId we hash the prefix to either of them while keeping the queue name the same.

Is this a supported setup? What would be the expected behaviour from the bullmq side if the same QueueName is being used with different prefixes?

[manast]

When using cluster you must either prefix the queues with some word surrounded by braces, or surround the queue name with braces. This is the same technique required by Dragonfly for instance: https://www.dragonflydb.io/docs/integrations/bullmq
Btw, it is strange that your Redis reaches 100% with just 4k/s messages, that should not be normally the case using a decent Redis host.

[Steffen911]

@manast Thank you. What happens if I use a static queue name and combine it with different prefixes, e.g.

const queue1 = new Queue('my-queue', { prefix: '{my-queue-1}' });
const queue2 = new Queue('my-queue', { prefix: '{my-queue-2}' });

Could this produce any conflicts or have any other unintended behaviour? My goal is to make it feel like a single queue, but distribute the accesses across multiple nodes.