Local Group Rate Limit and Local Group Concurrency cannot be configured on a BullMQ worker at same time

[vishuvig]

(Follow up from previous issue: #116)
We're not able to configure a BullMQ Pro worker to have both local group rate limit and local group concurrency at the same time.

Code to replicate issue:

const generalPurposeQueue = newQueuePro("general-purpose", { connection });

const worker = new WorkerPro("general-purpose", processFunc, {
  concurrency: 1, // only job at a time per worker
  group: {
    concurrency: 1, // default 1 job per group across worker pool
    limit: {
      max: 1,
      duration: 100000, // 100 seconds
    },
  },
  connection,
});

Expected Behaviour

The worker starts up and we can use both concurrency and rate limit configs for groups in the queue.

Actual Behaviour

Exception on startup

stack: Error: Rate limit and concurrency cannot be used together
  at new WorkerPro (/obsidian/node_modules/@taskforcesh/bullmq-pro/src/classes/worker-pro.ts:161:13)
  at startWorker (/obsidian/packages/core-infra/src/bullmq/core/worker.ts:305:18)
  at processTicksAndRejections (node:internal/process/task_queues:105:5)
  at startup (/obsidian/apps/general-purpose-worker/src/index.ts:84:3)

We're on version 7.35.2 of bullmq-pro and version 5.53.1 of bullmq. Let me know if you have any question or anything I can clarify further.

[roggervalf]

hi @vishuvig, currently it's not possible and also it was not though to work together when concurrency was introduced. Could you pls explain a little bit your use case as your example should be the same as just applying limit only because both cases have same max value 🤔

[vishuvig]

Hey @roggervalf , sure thing. An overview of our use case is that we have a common worker pool with different teams writing jobs to be run in this shared pool. Each job has slightly different characteristics and we want to be able to have each team configure the concurrency and rate limit for their jobs (using local groups).

The specific example above is just to set a baseline but we would be using the setGroupRateLimit(...) and setGroupConcurrency(...) functions to set the config for each group. I don't think we can just use rate limits for our desired worker setup.

Would it be possible to add support for this use case?

[roggervalf]

hi @vishuvig I started a discussion with my team. Looks like when concurrency of groups was introduced, there was not an intention to allow rate limits and concurrency being set at the same time and it may require some changes from our side to allow it. I'll keep you posted here about it