Advice on use case: one queue per job?

[Valahaar]

Hi! BullMQ user for around two years now, been loving it!

I have a use case for which I'd like to get some advice.

Say that I have a potentially long-running (from 1 min to a few hours) job that I need to execute as a (Google) Cloud Run Job, and that I can have many of these jobs running at the same time. I have thought of two possible ways to do this, and I'd like to know whether I've got my bases covered :)

Scenario 1:
Job J gets placed on queue Q. Worker for Q picks up J, and its processor updates J's data with some essential stuff that is needed for the cloud execution, and then spawns an instance of the Cloud Run Job providing it with J's id. The cloud runner connects to Q and requests job J, and starts its long-running operation.

Scenario 2:
Job J gets placed on queue Q. Worker for Q picks up J, and its processor adds a job on queue X (where X is a unique name) and spawns the cloud runner, providing it with X. The runner then creates a worker for X and processes the (only) job available, which will run until completion.

In both scenarios, I do update the progress with some custom objects that help me track what's going on in the job.

There are a few questions I have, because I'm not sure if the Sc2 is overkill or is actually the better way to do this:

• In Sc1, would I be able to just do job.progress from within the processor? Or would I have to start a QueueEvents to intercept those updates? (I'm assuming the latter)
• In Sc1, would I be able to run job.updateProgress() and job.updateData() from within the cloud runner, or is there a lock placed which would prevent me from doing so? (I'm assuming the lock mentioned in the docs is just to prevent two workers from working on the same job, but in Sc1 the runner just picks up the job data from the queue)
• In Sc2, what would be the best way to get progress updates from the cloud runner? A QueueEvents instance again, right?
• What if I want to run some post-execution cleanup code after the runner has finished executing? Would it be better to keep Q's worker on hold until the runner finishes (and then execute it there), or to create another queue for post-job cleanup? (think of this as a finally in a try...catch...finally)
• And for the most important question: what about retries? What if I want the cloud part of the execution to be the only one retried in case of a failure? My guess is that in Sc1 I'd have to just re-run the cloud job itself, while in Sc2 it would re-run by itself given that the Worker and Job have the retry options set.

By reading #2489 it appears that having many queues is just painful in terms of debugging (but we can solve that with observability tools), but I'd like to be sure before proceeding.

Hope I've been clear in my explanation :D

Thanks in advance!

[manast]

Another option, use the BullMQ Proxy so that your cloud runner is called by the processor itself, not sure it will work in your specific case but it seems doable: https://github.com/taskforcesh/bullmq-proxy

[Valahaar]

Uhm honestly I'm not sure what you mean here. Can you write down the flow? Thanks!

[manast]

With the proxy you register the URL that you want to call when a job is processed, here is more info, let me know if you need more details: https://docs.bullmq.net