🔄 How to correctly push status updates for parallel jobs using BullMQ and Airflow

[NicolasBearingPoint]

Hi BullMQ team 👋

I'm using BullMQ in combination with Apache Airflow as an event-driven system. I understand that Airflow isn't natively designed for such an architecture, but it works well in practice for our use case. However, I’m facing a design challenge around job tracking and parallel executions.

Context:

• Our backend pushes jobs into a {dags-trigger} queue to trigger Airflow DAGs.
• A separate {dags-finished} queue is used to track the final status of each DAG.
• Each job carries a unique_id that ties it to a specific forecast process.
• My Airflow DAGs, which can run in parallel (based on the jobs in {dags-trigger}), include a final task that needs to update the job with the matching unique_id in the {dags-finished} queue.
• Every tasks are independent and running in a separate pod.

Since multiple DAGs can be running concurrently, each final task creates a BullMQ worker that consumes jobs from {dags-finished} in FIFO order.

Problem:

How can I reliably find and update the exact job (matching unique_id) in a shared queue without blocking other jobs needed by other concurrently running DAGs?

Describe alternatives you've considered:

The workaround I’ve implemented is as follows:

• Loop over the jobs in the queue:
• If the unique_id matches, set the job’s state to completed (or failed), then stop the loop and task.
• If it doesn’t match, call Scripts.moveToDelayed() and then Scripts.promoteJobs() to push the job back into the queue, and continue the loop.
• This allows other jobs to eventually get picked up by their corresponding DAGs later.

Caveats:

I had to add a time.sleep of ~1s when triggering the DAG to avoid race conditions, especially when two DAGs are triggered at the exact same time or within a very short delay (less than the delay of the moved job).

This problem does not occur when triggering DAGs, since those jobs are handled sequentially and don’t depend on matching parameters.

Question:

Is there a better way to find and update a specific job in a shared queue (e.g., by ID or metadata) without consuming it first via FIFO?
Have you faced
Or is there a BullMQ-native way to target a job in a queue (e.g., by jobId or unique_id) and change its state directly?

Any help or best practices would be greatly appreciated. Thanks for the great work on BullMQ! 🙏

[manast]

Have you looked into flows to see if there is anything there that can help you in your use case?https://docs.bullmq.io/guide/flows https://blog.taskforce.sh/splitting-heavy-jobs-using-bullmq-flows/

[NicolasBearingPoint]

Hi @manast, thank you for your help! Yes I am already using flows for my {dags-finished} queue where parent jobs are set to active when the {dags-trigger} child jobs are completed. Also I don't really need chunks as each DAGs' job finished status is independent.

My main concern is on how to set {dags-finished} jobs to completed (or failed) for every parallel DAGs once finished? Knowing the FIFO method will lock job for the first DAG which get it, and the others won't be able to get it then.

[roggervalf]

hi @NicolasBearingPoint. Parent jobs as in your example {dags-finished} jobs will be moved to waiting or prioritized as soon as their children are finished successfully. As they are part of a queue, they will be processed (moved to active) depending on your configuration, like how much concurrency you have defined (how many parent jobs can be executed at the same time) and how many dedicated workers you have for that queue. Locking jobs is expected as this is how workers are managing jobs. The time of locking will depend on how much time it takes for those jobs to finish. Jobs must be in active state before passing to completed or failed.