Add "requeue" deduplication mode to ensure at least one execution after last trigger

[jtomaszewski]

I need to ensure a job runs at least once after an action occurs, while avoiding unnecessary duplicate executions when multiple triggers happen in rapid succession.

Currently, BullMQ's deduplication modes have a critical gap:

• Simple mode: Deduplicates until job completes, but if a trigger occurs while the job is actively processing, that trigger is lost forever
• Throttle mode: Similar issue - triggers during job execution are ignored
• Debounce mode: Requires a delay and doesn't handle triggers during active processing

This is a common pattern needed for:

• Data synchronization: Multiple data changes should eventually sync, with at least one sync after the last change
• Cache invalidation: Multiple updates should trigger invalidation, ensuring cache is refreshed after the last update
• Webhook processing: Multiple webhook events for the same resource should be processed at least once after the last event

Related to #3151 (comment), which describes a similar need, but I think hasn't been completely fulfilled yet.

Describe the solution you'd like

A new deduplication mode that guarantees at least one job execution after the last trigger attempt:

await myQueue.add(
  'process-action',
  { data },
  {
    deduplication: {
      id: 'unique-job-id',
      requeueIfActive: true // new option
    }
  }
);

Behavior:

1. If job with same dedup ID is waiting/delayed → Deduplicate (ignore new job)
2. If job is active (processing) → Mark for automatic requeue after completion
3. If job is completed/failed → Allow new job immediately
4. While marking a job as complete → Check if requeue was requested and automatically schedule a new job

This ensures:

✅ No unnecessary duplicate processing when job is already queued
✅ No lost triggers when job is actively processing
✅ Automatic handling without manual event listeners
✅ Works across multiple Node.js processes

Example use case

// Deployment pipeline triggered by git webhooks
async function onGitPush(commitData) {
  await deployQueue.add(
    'deploy',
    { commit: commitData.sha },
    {
      deduplication: {
        id: `deploy-${commitData.repo}`,
        requeueIfActive: true
      }
    }
  );
}

// If multiple pushes happen:
// - Push 1 at 0ms: Job scheduled
// - Push 2 at 100ms (job waiting): Deduplicated
// - Push 3 at 5000ms (job active): Deduplicated but marked for requeue
// - Push 4 at 5100ms (job active): Deduplicated, requeue already marked
// - Job completes at 10000ms: New job automatically scheduled
// Result: Exactly 2 deployments run, last push is always deployed

Describe alternatives you've considered

• Manual tracking with deduplicated events: Requires complex state management in Redis, prone to race conditions
• Using job.updateData() during processing: Worker needs to check for updates, prone to race conditions too
• Version-based deduplication: Every trigger creates a job (no deduplication benefit)
• External pending queue: Adds complexity and potential for race conditions

None of these alternatives are as clean or reliable as having this built into BullMQ.

Additional context

I'm willing to contribute a PR for this feature if the maintainers agree with the approach. The implementation could:

• Use atomic Redis operations to track requeue requests
• Integrate with existing deduplication infrastructure
• Include comprehensive tests
• Maintain backward compatibility
• Add minimal performance overhead (only for jobs using this mode)

Would the team be open to this feature? If so, I'm happy to discuss implementation details and submit a PR.

[jtomaszewski]

Example implementation (could improve of course): master...jtomaszewski:bullmq:vk-cfa1-requeue-mo

[manast]

Unless I am missing something, what you are actually requesting is that in simple deduplication mode we do not deduplicate if the job is already in active status. If this assumption is correct, then I think "at least once" is a matter of definition, as simple mode works in at least once semantics already, it is just that what we consider a job to be completed or failed as the actual status that determines the semantics, and not "started to process". I am not sure why this request would be so important for you. For instance, lets say that the operation is a database update, it seems to me it is much better to deduplicate until the previous update has been completely processed to avoid out-of-order scenarios. With your suggestion of adding a "requeueIfActive" option it will not prevent the workers to process the next job before the previous one has been processed, opening for out-of-order execution.

[jtomaszewski]

I think "at least once" is a matter of definition, as simple mode works in at least once semantics already,

Current behaviour is "a job completes at least once after it is scheduled".

What I need is "a job starts and completes at least once after it is scheduled".

it seems to me it is much better to deduplicate until the previous update has been completely processed

I need to be sure that a job is run for the most fresh data at least once.

Current deduplication doesn't allow me for that. Unless I set some super long arbitrary delay + TTL time, but this will greatly delay any work done by my application.

To illustrate my need, see the "Example use case" I placed in the original post ("Deployment pipeline triggered by git webhooks"). In that use case:

1. I want any deployments to start as soon as possible.
2. I want the most recent deployment to happen at least once.
3. I don't want deployments ever to be processed in parallel.

[manast]

Current behaviour is "a job completes at least once after it is scheduled".

What I need is "a job starts and completes at least once after it is scheduled".

A job cannot complete unless it has started so I am not sure what you mean :thinking_face:

I need to be sure that a job is run for the most fresh data at least once.

Current deduplication doesn't allow me for that. Unless I set some super long arbitrary delay + TTL time, but this will greatly delay any work done by my application.

To illustrate my need, see the "Example use case" I placed in the original post ("Deployment pipeline triggered by git webhooks"). In that use case:

1. I want any deployments to start as soon as possible.
2. I want the most recent deployment to happen at least once.
3. I don't want deployments ever to be processed in parallel.

I think it does, it's just that your definition of freshness does not align with what BullMQ is offering. Basically you want to consider a job old as soon as it starts processing, whereas for BullMQ a job is fresh until it completes. In my view what you seem to want to achieve is having at least one job enqueued if there is already a job being processed right now. As of today, this is the first time we hear about a requirement like this one, so we are most likely not going to implement this particular option that you are proposing. However if more users request similar use cases we may try to find a more general solution that covers these and other similar cases.

[manast]

Btw, I think that if you had a deploymentId that is unique for every deployment that actually starts, that would for the most part do what you want, if you add a new job with a deployment and that deployment has not yet started it will be deduplicated as they will have the same id, on the contrary, if the job has already started and updated the next deployment id, if you add a new job it will not be deduplicated.

[jtomaszewski]

Btw, I think that if you had a deploymentId that is unique

Unfortunately there's a few scenarios where I can't just simplify the answer to a question of "should the job x be reexecuted for the most fresh data?" to a single ID. I just want a simple "fire and forget" call, i.e. "bull pls just make sure that this job is run at least once from now on".

In my view what you seem to want to achieve is having at least one job enqueued if there is already a job being processed right now.

Yes.

As of today, this is the first time we hear about a requirement like this one

#3151 seems to be talking exactly about this. Perhaps others are fine with long delays and the job being processed "at some later point in time", rather than as soon as possible; and that's why they're not being vocal about this being needed.

To give example from another similar project, Ruby on Rails & ActiveJob, that need is fulfilled there through concurrency and uniqueness controls.

In Temporal, that need is fulfilled through BufferOne overlap policy .

we are most likely not going to implement this particular option that you are proposing

:( Please anybody who's reading this, give feedback or 👍 to the thread to highlight that's needed.

Until then, I guess I'll maintain a fork ( master...jtomaszewski:bullmq:vk-cfa1-requeue-mo , published on npm as @jtomaszewski/bullmq ), which I've already tested out to be working well in my project.

[manast]

The difficulty with this feature relies in avoiding the same job running in parallel, in general it will only work as designed for 1 worker with concurrency 1.

[jtomaszewski]

Don't we have locks for this? (to prevent workers from fetching a job that is already being processed by another worker)

And lua scripts for atomicity of Redis operations (such as requeuing a job while it's being marked as complete, for example).

[manast]

Yes, but they are not really the same job, one job is the one being processed and the other one is the one you add afterwards.

[jtomaszewski]

Can you describe me a scenario in which the job would be executed twice in parallel? I can't think of any.

Nevertheless, I'd keep that as implementation detail that can be tackled once you're open to a PR for this.

[manast]

@jtomaszewski it is quite trivial actually, the first starts processing, you add the second job and then if there is a free worker you will have two workers working in each of the jobs in parallel.

[manast]

again, you may get confused by the nomenclature. These are not the same job, they are different jobs.

[jtomaszewski]

I think I've described that scenario in the "Example use case" section of the original post?

If a job is scheduled while a different one is being currently processed (with the same deduplication ID), then the only thing that happens is that we mark the job as "to be requeued" (by adding a ${queue}:re:${deduplicationId} key). See master...jtomaszewski:bullmq:vk-cfa1-requeue-mo#diff-d9f13c1f34003d3ac0ae19b4cd3acdcac17f7668118fecba2df0916a437531c8R18 . No second job is being added to the queue, so there's no way for it to be processed in two workers simultaneously.