Skip to content

Conversation

roggervalf
Copy link
Collaborator

@roggervalf roggervalf commented Mar 25, 2025

Why

  1. Why is this change necessary? When someone ones to combine job scheduler logic with deduplication, there is a way to do it without breaking these iterations

How

  1. How did you implement this? Provide a pattern

Additional Notes (Optional)

ref #3165

Sometimes it is desired to deduplicate jobs that are generated by job schedulers to save resources and avoid unnecesary work:

Deduplication options are not available on [`JobSchedulerTemplateOptions`](https://api.docs.bullmq.io/types/v5.JobSchedulerTemplateOptions.html) because iterations need to be generated always, but it could be handled in a different way, so let's see an example:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something must be missing in this example because it currently does not make a lot of sense to me. You are adding a standard job with a deduplication id, and then a job scheduler with no relation to the previous job nor the deduplication id.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so the job scheduler, works as a producer for a job to be deduplicated. We cannot deduplicate a job generated by a job-scheduler, but if this generated job produce a new job, this one can be deduplicated

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the text needs to be improved then to explain this in detail. I do not think it is important to mention this last sentence: "but if this generated job produce a new job, this one can be deduplicated", as this is nothing particular to repeatable jobs or anything, it is trivial that you can add new jobs to any queue from the process function and therefore use any options you want including deduplicationId

@roggervalf roggervalf force-pushed the deduplicate-job-schedulers branch from bf3171d to 8e0fe26 Compare April 1, 2025 05:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants