[Bug]: Jobs marked failed and stuck in Redis. getJobs doesn't return the job.

[aliabbasrizvi]

Version

v5.21.2

Platform

NodeJS

What happened?

We are noticing an issue where we add a job with ID and set removeOnFail to true and removeOnComplete to true as well.

What we notice is that sometimes the job fails and keeps occupying a key in Redis.

When we look up data for the key we see the following:

redis:6379> hgetall bull:content-changed:zarxmjtdcgoko19
1) "failedReason"
2) "Missing key for job zarxmjtdcgoko19 finished"
3) "stacktrace"
4) "[\"Error: Missing key for job zarxmjtdcgoko19 finished\\n    at Object.finishedErrors (/app/node_modules/bull/lib/scripts.js:223:16)\\n    at /app/node_modules/bull/lib/scripts.js:212:23\\n    at processTicksAndRejections (node:internal/process/task_queues:95:5)\"]"
5) "attemptsMade"
6) "1"

As a result, subsequent jobs with same ID are not getting added to the queue thereby causing issues for us.

When we look up jobs using getJobs (https://api.docs.bullmq.io/classes/v5.Queue.html#getJobs), the failed job above doesn't appear.

Could you help us understand why this is happening and how we could avoid this?

How to reproduce.

No response

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[roggervalf]

hi @aliabbasrizvi from what I can see in your log, it's pointing to bull package, no bullmq. You may be using bull and bullmq pointing to the same queue. This is not recommended as bullmq has many other changes that are not compatible with bull anymore.

[roggervalf]

also for reference, some time ago this fix was introduced in bull package OptimalBits/bull#2755 as part of v4.15.1