[Bug]: One specific job stays stuck in waiting indefinitely, very odd

[vlrevolution]

Version

"bullmq": "^5.41.2",
"@taskforcesh/bullmq-pro": "^7.28.0",

Platform

NodeJS

What happened?

I have a wierd rogue job that gets stuck for one user, what could be the reasons for this? It just stays waiting and is never started, while other jobs work.

This is what I see in redis:

How to reproduce.

No response

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct

[vlrevolution]

Events stream shows this:

Nothing happens after the last waiting...

[vlrevolution]

submitting from a different user, everything works.. everything seems to be same, only difference is in the ULID based ids?

I'm really dumbfounded and been trying to figure this out, such a wierd thing, everything seems the same in terms of queue settings etc for both users, yet one of them gets stuck in waiting. And btw if there is no rate limit ttl when submitting, then it works also for the account that otherwise would get stuck?

[vlrevolution]

I'm also using redis-cluster setup, could this be somehow the reason why it can't find those waiting jobs?

There is a long running issue with ioredis (which is mostly unmaintained now) which I'm not sure if you guys are aware of, described here:
redis/ioredis#1759

and more precisely here:
redis/ioredis#1759 (comment)

I'm not sure how the implementation of bullmq clusters is working but perhaps it is related? Seems odd that something I'd exepct to work and works with other IDs, doesn't randomly work... seems like a nasty bug.

I'm wondering what my options are here :(

Perhaps I should have two redis, one of them non cluster for bullmq and the other for other stuff based on node-redis in cluster (which works perfectly)

[vlrevolution]

Gonna test now if changing to non clustered redis will work in the exact same setup...

[vlrevolution]

Gonna test now if changing to non clustered redis will work in the exact same setup...

Yup, it works perfectly in non cluster mode with exact same setup, so I think it is highly likely that cluster mode is actually broken because of the issues linked, and we need to start thinking about migrating to node-redis if possible since ioredis for scaling is broken. I think it would benefit the project to have cluster mode working as well

[manast]

Weird, I do not think IORedis is the issue here though.

[vlrevolution]

Weird, I do not think IORedis is the issue here though.

Interesting, what's your thinking? I have tried to use IORedis with cluster and it doesn't work reliably tho, so it seems related to me

[vlrevolution]

Under the hood, BullMQ uses subscribe right? So with cluster mode with ioredis, to have it know which slot to route to etc you are supposed to use ssubscribe: https://redis.io/docs/latest/commands/ssubscribe/

So it makes sense that it is not working properly, because subscribe doesn't by default work with cluster:
https://redis.io/docs/latest/commands/subscribe/

That's why there's ssubscribe, and unfortunately ioredis doesn't correctly implement ssubscribe actually rendering entire ioredis unsuitable for clustered mode

[manast]

BullMQ does not use subscribe. For detecting new jobs we use a mechanism based on BZPOPMIN, but by default, every 5 seconds we try to get jobs from the wait status anyway. But I wonder, when this job gets "stuck" in wait, are other jobs being processed and this is the only one not being processed?

[vlrevolution]

BullMQ does not use subscribe. For detecting new jobs we use a mechanism based on BZPOPMIN, but by default, every 5 seconds we try to get jobs from the wait status anyway. But I wonder, when this job gets "stuck" in wait, are other jobs being processed and this is the only one not being processed?

Yep, exactly as you describe. All the jobs from this user are stuck waiting, other jobs work fine and rate limit also is cleared so if I wait, new jobs are processed instantly

[manast]

That sound really weird because a job cannot just stay in wait while others are being processed, there is no mechanism in BullMQ to "skip" a job so to speak. Something to consider in any case would be to upgrade to a newer version, as the one you are using is pretty old.

[vlrevolution]

Actually I am using latest version, I mistakenly put that version hmm.

It must be related to redis & cluster I think, it's as if it cannot see that record, so it might be worth running some tests with actual sharded cluster because as you can see, the issue did happen and it's a freak issue, if I didn't notice it, then later in production, some users would report getting stuck and it would be impossible to debug or remedy it then

[vlrevolution]

Look

That sound really weird because a job cannot just stay in wait while others are being processed, there is no mechanism in BullMQ to "skip" a job so to speak. Something to consider in any case would be to upgrade to a newer version, as the one you are using is pretty old.

here you can clearly see that there is a waiting job that doesn't get processed before another job is processed, and it never gets processed (group concurrency is set to 1), other jobs continue to process:

[manast]

I am not sure what should I see in that screenshot, but I am quite sure that the job is not in waiting status in front of other jobs that are processed skipping it, thats simply impossible. It is better if you use a proper front end for BullMQ, I think you will see what is going on much better.