Being able to notify the job/queue to avoid stalling when job is CPU intensive

[maxime-dupuis]

Is your feature request related to a problem? Please describe.

My CPU intensive jobs would do "correct" work, but they stall with errors:

• Missing lock for job JOB_ID
• Error: job stalled more than allowable limit

async checkStaleCreatives(): Promise<any> {
    // Keep busy
    const startTime = Date.now();
    while (Date.now() - startTime < 30000) {
        // Spin
    }
}

Describe the solution you'd like

I'd like a way to tell BullMQ that my job is still working correctly. So that my job is not considered stalled.

await job.notifyNotStalled()
or
await queue.notifyNotStalled()

AND/OR the snippet below (or a better fix) should be in the documentation about stalled jobs so people know how to prevent the stalls

• https://docs.bullmq.io/guide/workers/stalled-jobs
• https://docs.bullmq.io/guide/jobs/stalled
• https://docs.bullmq.io/guide/troubleshooting#missing-locks

Describe alternatives you've considered

Adding this in the CPU intensive loop solves my problem

// Yield to the event loop to avoid stalls
await new Promise((resolve) => setTimeout(resolve));

[roggervalf]

hi @maxime-dupuis sorry for the delay. Looks like we need to improve our docs a little bit. We have these extra options available in Worker class https://api.docs.bullmq.io/interfaces/v5.WorkerOptions.html#lockDuration and https://api.docs.bullmq.io/interfaces/v5.WorkerOptions.html#lockRenewTime that you can increase. You can make some tries to check which values work for your use cases.

[maxime-dupuis]

lockDuration

Thanks, I think it might be useful sometimes if the expected duration is known.

In our case, the "yielding" trick works best for us because

• We can't predict the duration of the job (not obvious from my code example, sorry!)
• We have natural points where it makes sense to "yield" (every loop)

[manast]

In general you should not block the nodejs event loop unless you want to use the process for one thing and only one thing, in this case you can use sandboxed processors in BullMQ. Otherwise it is always necessary to divide the job in small chunks and yield.