Getting complete duration in metrics

[Xrazik1]

Does this library have a built-in mechanism for retrieving information about jobs completion duration? I could not find any methods containing execution time info in QueueGetters.ts

[manast]

Jobs include a timestamp and a finishOn fields that can be used for computing durations: https://api.docs.bullmq.io/classes/Job.html
Furthermore there is a metrics facility: https://docs.bullmq.io/guide/metrics

[Xrazik1]

Jobs include a timestamp and a finishOn fields that can be used for computing durations: https://api.docs.bullmq.io/classes/Job.html Furthermore there is a metrics facility: https://docs.bullmq.io/guide/metrics

Thank you, could you also explain the meaning of parameters in that object?) Don't see any explanation in docs.

export interface Metrics {
    meta: {
        count: number;
        prevTS: number;
        prevCount: number;
    };
    data: number[];
    count: number;
}

For example, I have maxDataPoints equaling MetricsTime.ONE_HOUR
After a several jobs metrics method returns following:

{
"meta": 
   {
     "count":12,
     "prevTS":1661381723723,
     "prevCount":11
   },
 "data":["3","8"],
 "count":2
}

What does each of them mean?

[Xrazik1]

I'll write how I see that, correct me if I misunderstand something.

meta.count - count of all jobs per hour(referring to maxDataPoints)
meta.prevTS - ?
meta.prevCount - a value before updating meta.count

data - an array of counts of jobs per point (per minute)
count - the size of the array above (max is maxDataPoints)

[manast]

Meta is actually data used internally, maybe we should not expose it to avoid confusion. count is the number of jobs processed so far, prevCount the number of jobs processed up to the last minute boundary, prevTS is the previous TS where prevCount was updated. This is used to calculate the number of jobs that have been processed per minute.

[Xrazik1]

@manast
I have one more question about getting recent completed and failed jobs. Trying to get average job completion time.
There is a code below await queue.getJobs( [ 'completed', 'failed' ], 0, 60 ) here you can pull timestamp and finishedOn fields for that. But I've noticed this method returning static jobs(not recent). I tried using asc flag and there I got the same list of static data but from the other end of the list.

How to use this method to get the most recent jobs for a minute?

[manast]

But I've noticed this method returning static jobs(not recent)

Not sure what you mean with that. The method returns the completed/failed jobs with "descending" order by default:
https://api.docs.bullmq.io/classes/Queue.html#getJobs

[Xrazik1]

@manast
The problem here is that I have about 300 completed jobs per minute.

But the .getJobs method always returns static job data.
const jobs = await queue.getJobs( [ 'completed', 'failed' ], 0, 5, false )
JSON.stringify(jobs.map((job) => job.id ))
=> ["2048407","2048406","2048405","2048403","2048402","2048401","2048383","2048327","2048280","2048218","2048031","2048023"]

As well as with "asc = true" parameter

[manast]

I am sorry but I do not understand what you mean. What is "static job data" ?

[Xrazik1]

I mean that the result of that function does not change over time

[Xrazik1]

If I have more than 5 completed jobs done per minute I expect that this method will return 5 last completed jobs

[manast]

Are you sure you are getting the complete from the correct queue? And what settings do you have on the worker? The best is that you provide a complete use-case that demonstrates the issue.

[JCMais]

@Xrazik1 did solve this?

[krisskod]

export async function getJobsPerMinute(queue: Queue): Promise<number> {
  // Fetch metrics for completed jobs
  const metrics = await queue.getMetrics('completed');

  // Check for insufficient data: prevTS might be zero or metrics might be empty
  if (!metrics || metrics.meta.prevTS === 0) {
    return 0;
  }

  // Calculate elapsed time in minutes using the difference between timestamps
  const elapsedTimeInMinutes = (Date.now() - metrics.meta.prevTS) / (1000 * 60);

  // Handle division by zero
  if (metrics.meta.count === 0) {
    return 0;
  }

  // Calculate and round the average jobs per minute
  const avgJobsPerMinute = metrics.meta.count / elapsedTimeInMinutes;
  const roundedAvg = Math.round(avgJobsPerMinute);

  return roundedAvg;
}