Are there any plans to implement Unique Jobs for workers?

[oco-adam]

Rusty sidekiq has a unique_for option for the worker. Are there any plans to expose/implement this?

[jtwaleson]

Not sure if this is what you need from unique_for, but I created this check (for postgres) to avoid scheduling duplicate jobs:

pub async fn has_similar_job_in_queue(
    db: &DatabaseConnection,
    worker_name: &str,
    task_id_field: &str,
    task_id: i32,
) -> Result<bool> {
    let result = db
        .query_one(sea_orm::Statement::from_sql_and_values(
            sea_orm::DatabaseBackend::Postgres,
            r#"
            SELECT COUNT(*) as count FROM pg_loco_queue
            WHERE name = $1
            AND status IN ('queued', 'processing')
            AND task_data->>$2 = $3
            "#,
            [
                worker_name.into(),
                task_id_field.into(),
                task_id.to_string().into(),
            ],
        ))
        .await?;

    if let Some(row) = result {
        let count: i64 = row.try_get("", "count")?;
        Ok(count > 0)
    } else {
        // No result row returned, defaulting to false
        Ok(false)
    }
}```

[oco-adam]

Hi @jtwaleson thanks very much for sharing this! We're currently using Redis for the job queue but it's still very helpful to know that you're doing this.

[kaplanelad]

Thanks for the idea, and thanks @jtwaleson for the suggestion!

I believe we can implement this function for each queue type here.

Once implemented, this function would be available in AppContext like this:

app_context.queue_provider.unwrap().has_similar_job_in_queue()

This means you’ll be able to use the same functionality across all our workers. For example, @jtwaleson, if you were to migrate from Postgres to another supported queue, it would be more challenging without this.

(The function name has_similar_job_in_queue is just an example.)

Any thoughts?

[jtwaleson]

I like it but not sure if all of the queues can be queried. E.g. if you'd want to implement something like AWS SQS, not unreasonable for a job queue, you can't query.

[kaplanelad]

I tend to agree, but I believe it could still be achieved through an API request—though that might be overkill.