Skip to content

Cross-check my TODO list for a Python worker to work with SolidQueue #682

@MRigal

Description

@MRigal

In my current job, we have an open-source monorepo featuring a steering website, doing the data modelling and more in Ruby, linked to ETL processes ran in Python. We currently use faktory to communicate and send jobs between the two languages, but its limitations brought us to consider a move to SolidQueue, including writing a Python library worker+client/enqueuer.

I've been through the SolidQueue code and did not encounter difficulties to understand the main concepts, but would appreciate a short cross-check of my TODO list / plan to make sure I don't oversee something. I could see that there was some interest in #286 and #679 for that.

Python library to interact with SolidQueue

Basic choices

  1. Not reimplementing everything, no scheduler, no real dispatcher, almost just a worker.
    A thin dispatcher layer for threads, signals, config and co is probably required.
  2. Also no recurring tasks support in a first time
  3. Recommend to use language-specific queues to not mix Python and Ruby jobs

Worker feature set

  1. Get jobs by queue
  2. Get all jobs probably not in a first time
  3. Feature to have a Process (or Interpreter in Python 3.14+) Pool for concurrency
  4. Support for threads maybe in a second time
  5. Sync and async support?
  6. Add retry logic mimicking ActiveRecord that SolidQueue is relying on?

Client/enqueue feature set

  1. Possibility to enqueue a job.
  2. Bulk enqueue capacity maybe later

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions