Skip to content

Commit

Permalink
feat(dgw): persistent job queue for crash resistance (#1108)
Browse files Browse the repository at this point in the history
This year we added some background tasks in the Gateway that should
not be canceled, or if they are, should be restarted later. Essentially
two tasks: mass deletion of recordings (relatively important, but
it's always possible to launch indexing in DVLS in case of a problem)
and remuxing recordings to webm format (good to have). If the service
is killed in the middle of one of these operations, we should resume
execution on the next startup.

This persisent job queue is implemented using Turso’s libSQL. Using
libSQL (or SQLite) for implementing the queue allow us to benefit from
all the work put into implementing a reliable, secure and performant
disk-based database instead of attempting to implement our own ad-hoc
storage and debugging it forever.

Inspiration was taken from 37signals' Solid Queue:

- https://dev.37signals.com/introducing-solid-queue/
- https://github.com/rails/solid_queue/

And "How to build a job queue with Rust and PostgreSQL" from kerkour.com:

- https://kerkour.com/rust-job-queue-with-postgresql

The 'user_version' value, which is a SQLite PRAGMA, is used to keep track
of the migration state. It's a very lightweight approach as it is just an
integer at a fixed offset in the SQLite file.

- https://sqlite.org/pragma.html#pragma_user_version
- https://www.sqlite.org/fileformat.html#user_version_number

Introducing Turso’s libSQL, as opposed to SQLite, will serve us for
"Recording Farms" in the future. We’ll want instances of a same
Recording Farm to coordinate. At this point, we’ll want to use Turso's
libSQL network database feature. Indeed, putting the SQLite database
file on a virtual filesystem is not recommended. This can lead to
corruption and data loss. Turso will allow us to have a local mode for
the simplest setups, and a network and distributed mode for Recording
Farms when we get there.
  • Loading branch information
CBenoit authored Nov 15, 2024
1 parent 1442a1d commit 2420b07
Show file tree
Hide file tree
Showing 14 changed files with 1,736 additions and 69 deletions.
661 changes: 630 additions & 31 deletions Cargo.lock

Large diffs are not rendered by default.

21 changes: 21 additions & 0 deletions crates/job-queue-libsql/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
[package]
name = "job-queue-libsql"
version = "0.0.0"
edition = "2021"
authors = ["Devolutions Inc. <[email protected]>"]
publish = false

[lints]
workspace = true

[dependencies]
anyhow = "1"
async-trait = "0.1"
job-queue = { path = "../job-queue" }
libsql = "0.6"
serde = "1"
time = { version = "0.3", default-features = false, features = ["std"] }
tracing = "0.1"
typed-builder = "0.19"
ulid = { version = "1.1", features = ["uuid"] }
uuid = "1.11"
Loading

0 comments on commit 2420b07

Please sign in to comment.