Replication via Cloudflare Durable object SQLite Storage

[willhoney7]

Hi!

I’ve spent a bit of time researching and have been unable to find anything related to this, so please let me know if I missed a doc somewhere.

I’m considering RxDB for my local first application. I have been using Cloudflare durable objects to basically have a unique database per user. Has anyone worked on a first party replicator? Docs here: https://developers.cloudflare.com/durable-objects/api/sqlite-storage-api/

If I’m understanding correctly, to do this myself, the best approach would be to follow the HTTP replication guides and adapt the storage for Cloudflare, right?

Thanks!

[willhoney7]

Maybe my official request is for a durable-sqlite storage wrapper?

EDIT: I couldn't find it earlier, but looks like the adapters are available and I definitely could build my own: https://github.com/pubkey/rxdb/blob/b8823372608c60197ea482bcd28a1a8580ea4ec8/src/plugins/storage-sqlite/sqlite-basics-helpers.ts. I'm going to work towards this and may open a PR if there's interest?

[pubkey]

Yes you could make an sqlite-basics-implementation for durable-sqlite. Should be only a few lines of code. PR will be welcomed.

[willhoney7]

@pubkey Okay, I've spent a bit more time reading the docs and understanding. I can build a sqlite wrapper for durable sqlite and that would be fairly easy I think.

The thing that I'm now trying to figure out is how to use durable objects at all! Durable objects basically become a unique server+db for each customer (or however you want to split it up). For client side communication, generally you send of a request (http, websocket, etc.) to a worker (which is a more normal node backend) and it gets forwarded along to the unique durable object for that customer. So it's a unique server/DB for each customer.

I think I understand how I could build this following the pattern of using an HTTP replication client. I would just send up the requests and then forward them along to the correct customer's durable object/db which would then run the queries in the customer's unique DB. With this approach, the server doesn't care or know about rxdb directly (besides the DB structure and logic).

What I would REALLY like to do is be able to instantiate an RxDB within the durable object. The reason being that I want to be able to run queries and updates server-side. That's where using the RXStorage durable-sqlite wrapper would come to play. So, the client would have the RxDB with its storage and then the durable object server would have an RxDb with storage to durable-sqlite. The problem is that I'm not sure how to implement this concept. It's basically a custom RxServer implementation where it's synced to separate durable objects/DBs.

Is there a guide for implementing custom replication across two RxDb instances? (One on the client and one on the server.) Could I use the websocket replication from client to server? I would need to support authentication eventually which it doesn't currently support.

Also, am I along the right path? I've only been looking into this for a couple days 😅 . Appreciate you making it this far! I've been super impressed with the library.

[wuzzeb]

I've been thinking about this too, and I don't think putting rxdb into the durable object is the right approach. The server needs to be the authority on the checkpointing and conflict rejection. But, because of how durable objects work, checkpointing is very easy so it makes writing a replication backend straightforward.

Instead, I think the following approach:

• Use drizzle ORM in the durable object (drizzle supports the new durable object SQLite API)
• For the clients, write a custom rx replication plugin similar to the HTTP example (actually, I think looking at the firebase one for inspiration might be better). The client sends push/pull over a websocket to the durable object.
• For the server durable object, write some code to respond to the websocket messages using drizzle ORM.

The only downside is you need to describe the schema twice, once in drizzle and once in rxschema. But, drizzle does have support for typebox. So I think you could write the schema in drizzle, convert it to typebox, and the use the typebox schema as rxschema.

---

Long story short, I think the improvement for rxdb would be to support typebox. Typebox creates json schema (unlike say zod), so I think it would be enough to just extend typebox for the custom stuff rxschema needs like version. In fact, the following may already work

const T = Type.Object({
  id: Type.String(),
  foo: Type.Number()
});

// T is a json schema definition, so extend with the rx stuff

const rxT = {
  title: "The T type",
  version: 0,
  primaryKey: "id",
  ...T
};

[willhoney7]

@wuzzeb

I've been thinking about this too, and I don't think putting rxdb into the durable object is the right approach. The server needs to be the authority on the checkpointing and conflict rejection. But, because of how durable objects work, checkpointing is very easy so it makes writing a replication backend straightforward.

Hmm, I'm not sure I'm with you here. I want RxDB in my durable object because I want to do server-side logic with RxDB (e.g., run queries or ORM methods.) And I agree, having to define the schema twice is a big drawback.

Also, FYI, zod 4 does support json schema. See https://zod.dev/json-schema.