Add NATS replica client #596

delaneyj · 2024-07-31T18:24:14Z

NATS is a distributed application framework. It has the ability to store objects similar to S3/Minio in a replicated persistence store. This replica client supports this feature for Litestream.

Note
The replica_client_test.go was geared towards only being tested against file. Have moved from fs/os errors to Litestream specific ones. This meant updating the file replica client to use these as well.

benbjohnson

I added a few points of feedback. Can you wire this up to cmd/litestream and add some instructions on how to test it?

file/replica_client.go

nats/replica_client.go

benbjohnson · 2024-08-13T20:32:41Z

nats/replica_client.go

@@ -0,0 +1,402 @@
+package nats


This doesn't seem to be imported or usable by the cmd/litestream binary. Am I missing something?

I had hooked it up (i thought) in cmd/litestream/replicate.go around line 125. Apparently this code wasn't commited

nats/replica_client.go

betamos · 2024-08-29T12:17:13Z

Big NATS fan here (oh, and Litestream too of course). Just stumbled across this PR accidentally and looked through it briefly. Some thoughts:

Why auto-create the bucket? IIUC litestream wants you to provision resources yourself (like the S3 client), which is trivial with the NATS CLI. If litestream is resposible, NATS has tons of config options that could pollute the litestream config in the future (currently I see replica count, which isn't necessary for litestream to know about).
Related: Retention is set on the NATS bucket. Without official support for out-of-bounds retention, I think it's much safer to let litestream manage retention in the same way it does elsewhere, using periodic checks.
Only JWT authentication? I think user/pass should absolutely be supported as a base-case, it's very common. (It can technically be encoded in the URL but since user/pass fields exist in litestream.yml, it violates the principle of least surprise).

Apologies for stepping in unannounced. Let me know if I can help out any other way.

delaneyj · 2024-08-29T14:14:47Z

@betamos thanks for taking the time

Why auto-create the bucket? IIUC litestream wants you to provision resources yourself (like the S3 client), which is trivial with the NATS CLI. If litestream is resposible, NATS has tons of config options that could pollute the litestream config in the future (currently I see replica count, which isn't necessary for litestream to know about).

I can see it both ways, if it's a blocker I'll remove it.

Related: Retention is set on the NATS bucket. Without official support for out-of-bounds retention, I think it's much safer to let litestream manage retention in the same way it does elsewhere, using periodic checks.

This is a good point. In general you want to set limits on buckets but this goes to your previous point.

Only JWT authentication? I think user/pass should absolutely be supported as a base-case, it's very common. (It can technically be encoded in the URL but since user/pass fields exist in litestream.yml, it violates the principle of least surprise).

I teach NATS usage at Synadia to customers. We highly recommend JWT only. User/pass, while supported currently support it really shouldn't be used. It's very easy to create JWT programmatically and have far superior control available.

Apologies for stepping in unannounced. Let me know if I can help out any other way.

Not at all, I think Litestream is an amazing tool and should be available in a NATS centric pipeline.

stumct-sa · 2024-10-14T19:10:58Z

I was looking through the litestream docs to see if it supported NATS then came across this PR. Would be great to see this get over the line.

I agree with the previous comment about the bucket configuration. I would expect to configure my own bucket and just tell litestream which one to use. There is a number of bucket configurations that will be set depending on my use case at the time. I suggest just taking in a bucket name and expecting it to exist.

JWT authentication is my preferred approach but again I wouldn't have litestream be so opinionated. It should support any NATS auth mechanism

delaneyj · 2024-11-14T14:30:29Z

I have been slammed with other projects but I agree with the comments above and will remove the bucket config from this sorry and look for a review

bruth

@delaneyj Took some time to review given our conversation this past week.

bruth · 2024-12-08T19:06:34Z

nats/replica_client.go

+}
+
+func (c *ReplicaClient) Cleanup(ctx context.Context) (err error) {
+	if c.nc == nil {


Suggested change

if c.nc == nil {

if c.nc != nil {

bruth · 2024-12-08T19:09:43Z

nats/replica_client.go

+
+func (c *ReplicaClient) Cleanup(ctx context.Context) (err error) {
+	if c.nc == nil {
+		c.nc.Close()


Suggested change

c.nc.Close()

c.nc.Drain()

// Block until draining is done. There is also a `ClosedHandler`

// callback option for the NATS connection where a channel

// could be used..

for c.nc.IsDraining() {}

bruth · 2024-12-08T19:18:24Z

nats/replica_client.go

+	BucketReplicas int
+	BucketMaxBytes int64
+	BucketTTL      time.Duration


I presume the goal is an out-of-box "no setup required", but for all other backends (based on a quick scan of the docs) require setting up the bucket ahead of time. These particular properties are operational concerns which should, IMO, be handled out-of-band and would be safer. For instance, the replicas would need to change for some reason. When Litestream would startup, an existing configuration could conflict with the remote settings of that bucket and either overwrite it (depending on the setting) or error since it has changed.

I suggest removing these and the requirement would be that a bucket is pre-created and you only need credentials and the bucket name when configuring this backend.

This would also remove the need to have upsertObjectStore.

bruth · 2024-12-08T19:23:38Z

nats/replica_client.go

+}
+
+// Returns a list of available generations. Order is undefined.
+func (c *ReplicaClient) Generations(ctx context.Context) (_ []string, err error) {


I did not peak at the other backend implementations, so if this is the idiomatic expectation in the codebase then 👍. However, at least for this backend implementation is a bit more difficult to read which values are being returned at times across some of the methods since there is scoped explicit assignment of err in particular vs. implicit.

Sorry, I didn't define what "it" is in the first sentence.. named return values.

bruth · 2024-12-08T19:30:36Z

nats/replica_client.go

+	}
+
+	b := buf.Bytes()
+	natsInfo, err := objectStore.PutBytes(ctx, filename, b)


Could use Put which takes an io.Reader directly.

bruth · 2024-12-08T19:35:42Z

cmd/litestream/main.go

+	JWT            string `yaml:"jwt"`
+	Seed           string `yaml:"seed"`


All auth methods should be supported. If an additional dependency is welcome, this package handles all the options as well as looking up the nats context profile. https://pkg.go.dev/github.com/nats-io/jsm.go/natscontext

bruth · 2024-12-08T20:40:08Z

Annnd.. I just realized I was a bit redundant with the other commentary 🤦. Apologies folks..

@delaneyj

This PR adds support for NATS JetStream Object Store as a replication destination for Litestream databases. Key features: - Modern NATS Go client (v1.44.0) with JetStream Object Store support - Multiple authentication methods (JWT, credentials, NKey, username/password, token) - TLS support with configurable connection parameters - Full compatibility with Litestream's LTX file format - Comprehensive unit tests and testing documentation - Example configuration file Implementation details: - Uses the latest jetstream package API for optimal performance - Requires pre-created buckets (following community feedback) - Supports all standard Litestream operations - Efficient chunking for large objects via JetStream This implementation is based on the original work in PR #596 by @delaneyj, updated to work with the current Litestream codebase and incorporating all community feedback. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]>

colinmollenhour · 2025-10-31T22:11:04Z

This is really interesting, can someone describe the use cases or advantages for it compared to a regular S3 bucket? I.e. how is this being utilized to do something different than before?

corylanou · 2025-10-31T23:19:33Z

Great question! Litestream supports multiple replica destinations (S3, GCS, Azure, SFTP, File, and NATS), each with different advantages depending on your infrastructure and deployment model.

Here are the key use cases and advantages for using NATS as a replication backend:

Infrastructure Consolidation: If you're already running NATS for messaging/eventing in your stack, you can use the same infrastructure for database backups without needing separate cloud storage services or credentials. See the NATS replica guide for configuration details.

Edge and Distributed Deployments: NATS JetStream excels at edge computing scenarios where you might want to keep backups close to where your application runs. The built-in clustering and replication capabilities make it easier to maintain backups across geographically distributed locations without managing multiple S3 buckets or regions.

Self-Hosted Infrastructure: For organizations that prefer to maintain complete control over their infrastructure, NATS provides a robust option without dependency on cloud providers. This aligns well with Litestream's philosophy of simplifying single-server deployments while still maintaining reliable replication.

Simplified Operations: In NATS-centric environments (especially Kubernetes clusters with NATS already deployed), you can avoid managing cloud provider accounts, API keys, and IAM policies. Authentication is handled through NATS's existing mechanisms (JWT, NKey, or username/password).

Cost Control: For organizations already running NATS infrastructure, there's no additional cost for object storage beyond the infrastructure you already have. This can be more economical than paying per-GB to cloud storage providers, especially for development/staging environments.

On-Premises and Air-Gapped Environments: NATS provides an excellent option for environments that can't or don't want to use cloud storage services, while still getting distributed, replicated backup storage.

In summary, if you're already invested in the NATS ecosystem or need self-hosted infrastructure, NATS is a natural fit. If you're AWS-native, the S3 replica remains an excellent choice. The beauty of Litestream is having these options to match your specific infrastructure needs.

initial commit

008dbab

benbjohnson requested changes Aug 13, 2024

View reviewed changes

delaneyj added 2 commits August 22, 2024 07:40

add nats to cli

eff24b7

fixes from PR review

f9898e4

delaneyj requested a review from benbjohnson August 22, 2024 16:42

bruth reviewed Dec 8, 2024

View reviewed changes

corylanou mentioned this pull request Jul 31, 2025

Review and help merge community pull requests #676

Closed

40 tasks

corylanou mentioned this pull request Aug 15, 2025

feat: add NATS JetStream Object Store replica client #712

Merged

5 tasks

corylanou closed this in #712 Aug 15, 2025

-		c.nc.Close()
+		c.nc.Drain()
+		// Block until draining is done. There is also a `ClosedHandler`
+		// callback option for the NATS connection where a channel
+		// could be used..
+		for c.nc.IsDraining() {}

Add NATS replica client #596

Add NATS replica client #596

Uh oh!

Conversation

delaneyj commented Jul 31, 2024

Uh oh!

benbjohnson left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

delaneyj Aug 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

betamos commented Aug 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

delaneyj commented Aug 29, 2024

Uh oh!

stumct-sa commented Oct 14, 2024

Uh oh!

delaneyj commented Nov 14, 2024

Uh oh!

bruth left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bruth commented Dec 8, 2024

Uh oh!

colinmollenhour commented Oct 31, 2025

Uh oh!

corylanou commented Oct 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

delaneyj Aug 22, 2024 •

edited

Loading

betamos commented Aug 29, 2024 •

edited

Loading