Redis Sentinel vs. Redis Cluster

[begemotik]

I found the documentation describing different options for setting up HA Centrifugo. Though it's not clear which way of Redis config is preferable.

I found an article 8 years old where it seems that Sentinel was a better choice because of some PUB/SUB limitations of Redis cluster.

What about nowadays? Do you have any recommendations on the backend for Centrifugo if the expected load is somewhere:

• ~10k mes./sec
• ~50k mes./sec
• >100k mes./sec

[FZambia]

Using Redis is not only about PUB/SUB - it's also about possible message history storage, keeping presence information. I can't say nothing about exact numbers since given only message number is not enough - hardware, features used, latency – everything introduces side-effects. You need to benchmark for your app workload.

Also 100k msgs/sec - this can be fan-in messages, which introduce challenges and usually require scaling (more about this case below). Or fan-out number - in that case, it's very simple to scale with Centrifugo - as we showed in our blog 500k msgs/sec were simply achieved with hardware comparable to one modern server machine - and it's far from limit.

But some things I know exactly though. Centrifugo supports several models of HA Redis setup:

• Redis with Sentinel
• Redis Cluster
• Client-side sharding between several isolated Redis instances with Sentinel
• Client-side sharding between several isolated Redis Clusters

And some facts:

• Redis Cluster by default still uses PUB/SUB model where each message is sent to other nodes in cluster. This becomes a limiting factor on high fan-in (incoming) message rate. Increasing the number of masters in Redis Cluster results into more messages travelling over Redis nodes.
• Using isolated Redis instances with Sentinel and isolated Redis Cluster when using client-side sharding improves PUB/SUB throughput when we talk about fan-in. So you can set 8 isolated Redis instances with Sentinel, or 8 isolated Redis Clusters with single master in each - and get increased throughput. Sharding will be done on Centrifugo level. Which is good but requires accurate configuration, and not so automatic as it could be in the ideal case
• We also have implementation for Redis sharded PUB/SUB - https://redis.io/docs/interact/pubsub/#sharded-pubsub - but this support is not enabled now since it requires a real use case and testing. See Support Redis 7 sharded PUB/SUB centrifuge#228 . Some early benchmarks showed that it generally helps to improve throughput – but it comes with its own downsides unfortunately. This is a large topic to discuss TBH.
• You can use Nats as broker for Centrifugo if you don't need history and presence features. Nats has slightly more efficient PUB/SUB at scale than Redis as messages are distributed to adjacent nodes in Nats cluster. But it's also not infinite and has its own limits and cons – for example speed how subscriptions distribute over Nats cluster is limited. So in that perspective using Redis may be better.
• And Centrifugo has unreleased mode where presence/history are kept in Redis Cluster - and this scales very well. While PUB/SUB is done over Nats. This also requires separate discussion and validation on top of real use case.

[FZambia]

To show some numbers in fan-in case from local experiments (Mac M1, ARM, Number of Cores - 8):

Publish using centrifuge.Node.Publish (no subscribers, no history, no presence, 20 byte payload):

Single Redis: 1157668 op/s
Redis Cluster 4 nodes:  462815 op/s
Redis Cluster 8 nodes:  244955 op/s
Redis Cluster 4 nodes (unreleased sharded PUB/SUB with 128 cluster slots): 1650488 op/s
2 Redis instances with Centrifugo-level sharding: 1538624 op/s
4 Redis instances with Centrifugo-level sharding: 1801048 op/s
Single Nats: 2268514 op/s

Now with history enabled (saving to stream requires additional work here):

Single Redis: 88847 op/s
Redis Cluster 4 nodes:  243346 op/s
Redis Cluster 8 nodes:  135622 op/s
Redis Cluster 4 nodes (unreleased sharded PUB/SUB with 128 cluster slots): 280926 op/s
Redis Cluster 8 nodes (unreleased sharded PUB/SUB with 128 cluster slots): 331717 op/s
2 Redis instances with Centrifugo-level sharding: 163635 op/s
4 Redis instances with Centrifugo-level sharding: 280512 op/s
Redis Cluster 4 nodes + single Nats (unreleased mode): 300083 op/s

It's totally unscientific, publisher on the same machine which consumes up to 50% of CPU resources, etc - but it demonstrates some of the points above.

The last mode shown - Redis + Nats – possibly the most scalable for cases with history and presence, and has good potential if we add client-side sharding on Centrifugo level between different Nats clusters. But there are some restrictions in terms of publication ordering – publishers must ensure that publications sent one after another towards one channel. This may be the case when publications processed from Kafka topic partitions for example. Parallel publications to one channel may lead to client disconnections – this generally prevents to release the mode until we will have a real use case which will require this and will be fine with the restrictions.

[begemotik]

Much appreciate your thorough reply on the subject