[PERF] Cluster Sharding perf issue

[carl-camilleri-uom]

Version Information
Version of Akka.NET? 1.4.23
Which Akka.NET Modules? Akka.Cluster.Sharding

Describe the performance issue

A minimum viable repo that reproduces the issue is at https://github.com/carlcamilleri/benchmark-akka-cluster

Running two n2-standard-8 nodes in GCP (8 CPU @ 2.80 GHz and 32GB RAM) with Windows Server 2019 in GCP ("instance-1" and "instance-2"), and a third machine to run the benchmarks from

First check:
curl http://instance-1:5000/5
Response: akka://ping-pong-cluster-system/system/sharding/PingPongActor/13/5(pid=2372,hostname=instance-2)

Therefore entity id 5 actor is hosted on instance-2 server

wrk -t48 -c400 -d30s http://instance-2:5000/5

Running 30s test @ http://instance-2:5000/5
48 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 19.54ms 70.10ms 1.09s 94.22%
Req/Sec 2.41k 429.13 12.12k 86.40%
3360471 requests in 30.10s, 666.60MB read
Socket errors: connect 0, read 0, write 202, timeout 0
Requests/sec: 111645.29
Transfer/sec: 22.15MB

wrk -t48 -c400 -d30s http://instance-1:5000/5

Running 30s test @ http://instance-1:5000/5
48 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 254.21ms 286.27ms 1.02s 78.54%
Req/Sec 119.09 127.71 0.94k 85.22%
73325 requests in 30.03s, 14.55MB read
Socket errors: connect 0, read 0, write 216, timeout 0
Requests/sec: 2441.41
Transfer/sec: 495.91KB

For interest I've also repeated test (1) (i.e. workload on the endpoint which requests the local actor) but with serialize-messages = on, and the result is:

Running 30s test @ http://instance-2:5000/5
48 threads and 400 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 24.77ms 90.16ms 1.15s 94.33%
Req/Sec 1.89k 339.47 4.94k 90.42%
2579946 requests in 30.08s, 511.77MB read
Socket errors: connect 0, read 0, write 246, timeout 0
Requests/sec: 85760.95
Transfer/sec: 17.01MB

So Hyperion serialisation drops throughput from >111k to >85k, which is probably expected

Data and Specs

ASKing a local actor I get >111k req/s throughput, but ASKing a remote actor drops throughput to 2.4k req/s.

Expected behavior
Cross-Machine communication in Cluster Sharding expected to be faster.

Actual behavior
Cross-Machine communication in Cluster Sharding seems to be extremely slow and unusable for my use case (an OLTP workload)

Environment
.NET 5.0
Windows Server 2019
n2-standard-8 machine in GCP (8 CPU @ 2.80 GHz and 32GB RAM)

[Aaronontheweb]

ASKing a local actor I get >111k req/s throughput, but ASKing a remote actor drops throughput to 2.4k req/s.

The local actor - is this actor hosted by the sharding system or is just a local actor floating in-memory?

[Aaronontheweb]

Need to differentiate where the bottleneck is:

• Sharding system
• Remoting system

If it's the remoting system, there's things you can do to adjust but ultimately we have to replace our transport system (it's the lion's share of the v1.5 roadmap)

If it's the sharding system we can patch that too - opened a similar issue on the entity spawning side last week: #5190

Can you give us some color here to inform these numbers? We'll take a look at the source too.

[carl-camilleri-uom]

Thanks, in reply to:

ASKing a local actor I get >111k req/s throughput, but ASKing a remote actor drops throughput to 2.4k req/s.

The local actor - is this actor hosted by the sharding system or is just a local actor floating in-memory?

Correct it's hosted by the sharding system, in both cases the REST endpoint performs this call:

var resPong = await _shardRegion.Ask<MsgPong>(new ShardEnvelope<MsgPing>(entityId, new MsgPing()));

So the "local" test is actually a call to the REST endpoint hosted on the machine where the sharding system has spawned the actor.

Happy to provide further details or tests that can help.

Thanks

[Aaronontheweb]

My working theory on this is, assuming you're getting both numbers from the same sample:

1. The hops used to route messages to remote Shards passing through the ShardRegion introduces additional routing latency - especially since you're not pipelining in that code sample;
2. The actual ShardRegion actor and Shard actors themselves, functioning as routers, are doing so in an inefficient manner and introduces latency in both local + remote scenarios, but it's more visible in the latter for reasons related to the additional routing that has to occur;
3. There is going to be inherent latency in the Akka.Remote I/O pipeline, which is unavoidable, but shouldn't be anywhere close to this severe at these levels.

[Aaronontheweb]

For running this test against a remote sharded actor in your scenario, do you just run two processes and target a entity id not hosted on the HTTP target node?

[carl-camilleri-uom]

Correct, basically the setup is as follows:
instance-1 : Windows VM running an instance of the code in the repo
instance-2 : Windows VM running an instance of the code in the repo
benchmark-machine : Makes calls to either instance-1 or instance-2

From benchmark-machine, curl http://instance-1:5000/5 returns akka://ping-pong-cluster-system/system/sharding/PingPongActor/13/5(pid=2372,hostname=instance-2, so we know that in this case entity id = 5 is hosted by the sharding system on instance-2 machine.

So hitting the endpoint http://instance-2:5000/5 constitutes a call to the local actor, whereas http://instance-1:5000/5 constitutes a call to the remote actor

[Aaronontheweb]

Thanks! I'll take a look at this and at the very least, resolve #3083

[Aaronontheweb]

We've been able to reproduce the 2.4k msg/s figure exactly in the benchmarks we created on #5209

I'm working on using Phobos to do some end-to-end tracing on the shard routing system to see where the most time is piling up. Did a read of the code last night and nothing obvious jumps out at me, but there is a drastic difference in remote vs. local shard performance that is clearly visible and consistent across different hardware profiles.

[Aaronontheweb]

First time a shard + entity is allocated, Phobos graph: