Improve HTTPServer performance

[mneumann]

A hello world server with Rust (warp) gives:

> wrk -c 1 -t 1 -d 10s --latency http://127.0.0.1:8080/
Running 10s test @ http://127.0.0.1:8080/
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    36.33us    8.49us   0.88ms   95.58%
    Req/Sec    26.40k     2.12k   27.87k    82.18%
  Latency Distribution
     50%   34.00us
     75%   36.00us
     90%   43.00us
     99%   53.00us
  265337 requests in 10.10s, 32.39MB read
Requests/sec:  26272.04
Transfer/sec:      3.21MB

While the same benchmark with the Savi HTTP server example from examples/http:

> wrk -c 1 -t 1 -d 10s --latency http://127.0.0.1:8080/
Running 10s test @ http://127.0.0.1:8080/
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    98.77us    1.27ms  25.76ms   99.76%
    Req/Sec     4.01k     0.00     4.01k   100.00%
  Latency Distribution
     50%   35.00us
     75%   39.00us
     90%   43.00us
     99%   57.00us
  409 requests in 10.13s, 16.38KB read
Requests/sec:     40.37
Transfer/sec:      1.62KB

Figure out where the problem is. Note that Savi can sustain 40k req/sec when running wrk with 1000 connections. But there clearly is an issue with the throughput of a single connection. Tested on FreeBSD.

[jemc]

Probably the most major issue is the fact that I believe we currently compile all programs without any LLVM optimizations applied. I'd expect the throughput to go way up once we have LLVM optimizations enabled.

[jemc]

Didn't look too much into it yet, but when I ran locally with the single-threaded wrk command you used, and I watched the HTTP server CPU load with htop, it was below 0.0% pretty much the whole time (in all 8 scheduler threads). So whatever the bottleneck is, it's not an excess of CPU instructions.

I'm not entirely sure how the wrk command does load, but it looks like the fact that we're in single-connection / single-thread mode for wrk means that it won't parallelize any requests, and it's not issuing a new request until the last response comes in - so this benchmark is really just measuring latency of a ping pong back and forth between the two processes. I see that in your output the latencies of the two servers were very similar in their latency distributions, even up to the 99th percentile, but the Savi server had a much "longer tail" above the 99th percentile, with a max of ~26ms for the slowest request. If I had to guess what is happening there, the long tail in Savi is likely coming from the few requests where we do garbage collection in between behaviors, causing it to take longer for us to pick up the next request and return it.

So that said, I'm not sure the single-connection / single-thread benchmark is the one I should care most about. Savi (and Pony) are really meant to shine in highly concurrent workloads, so I'd prefer to focus on improving benchmarks for higher numbers of connections.

I get similar results as you when I use 1 connection / 1 wrk thread - 4k throughput (though I have a rather different latency distribution).

wrk -c 1 -t 1 -d 10s --latency http://localhost:8080

Running 10s test @ http://localhost:8080
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    48.99us   32.53us 299.00us   85.82%
    Req/Sec     4.09k     0.00     4.09k   100.00%
  Latency Distribution
     50%   43.00us
     75%   58.00us
     90%   90.00us
     99%  160.00us
  409 requests in 10.02s, 15.18KB read
Requests/sec:     40.82
Transfer/sec:      1.51KB

But when I use 1000 connections / 4 wrk threads, I get 177k avg throughput.

$ wrk -c 1000 -t 1 -d 10s --latency http://localhost:8080

Running 10s test @ http://localhost:8080
  1 threads and 1000 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     2.72ms  403.25us   6.28ms   78.25%
    Req/Sec   177.20k    16.79k  190.00k    86.96%
  Latency Distribution
     50%    2.67ms
     75%    2.89ms
     90%    3.15ms
     99%    4.20ms
  409000 requests in 10.06s, 14.82MB read
Requests/sec:  40664.19
Transfer/sec:      1.47MB

Interestingly, if I go any further than that in the number of connections I tend to hit some bug that results in some kind of infinite loop that I need to spend more time troubleshooting.

[jemc]

Created a new ticket (savi-lang/TCP#2) to track the infinite loop bug (which I mentioned at the end of my comment above):