About request distribution across workers

[papa99do]

Hello Granian team,

Thanks for implementing such a performant server for ASGI apps. I'm in the process of switching from uvicorn due to an issue of uneven distribution of requests when running with multiple workers. (A similar issue is discussed in this ticket: encode/uvicorn#2467). The PR to solve this issue is still in draft status.

I was wondering if Granian is free of this issue. I checked the code, and the reuse_port is only set in WSGI implementation here:

granian/src/tcp.rs

Line 43 in 5074d01

socket.set_reuse_port(true)?;

. But not in ASGI implementation. I've tested it briefly (with a FastAPI app) and the result shows that the load distribution is not even across multiple workers.

In my test, the work load is CPU-bound, and the GIL contention is a major issue that restricts throughput if the load is uneven. I have to set the backpressure to 1 to enforce a balanced distribution across workers under high load, but this is apparently a hack not a solution.

So is there any config I can use to ensure the balanced distribution of incoming requests across all workers? Thanks a lot.

[gi0baro]

I'm in the process of switching from uvicorn due to an issue of uneven distribution of requests when running with multiple workers. (A similar issue is discussed in this ticket: encode/uvicorn#2467). The PR to solve this issue is still in draft status.

That issues appears to exist only when you pass an external file-descriptor to Uvicorn, leaving the accept loop to an external supervisor, which is not an available feature on Granian. So if that's your use-case, it cannot be achieved with Granian.

I checked the code, and the reuse_port is only set in WSGI implementation here:

I'm not sure why you're stating that; the code you mentioned is used on all protocols, not on WSGI only.

I've tested it briefly (with a FastAPI app) and the result shows that the load distribution is not even across multiple workers.
In my test, the work load is CPU-bound, and the GIL contention is a major issue that restricts throughput if the load is uneven. I have to set the backpressure to 1 to enforce a balanced distribution across workers under high load, but this is apparently a hack not a solution.

The load balancing in connections between workers is performed by the Linux kernel (which is the only effective OS to use SO_REUSEPORT), it's not something done by Granian and thus there's nothing Granian can do about it.
Given what you said, and assuming by CPU-bound you mean you're blocking the Python event-loop, there's nothing Granian can do in order to avoid that, as using ASGI for such scenarios is just bad. If you need to do a lot of blocking operations with ASGI, then you should run those operations in a ProcessPoolExecutor (https://docs.python.org/3/library/asyncio-eventloop.html#executing-code-in-thread-or-process-pools) and scale the pool workers, not the server ones.
Setting backpressure to 1 gives you better results just because you're instructing Granian to wait for the whole request to complete before accepting the next connection; but again, if you're CPU bound there's no point in spawning more workers, as you won't get any additional throughput.

[papa99do]

@gi0baro , thanks for the detailed explanation.

That issues appears to exist only when you pass an external file-descriptor to Uvicorn

The fix is not for the external fd, but for socket. Please see encode/uvicorn#2472 for details. However, I didn't manage to get this fix working in my test. Instead, an alternative solution worked for me. encode/uvicorn@f2b1037.

assuming by CPU-bound you mean you're blocking the Python event-loop

It's not blocking event loop since I'm not using async functions, just normal function, so they are all scheduled to run in threads. But these work load holds GIL so basically blocks the whole process/worker. That's the reason I need to use multiple workers.

f you need to do a lot of blocking operations with ASGI, then you should run those operations in a ProcessPoolExecutor

This is a good suggestion. In my case, most of the load comes from json serde and pydantic validation(parsing large response from a data store, convert it to pydantic model, and convert to some other pydantic model, finally generate a json response to return to client). It is not very convenient to shed the load to a dedicated pool of processes. Multiple workers are pool of processes as well, and they handle the load end to end, which seems to be a better solution for my case.

The load balancing in connections between workers is performed by the Linux kernel, it's not something done by Granian
and thus there's nothing Granian can do about it.

Uvicorn's fix seems to make the load distribution more even across workers, at least in my test. I also tested the OS I used with a short python script and verfied that the SO_REUSEPORT works as expected, my request can be routed randomly to multiple processes bound to the same port. But when running Granian with default config, most of the load are distributed to only 1-2 workers/processes. I'm not sure why. I'll dig deeper and will also keep you updated. Thanks again.