high-load http requests cause hanging tcp connection in close wait state

[so-schen]

Hi,

I have http server running using ntex, and I can see high number of hanging tcp connection
there are hundreds of them..
Note that <ip> is from same ip that making the http requests, I just hide the actual ip with <ip>.

I wonder if there is issue in handling tcp connection close properly in ntex.

State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
CLOSE-WAIT 1 0 172.31.0.9:26000 <ip>:59050
CLOSE-WAIT 1 0 172.31.0.9:26000 <ip>:57932
CLOSE-WAIT 1 0 172.31.0.9:26000 <ip>:54710
CLOSE-WAIT 1 0 172.31.0.9:26000 <ip>:58766
CLOSE-WAIT 966 0 172.31.0.9:26000 <ip>:34230
.....

[fafhrd91]

ntex tries to close connection gracefully, it takes some time. also if your app processing request and does not ready/write anything, socket could be kept open.

is number of socket keep growing or is it steady?

[fafhrd91]

what runtime do you use?

[so-schen]

@fafhrd91, it is growing along with time
I use ntex tokio runtime
in about 2h test, I see ~950 connection with CLOSE WAIT there

[so-schen]

I can see from the output, the Recv queue mostly have 1 byte left

[fafhrd91]

could you try ntex-tokio v0.5.1

[gftea]

OK I WILL do a cargo update.

[so-schen]

upgrade to 0.5.1 does not help much, also once it gets into high load, it seems can not recover and take any requests

error return
Query transaction status failed: Request Error: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27000), path: "/rpc/v1/transactions/19c62f80407af594bcd199d4b4a499f0fe9c8358d8ac4480658aa30daec45661", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 49, kind: AddrNotAvailable, message: "Can't assign requested address" })) } Query transaction status failed: Request Error: reqwest::Error { kind: Request, url: Url { scheme: "http", cannot_be_a_base: false, username: "", password: None, host: Some(Domain("localhost")), port: Some(27000), path: "/rpc/v1/transactions/7ad47ffae0eca398bc954a5ba3d375846d4395997230a6da0e4e3877ed63a2f6", query: None, fragment: None }, source: hyper::Error(Connect, ConnectError("tcp connect error", Os { code: 49, kind: AddrNotAvailable, message: "Can't assign requested address" })) }

[fafhrd91]

how many concurrent requests do you run?

[so-schen]

10K, with 20 workers on 10 core M2 machine

[fafhrd91]

are you sure that your handler completes request processing? ntex holds socket while handler is in process.

you also beed to increase number of file descriptors on osx, it is not very high by default