Output forward connections are in CLOSE_WAIT state #4618
Comments
|
Hi, Can anyone please help me out with the issue that I am facing ? |
|
Looks like we need more information to investigate this.
|
daipom
added
waiting-for-user
|
@daipom Thanks for responding. I see the below logs along with the logs that I have already shared. I am not sure if these logs will help in analyzing better. Please let me know if you are looking for some specific keyword in the logs, I can try to get the output of those logs. |
|
Thanks! One more question.
Why does the server send reset packet? Could you attach server-side ( |
|
@daipom, I do not have access to the server-side logs. I was just checking the output plugin documentation, If we make the tcp connection as long-lived/persistent i.e., my setting Could this be the reason why fluentd is not able to clean-up connections in close_wait state. |
No,
Here are the documents for plugin users. |
|
@daipom, ok sure, But I don't see CLOSE_WAIT connections getting closed. Am I missing any configuration that should be added? |
|
The However, the problem is that I can't reproduce this issue. I found out one thing. On non-Windows, However, when setting linger_timeout to <source>
@type forward
<transport tcp>
linger_timeout 1
</transport>
</source>However, in my testing, this was not a problem. |
|
At least, the close_wait socket should be closed after I need more information to reproduce the phenomenon... |
daipom
added
moreinfo
|
If you can, please try a more recent version. |
|
This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days |
|
Hi, Expected Behavior: Since keepalive is enabled and keepalive_timeout is set to 300 seconds, the expectation is that the tcp connections stays persistent and just inactive or CLOSE_WAIT connections would be checked and closed at the 300-second mark. If all sockets are closed, it results in the loss of in-transit data, requiring new connections to be established. Could you please advise on how I can limit the cleanup to only inactive or CLOSE_WAIT sockets? |
|
Maybe, this issue will be solved by #3711 |
ashie
added
waiting-for-user
|
Thanks @Watson1978, @daipom, I tested with Fluentd version 1.16.6 and observed that the available/active sockets are not being closed, which is a good. However, I suspect there may be a bug in Fluentd's handling of CLOSE_WAIT connections. When the server closes the connection (e.g., by calling socket.close), Fluentd receives a FIN packet but does not respond with its own FIN packet to gracefully close the connection. As a result, the socket remains in a CLOSE_WAIT state on the Fluentd side. Ideally, we do not want to rely on the keepalive_timeout to manage the closure of the socket; instead, Fluentd should close the socket on its end when the server closes it on its end. Please note, I have observed this when keepalive is set to true. |
|
Can anyone please help me out resolve the close_wait connection. Please let me know if any support is required to reproduce the issue. We can try to reproduce the issue together. Everytime server executes socket.close() this issue is seen. |
Is this reproducible with Fluentd 1.16.6? |
|
@daipom, Yes. It is reproducible with Fluentd 1.16.6 |
|
Is there any update on this issue or Can anyone help be resolve this issue? @daipom, were you able to reproduce the issue? |
Describe the bug
I have fluentd deployed as a deamonset in kubernetes cluster which is sending logs using output forward plugin. I have shared the config below. In the netstat output of fluentd, I see alot of close_wait connections. How can I resolve this ?
tcp 1 0 fluentd-operator-:<port> 1.1.1.1:<port> CLOSE_WAITTo Reproduce
This issue is seen when the remote side of the connection has closed the connection.
Expected behavior
Ideally fluentd should handle closing the connection and relive the socket.
Your Environment
Your Configuration
Your Error Log
Additional context
No response
The text was updated successfully, but these errors were encountered: