there are lot of tcp connections on 127.0.0.1,that seems it was a tcp connections leak.

[Musknine]

I am using dolphinscheduler3.0.0 of my ds cluster,it seems Connections increase as time goes by,end of the cluster was crashed.
now,the cluster crashes every five days.
i took this picture at 2024-5-7 17:19:

i took this picture at 2024-5-8 15:17:

from below picture，I found these connections is between workerServer and alertServer:

this is one of the connections and what packets are on the connection. workerServer push "{"items":{}} heart_beat" to alertServer every 6 seconds.

I found the heart_beat packet was trigged from io.netty.handler.timeout.IdleStateHandler.ReaderIdleTimeoutTask run fuction. but i didn't know why there are so many that task(I am not familiar with dolphineshcedular codebase). Could someone help me to deal with this problem? thanks advance

[Musknine]

closeChannel function couldn't close the channel anymore, and the channels lose the quote of the channel forever(there is a wrong log of channel.isdone as channel.close is a async operation,if i want to get this,i need to add listenner).

this is my log info:

i saw this tcp connection are there for long time:

accoding to this picture's code one step by step , there is a concurrency problem that could lead the tcp connection leak.