Client gets ClosedSessionException when using HTTP/1 #6225
Description
Hello, we use 1.32.5 Armeria HTTP client
The most relevant settings to this issue are:
maxNumEventLoopsPerHttp1Endpoint=8
idleTimeout=5 minutes
writeTimeout=30 seconds
responseTimeout=30 seconds
transportType=NIO
From time to time, the ClosedSessionException is raised, which is most frequent when the load we have is at its peak, but it also happens when the load is small, too, just less frequently.
The interesting thing is that we analysed TCP dumps, and for cases where we get this error, the whole response was actually sent, and only then did the server close the connection.
We tried tuning -Dcom.linecorp.armeria.numCommonWorkers with different values, tried tweaking other settings and updating netty to 4.2.0.Final, also played a bit with maxNumEventLoopsPerHttp1Endpoint value with no success, overall nothing we tried till now seems to help.
From the stack trace, I see it is related to fireChannelInactive, but I lack an understanding of Armeria internals to understand why this can be happening. Overall, CPU usage is not at its peak, but we still get this.
Any recommendations on what to look into or analyse to get to know what could be causing this?
com.linecorp.armeria.common.ClosedSessionException: null
at com.linecorp.armeria.common.ClosedSessionException.get(ClosedSessionException.java:35)
at com.linecorp.armeria.internal.client.ClosedStreamExceptionUtil.newClosedSessionException(ClosedStreamExceptionUtil.java:51)
at com.linecorp.armeria.internal.client.ClosedStreamExceptionUtil.newClosedSessionException(ClosedStreamExceptionUtil.java:45)
at com.linecorp.armeria.client.HttpSessionHandler.channelInactive(HttpSessionHandler.java:557)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
at com.linecorp.armeria.client.Http1ResponseDecoder.channelInactive(Http1ResponseDecoder.java:157)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:305)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelInactive(CombinedChannelDuplexHandler.java:418)
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:377)
at io.netty.handler.codec.http.HttpClientCodec$Decoder.channelInactive(HttpClientCodec.java:410)
at io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:221)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
at io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:206)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:377)
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1192)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:274)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1352)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:301)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:281)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:850)
at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:811)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:1583)