Unexpected TCP FIN on HTTP/1.1 connections

[hiddenalpha]

Jetty version(s)
12.0.30

Jetty Environment
ee10 (I guess? At least mvn dependencies contain this string)

HTTP version
HTTP 1.1

Java version/vendor (use: java -version)
Where I ran the reproducer (windoze):
openjdk version "21.0.4" 2024-07-16 LTS
OpenJDK Runtime Environment Temurin-21.0.4+7 (build 21.0.4+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (build 21.0.4+7-LTS, mixed mode, sharing)

But originally discovered with (linux):
openjdk version "21.0.6" 2025-01-21 LTS
OpenJDK Runtime Environment Temurin-21.0.6+7 (build 21.0.6+7-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.6+7 (build 21.0.6+7-LTS, mixed mode, sharing)

OS type/version
Analyzed on windows 10 but discovered on on linux (debian-inside-docker-on-RHEL)

Description

Seems HTTP parsing gets messed up in Transfer-Encoding chunked mode, when the EOF-chunk of an incoming HTTP PUT message arrives a bit late.

Disclaimer: I'm not yet 100% sure if the issue is inside jetty. Maybe it also could be spring. But I have to start somewhere. Hopefully someone more familiar with the codebase could bisect the problem further?

How to reproduce?

Use a http client which sends multiple http/1.1 PUT requests with a body at least a few bytes in length on a single connection. Make sure that the EOF-chunk (0x30 0x0D 0x0A 0x0D 0x0A) gets fragmented in a later tcp packet. Then also ensure, that this packets only arrives as soon the server already has begun reading the http request body.

I did setup a minimal maven spring project reproducer-server.tar which is using jetty:
reproducer-server.tar.gz.b64:

H4sIAAAAAAACA+1Y60/jOBDvV/JXRP0E0tV59bGCtgvcwrISL1H27qTTaeUm0zZsEgfboaAT//uN
8+gDaLu93S2cLvMl8Xg8/mV+tmccwV2j8pPFRGk1GukT5ekzfbcalmWZltNylL7ZaDoVvVHZgCRC
Uo5TVv6fIpD/kPqR8Rb4dyzHrNcrytoyS/43yv8NvaPGG+C/YVVMy6zbrZL/zfPvjozX5L84//HN
tkv+X4X/ke95ENEgHv2w42B9/h27zP9vgH+4p2EcgLFx/ustq1ny/2b4P8qePeB3wImyXJv/ZlrX
vci/ZTt2kf9tp9GsmLbp1DH/myX/P11i6n6lQ9DdEZlhnuTM72maH8aMS53xIREx96PhgNMQxox/
JX3GJOmlyoM4DnyXSp9FeyuH0EQyl0UDf5hwyB0cYse3OhlDn/T9yCM0iphM7cllIs9oHKPd2kOv
QMhfWSQ5CwLg/2L4bYIeDpn3gOHS9l/8Hi1O+viuuwEVQp/bT/rfmraVdwvl09XvmO/paldu96Ry
9udfOuVDsYOmW1vPAk54Em3Pb9F0ml+yQXva1qOmPSpo81+qzYGZ6lNA+9OAblcNGvvVnQnKi/4N
uBig6CD2t/dnvr/o4XCrmhleDjLhkR7BOO/e3lHqwpkfSX3AmN7R6zZC3Xos8FZK2eT5z0GwhLsg
fvwNYP37n9Wwy/v/a/EvfHUeBGw4xJMk5iwGLn0Q38f/kvxvNq0J/82WpfhvOlgSlPl/A6KlaS4Y
1G9IxvtpxjvSf+wH0Ok9CAkhYYlcZClGbPyBSrj2Q+hInsAiQy83OmY8pLJzcrIbhrtCkF6vt8w3
vp7TFa6VXW+EWbswHtBALLW+HnGg3mpbDwY0CZTfU7iDoPP7wdW5tiRoRHWBG/ixAHIDUj4QkeVk
THTAsbwiJ1LGmG0jTIWYvTsfjg4/f3y1bBezkNyHwU+dY8X+dxq2Ndn/Tt1W/38dq6z/NyLt98i+
jutTqLVYtYhZ1SFymYeFX6f6+fq49q76vqu1MQ+kpR1aR6JTHeEa3jUwc9xBRCheIkagVr5xeXFm
1ImJXrCUy4x374U/GTAej8nYSU1tvPkZf5yd9nBsSGt+hERELmQDhb8rUv0py4rcb5hSX2SRKmp3
9S/mF5PcC6/a1bbaIfMg+C377m46vm3M6bAIbg85S+JPXvfl21HbKPrRlGKSHFBXYusKMFpe4sIn
2TZm9GiVB7qL0xGr1js/uOydXFy3jUKPJtmNDMPfvaG8bUybWtrJIZJoNsW26Jo1i24OXmZcUzY1
tfjxWKplfp+gncJ1SIO8m0NpFEgUqEmNkA7KlwrpJ37gkayqOMqXVDddUTh6mY1ykpHosjDGLMQJ
hwCogC4OXdCTYpoBgm0PYoiQNLdANlE8qOaTEKrjfC5m80FT3bXstH8apmmcbEVrazZQiGp+1uUg
VvG4kki8oT6HB/dukChAaRhmFVl7XRCrYUgW4r59hgSj8WTqxVBwz90yoXJqH7fAMgC5SS29z/ru
6lln2uIVOErLgmd77TswrJz+hVXxZL6ZZrF90r2Z7eggwRMoWzx5Iw/smutmUWSyEzrz/AJ9cA9u
Iqfrd0aTKxAJwzKuaGXNLof89xbCUO3C1pg1VothztmMIv9kY/rNxbtIY5aHSJscaN3yt0UppZRS
Sin/BfkHZG9gXQAoAAA=

Then when that server is running, I used this node.js client to trigger the undesired behavior. Contains a --help page. Make sure reproduce is present in the scripts file name to not confuse the primitive argv parser.
reproducer-client.js.gz.b64:

H4sIAAAAAAACA7VXbW/bNhD+7PyKqzA0cmPLdvfSzmk3uE2KdGiTIHHXD20HyNI5ViKTGknFCQL/
992RlCW7drcBmxMYEu94fO7t4fkwnJYiMZkUYfthby+RQhu4lMkNGngJCv8sM4VhINAE7citH3ot
vMtYp1AyQa0jfq1EubxqSLRJUalK5t52iq/QnMrFcZHM3mtSOooNRkIuDvf2atuXKFKSPSxUZnAI
Kw9gUk6B3GiRTmSFIa20D/eWy7X9F5jcfnN/r7fdQqVJ2DNhTjAvOGqthieyNH5fEOy1WgcBwGdR
PV0gaaZlgqpeq5/OChQaYhi/PgftMhCTnwoLpCCk+T1o8lvDyXh8bjOD2mgwEswMSaRum2aR3u5h
iguSkN+pjuAjUsoKTAzvmSDEkxz5MWULmQZyiy3ZjTqeY23MoenAIjMzchAo73FRqLIwBIoSZjJx
BUkuNabAcjkFStkMFUbbHeUY6uE2Wbc7k5SkF9qoX+rVS+sdgyVfhHMhgiOcxmXuYrA/ePos6tPf
YD/abreQiuyKct6we85rhLaO4LrV5/3n/V3mYjPbhHnOa97c+YdxlSS2VKCaSjXfAN2Li2wXXtrc
ncj0fvOQC2/UysgIF8W62VhQouN5Qen97fLstGH/4+jidAjHQpfKpl4KSmAmkrxMEZ5NMtONdZI9
gpGmBKZ4rSnleQ5GUQXarOva1ofxm+5zqolZlsxgIcs8hRzj1AKgNCkuD5gZU0AyK8UNCcUVhScT
VCIKQiNlbYvaaZ6Z7BbbkMR5UuYxV8h6YLgJmy0ohaMjYoiYGhppdxpyXXb8gdyZ9BpluhJfyjlS
qQsmJ6NKPHQKeawNU8KKdJocFNK5Lc8ZvrOt+cjIS0MscBW2N5FxRk7xzvhMeVBSHEmBFtVtrFa1
8RI8UXDBBAesG9naIre503uDaPB51dgn1BzDSs12ykEwrHZxMR8EtfJr1ysEaQg3iEU3zikKa3KD
wnTH9wWxIJnIs8QGvnetpaj1xioWeoqqeywSmZLLQxdhTGud1ZOFQs69ovKMXM7rUA1+asMO3cZ6
7wmcnsHx2RufyHmsbqj7mU9gwvXKqCHs333fh/5d/4i/RvVTG570LKbAZY5vC585H/S2Tzzz2nbJ
9pqZxrnmoiGA8/iGKIPbaIE24bbpa8yZSDmYUsEIXr0dw7vR+Pgissg0mnE2RyLKMMU8vsf0WE7f
Wx878LTfZwyrWtrUsPcNA7Bn0nlVBzIgbVurqqxEMgXQ9enisR6JoP9ZuKZqbYaCRJQImwwrdoUb
ijLPO8DfvLrcbEZfbL4HbZkzxDOBVRus+bUpe3AIKwjd+gOjCd8pnumIWyr/6IqB7t98aic3m9L3
JIXEOH9aTWah1ZBItEOHaaJV604rm4bwaFd1WA1OzCImogypTguZ5yRpAxE/XaFzhJSoCkLLiUoX
bUuVSpEdn6JmaTgQHfjhqYXfUmhKJfhx6QqA6k34kAhJL3XirS3mGIZiCS3cQnPdDZqjzvTkU2p8
r13MVnDW89WB0Nt+AYM2/AoDGPrTthSHKkVdFFWxESpB04nj8JCDF+e5XJzE+ZQHoaEl6A7ZWmtW
f/1b/ZnlwooISbXFDDhcceFXm2nCC2jSlCro8ByhZY4R19yEmjX0C+2vdvgjg06zyN2efocP+3oL
59nqf31DbW5cC9Q8zuwY3nKTKofsJZxNrunISGOcN9wOVvNOUHvO44p9o/uDVHi6sFJPsUPYfwhm
GV0Y1yUP4lySQmYag+W+22YzP2QSsnbYoaFteX7bUvVDR4ksXa+uIfSr8HMhxurqtjHx8yuTgVQh
sDgbkXAAh/zwwir7m4OWDg5o0TaXN0SqrPEpG3059E3J+9nA48fQU9WM3YuM7XNFHeiak6JKkypd
+9B7kl0JyTOQG3zRtswSyRdgg/YYuplpGqUZP/D7G0P/ITT6cds2SlK1bXVVe+DskYVeE4oVtx/8
b6OKBI/fnv4+eueue+dxd/CF5wKBSL8DbuO8REtv9qdYOKCH5Q48XB9NPHZW2I6n98enfvfnLw+D
zo/L76og+i3tfwGyXuDXf4qTKncNJ09Cu+Nmxf9n3KohvImpGlh2w6o0/lNk9vxvmauizHrVdndd
0D9zMFOOYxxHNPyLdmnH178AljkkbQYQAAA=

Which produces the following result for me:

reproducer-client.log
reproducer-server.log

tcp.pcapng.gz.b64:

H4sIAAAAAAACA9WVT2gcVRzHf/unoZ06/gmpior9uWjZxf0zf9bsMoo1JLERm3a1Wxq0UmZn3zaT
bOYNM283VqlGPOqhWoUKpSrSQ7tJqfTQHrQepIV4ECmIUOqhB+lFsF5DYX3zsmGXyVzEk294zG9m
3nzm+/2+mTeSLEvfAcD0C88+FuP7bq/FQQdVZ7O4hzj4isNIM/16BsepR9LV6QzapZyqlxRlagbT
izYfduDAZDGvZSABz8BoMVezGR6ynTpd9FFVMa3pU1omi7WW3ayjpo3qKn9WEjSYaC24luli+pDt
EX/W9OYzWMyP5hVMt8U+p+SOWvVGbbSkqaRUsjIgWqA50HsaNloS4jAChydI27bI4X2Vl4/spdSt
mdZ8Ap6GsbrpMuJhg3rY7J1H5pmNhm0hF8BaHtnGiUOctB0kAJd6DMtKWQG47194gp6mgDPTO56b
GIKTvw6zMq+DHud9EqDY+eQlWBp6TXmfWwn66r2dJy7dab4Y3LMU73YXSnxssvtbLJHYGosl4zMR
3KtR3E8Ft7DB3Xli9d7P6lt5zt69NNztjl2P5u4d4P7BuVleZ/vcdOek4Bohvbs5u1B5ENfe/HGd
EbD+HGB9e3OYrfJ6tc+61vksYFVOR7IexbULjwNUDlaxYLo2TlWrlYKaV2VpivrMQFUr5RW+qUYw
QbI0Th2HWMymjoHzhLg5s2m3iTjPiMNy1WMuMdB03aZtmcGwwpxPHVmqeqbjN4iXm3QsWredowZa
sy1nntRlSZbUsiy9m5q1U0ZqruUz9OkCQYfaPkkdl6Wex3Bu129G5Pa5yK0cmo8C93tL5Larn9sv
Ayz12ghb4fVKn3Wuc0bkdiGSxXPbfgRgIy7UFAX3vypLEybj/qstkkXtOdxH2/wKL1TN0FVDKeKe
6ep/DWssyKpBacooasd7PsLZHOR+NmXzpcgmH3oPbnFPF4Ns3jjbz6Y6wProxgjTea33WYXOV4K1
K5TNxY1stnkAsqQEcgUrrO+bGxH6vhbMbIS+y0LfmWh93/+1gwXrgdpn5TrnBevJSBbX9/AiwLo6
iNSXuLtjs76OYKZCni9z7m2h74u+vkHWI2sRrGXBeiqS9RBnnYpmqVGsFcHCkNfbnHdF6DoVvVY8
/6G+aa1Yvhqw9nuRLJ7b8t3/51rxNvcazm35B/F93x+agyuX7iy8VxkBeOfvdcYW3psDrA9+/4nF
4AkYpy3uw/PR9WjbrpM61o5hff0nG4etYiw9+7Gf6NXBfUleP9BjbeH1YAue8Q/iMlUNJAgAAA==

[joakime]

This is with HTTP/1.0?

I ask, as the token Keep-Alive doesn't exist in HTTP/1.1 or newer. (it's a token that only has meaning on HTTP/1.0, and was fully removed from newer HTTP specs)

[lorban]

If my understanding is correct, the following ExampleController.onApi() is called when the client sends a PUT request:

@RestController
class ExampleController {
	@PutMapping("/api")
	public Object onApi(@RequestBody Object reqBody) {
		return new Object(){
			public int foo = 42;
		};
	}
}

There are two things to take in consideration:

• the client sends its data via chunked encoding
• the server's controller does not read the uploaded data

When the above occurs, Jetty has to figure out what to do with the uploaded data as it cannot read the next request until that is consumed. So what it does is the following: it reads from the socket until it finds the eof chunk or the socket is empty, whichever happens first.

In the case the eof chunk could be read, Jetty then knows the next bytes are going to be the next request so it can keep using the connection. In the other case when the eof chunk could not be read, Jetty could be waiting potentially forever for the missing bytes to arrive so it decides to add the "Connection: close" header to the response and close the connection from its side.

Your description of the problem (you need EOF-chunk fragmented in a later tcp packet to reproduce) matches the above description: when Jetty manages to read the eof chunk, the connection is kept alive otherwise it gets closed.

The solution to this problem is to read all uploaded data from your controller.

[hiddenalpha]

If my understanding is correct, the following ExampleController.onApi() is called when the client sends a PUT request:

Correct.

the server's controller does not read the uploaded data

What about the @RequestBody Object reqBody argument? How should it be provided to the controller without reading the request body? Using a debugger stepping through attached example server, one can also see that the request json in fact gets read and parsed.

the client sends its data via chunked encoding

Yes, it does. And so far it looks like this is essential to trigger this behavior. Especially the 0\r\n\r\n has to be in a later TCP segment, which arrives just a bit later. (Edit: Seems irrelevant, can be reproduced with a json followed by a whitespace in a delayed TCP segment. So maybe the json parser is not reading until EOF, but stops reading early or something like that?)

I assume you've had a look at the server logs (towards the end). To me, looks like the server is not expecting that late data. So far I found no explanation how "not reading the request body" could trigger this exception:

13:31:05.081 DEBUG org.eclipse.jetty.server.internal.HttpConnection - caught exception HttpConnection@49c1983d::SocketChannelEndPoint@71feeab2[{l=/127.0.0.1:8080,r=/127.0.0.1:51708,OPEN,fill=-,flush=-,to=0/30000}{io=0/0,kio=0,kro=1}]->[HttpConnection@49c1983d[p=HttpParser{s=CLOSE,24 of -1},g=HttpGenerator@1012f02{s=START}]=>HttpChannelState@93322fd{handling=null, handled=false, send=SENDING, completed=false, request=null}] HttpChannelState@93322fd{handling=null, handled=false, send=SENDING, completed=false, request=null}
org.eclipse.jetty.io.RuntimeIOException: Parser is terminated
	at org.eclipse.jetty.server.internal.HttpConnection.parseRequestBuffer(HttpConnection.java:582)
	at org.eclipse.jetty.server.internal.HttpConnection.onFillable(HttpConnection.java:384)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:322)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:492)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.epcRunTask(AdaptiveExecutionStrategy.java:428)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:401)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:255)
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:196)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:981)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1211)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1166)
	at java.base/java.lang.Thread.run(Thread.java:1583)

In the other case when the eof chunk could not be read, Jetty could be waiting potentially forever for the missing bytes to arrive so it decides to add the "Connection: close" header to the response and close the connection from its side.

No, jetty is NOT waiting. It already starts sending the response even before the full request even reached the server.
No, it does not send this header. See provided tcpdump.

[hiddenalpha]

This is with HTTP/1.0?

Nope, 1.1. As already said in the description. Have a look at either the reproducer, which writes HTTP/1.1 in the http request line. Alternatively inspect the attached pcap to confirm.

Feel free to remove that header from the request. Makes no difference if that header is there or not. Behavior stays the same.

Edit: Just realized I had it in the issue title too. Changed that to HTTP/1.1 now, seems to make more sense.

[hiddenalpha]

If folks do not consider this a bug, honestly I do not care. I know how to "solve" this using ugly workarounds. My goal is to have this reported (including reproducer, logs and tcpdump).

I also recommend to read this part of my description again:

Disclaimer: I'm not yet 100% sure if the issue is inside jetty. Maybe it also could be spring. But I have to start somewhere. Hopefully someone more familiar with the codebase could bisect the problem further?

[lorban]

So maybe the json parser is not reading until EOF, but stops reading early or something like that?

I very much suspect the json parser is reading until the last } character leaving the eof chunk unread. This isn't a new problem, see #11460 (comment) for another instance of a similar issue, and how to configure Jackson to read until EOF by enabling FAIL_ON_TRAILING_TOKENS.

No, jetty is NOT waiting. It already starts sending the response even before the full request even reached the server.
No, it does not send this header. See provided tcpdump.

I did not say Jetty is waiting, I just listed the potential options and pointed out that waiting isn't a reasonable one. If the "Connection: close" header wasn't sent, that's probably because the server already committed the response.

Looking at your sniffer trace more closely, this seems to be what's happening. Here is the TCP stream that I annotated with <<< >>> markers to show which side is speaking:

<<<client speaks>>>
PUT /api HTTP/1.1
Host: 127.0.0.1:8080
Connection: keep-alive
Content-Type: application/json
Transfer-Encoding: chunked

18
{"hi":"just some noise"}
<<<server speaks>>>
HTTP/1.1 200 OK
Date: Tue, 25 Nov 2025 12:31:04 GMT
Content-Type: application/json
Transfer-Encoding: chunked

A
{"foo":42}
0

<<<client speaks>>>
0

PUT /api HTTP/1.1
Host: 127.0.0.1:8080
Connection: keep-alive
Content-Type: application/json
Transfer-Encoding: chunked

18
{"hi":"just some noise"}

and it shows that the server responds before reading all data sent by the client which prevents it from sending the connection:close header, but the connection is closed nevertheless: the RuntimeIOException: Parser is terminated exception is a symptom of the parser being fed with data after the connection was closed.

So far, everything points to a problem with the json parser not fully reading the data so we're not inclined to believe that there is a bug in Jetty. Our recommendation is to either make sure the PUT data is consumed until EOF, or avoid using chunked encoding.

[hiddenalpha]

This isn't a new problem, see #11460 (comment) for another instance of a similar issue

That sounds pretty much like there's an issue within the parsing inside the springframework then. So probably its not jetty but springframeworks parsing then?

[lorban]

Yes, Spring not enabling FAIL_ON_TRAILING_TOKENS is the most probable cause.

[joakime]

Your application.yml could also have ...

spring:
  jackson:
    deserialization:
      FAIL_ON_TRAILING_TOKENS: true

[joakime]

Or you can specify the ObjectMapper bean configuration in code ...

@Bean
@Primary
public ObjectMapper mapper() {
    return new ObjectMapper()
        .findAndRegisterModules().registerModule(attributesModule()).registerModule(dateSerializationModule())
        .configure(DeserializationFeature.FAIL_ON_TRAILING_TOKENS, true);
}