consumeEachBufferRange completes successfully on connection abort

[fluidsonic]

Ktor Version and Engine Used
Ktor Client 1.3.2 with OkHttpClient 4.5.0 on Android 21

Describe the bug
When the connection dies during a streaming download, consumeEachBufferRange erroneously completes successfully.

To Reproduce

1. I'm streaming a download using consumeEachBufferRange.
2. When I turn on Airplane Mode during the download, the visitor I've passed to consumeEachBufferRange will be called with a buffer object that has 0 remaining bytes and last set to true. The latter, to my knowledge, indicates download completion.
3. No IOException is thrown which further indicates that the download has completed.
4. The download (consumeEachBufferRange invocations) isn't even close to the number of bytes indicated in the Content-Length header.

Note that it doesn't happen 100% of the time.
Sometimes I do get a shorter packet (e.g. 64 bytes instead of a typical 4088 byte packet) with last = false plus a SocketTimeoutException afterwards.

Expected behavior

• consumeEachBufferRange shouldn't be called with a 0 byte buffer.
• consumeEachBufferRange shouldn't be called with last = true for an aborted connection.
• consumeEachBufferRange should throw an IOException for an aborted connection.

[dmitrievanthony]

Hi @fluidsonic,

I'm trying to reproduce the issue, but I always get java.net.ProtocolException: unexpected end of stream so far. Do you have some reproduced that I could use?

PS: I also tested it on Android where I got Exception: java.net.SocketException: Software caused connection abort.

[fluidsonic]

I've tested on a Google Pixel 2 with Android 10 now (EDGE connection) and have the same problem there.

It seems to be related to the workaround for issue #1708 as I cannot reproduce it without retryOnConnectionFailure(true).

Test code:

HttpClient(OkHttp) {
   engine {
      config {
         retryOnConnectionFailure(true) // https://github.com/ktorio/ktor/issues/1708
      }
   }
}.use { httpClient ->
   httpClient
      .get<HttpStatement>("http://ipv4.download.thinkbroadband.com/5MB.zip")
      .execute { response ->
         val contentLength = response.contentLength()
         println("Response: ${response.status} with total length of $contentLength bytes.")

         var receivedBytes = 0L

         response.content.consumeEachBufferRange { buffer, last ->
            receivedBytes += buffer.remaining()
            println("Received ${buffer.remaining()} bytes ($receivedBytes of $contentLength total) (last = $last)")

            buffer.position(buffer.limit())

            true
         }

         println("Completed after receiving $receivedBytes of $contentLength.")
      }
}

Output:

Response: 200 OK with total length of 5242880 bytes.
Received 4088 bytes (4088 of 5242880 total) (last = false)
Received 4088 bytes (8176 of 5242880 total) (last = false)
Received 3084 bytes (11260 of 5242880 total) (last = true)
Completed after receiving 11260 of 5242880.

[dmitrievanthony]

Thanks, @fluidsonic, I managed to reproduce it with retryOnConnectionFailure(true).

[dmitrievanthony]

Well, I spent time trying to understand what's the reason and finally realized that the issue is not reproduced using the current ktor master branch, only version 1.3.2.

[fluidsonic]

That's a good thing, I guess?

Is this bug actually unit-testable?
I have no idea how to simulate a network shutdown which triggers that issue.

[dmitrievanthony]

Hi @fluidsonic,

I'm also not sure how to simulate this. An additional issue is that the behavior we'd like to test is reproduced only on Android.

[oleg-larshin]

Please check the following ticket on YouTrack for follow-ups to this issue. GitHub issues will be closed in the coming weeks.

[paderick]

This issue is still persistent on ktor 1.5.2

Any solutions for it?