resteasy-reactive-client: Streaming JAX-RS client response while also accessing response headers

[otaconix]

Related to #50813: how can I stream the response body using a JAX-RS Client, while also accessing the response headers?

Context: I am trying to build a resource that fetches a (potentially large) file from another service, and send that document to clients in the resource's response. One of the things I want to achieve is to minimize resource utilization during this operation (so no slurping the entire response into memory).

My current experiment looks like this:

@GET
@Path("/streaming-response")
@Produces(MediaType.APPLICATION_OCTET_STREAM)
public Response getStreamingResponse(@QueryParam("url") URI url, @QueryParam("filename") String fileName) {
    if (fileName == null) {
        fileName = UUID.randomUUID() + ".txt";
    }

    Client client = ClientBuilder.newClient();
    InputStream upstreamResponse =
            client.target(url).request(MediaType.APPLICATION_OCTET_STREAM).get(InputStream.class);

    Response.ResponseBuilder response = Response.ok();
    response.header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename=" + fileName);
    response.header(HttpHeaders.CONTENT_LENGTH, null);

    return response.entity((StreamingOutput) output -> {
                byte[] buffer = new byte[streamingResponseBufferSize]; // 1 MiB buffer

                try {
                    int readBytes;
                    do {
                        readBytes = upstreamResponse.read(buffer);
                        output.write(buffer, 0, readBytes);
                        output.flush();
                    } while (readBytes >= 0);
                } finally {
                    client.close();
                }
            })
            .build();
}

This "works", as in I'm only using an 8KiB buffer per request to this endpoint, but I have no access to the response headers from the upstream service. Is there a way to combine this streaming approach with the ability to see the response headers?

EDIT: fixed issue where I'd write the full buffer even if fewer bytes were read from the InputStream.

[otaconix]

I'd really appreciate it if someone could help explain this to me. I've tried running Quarkus in a debugger to find out where InputStream's special-casing happens, because I'd already tried writing a MessageBodyReader for some:

record HeadersAndInputStream(MultivaluedMap<String, String> headers, InputStream bodyStream) {}

to no avail. I mean, it works, but it seems to consume the InputStream before I can get to it.

[sberyozkin]

CC @geoand @FroMage

[geoand]

Doesn't using Response work?

[otaconix]

It appears it does! I'm confused, as I was pretty sure I had a test setup where directly requesting an InputStream (without a Response.readEntity(InputStream.class)) would not result in an OutOfMemoryError, while using a Response would.

I know I've already mentioned what I'm trying to achieve, but basically I'd love to be able to use a fixed buffer to retrieve data from some upstream service, and use that same buffer to send the contents back downstream (to my "user", so to speak).

Using Response.readEntity(InputStream.class) seems to work, only then I run into issues whenever I get any backpressure. For example, when the downstream client (let's say: a curl invocation) is slow to read whatever data it is served, I still end up with OutOfMemoryErrors.

Again, it's time for me to get some sleep. I'll think it over some more, and perhaps come back with more questions later. In the meantime, thank you very much for your time and attention 🙏