LogbookClientHandler leads OutOfMemoryError on large Flux<DataBuffer> response body

[d-a-gerashenko]

org.zalando.logbook.netty.LogbookClientHandler tries to allocate ByteBuf with size of Content-Length and it leads to OutOfMemoryError in case of large (more than JVM memory limit) body size which is transferred via "Flux<DataBuffer>".

Steps to Reproduce

This method leads to OutOfMemoryError:

...
    public long test() throws Exception {
        Flux<DataBuffer> dataBufferFlux = WebClient.builder()
            .clientConnector(
                new ReactorClientHttpConnector(
                    HttpClient
                        .create()
                        .doOnConnected(conn -> {
                            //* <-- remove slash to make it work
                            conn
                                .addHandlerLast(
                                    new LogbookClientHandler(
                                        Logbook.builder()
                                            .condition(Conditions.requestTo("/**"))
                                            .requestFilter(RequestFilters.replaceBody(message -> ""))
                                            .responseFilter(ResponseFilters.replaceBody(message -> ""))
                                            .requestFilter(RequestFilters.replaceBody(BodyReplacers.binary()))
                                            .responseFilter(ResponseFilters.replaceBody(BodyReplacers.binary()))
                                            .requestFilter(RequestFilters.replaceBody(BodyReplacers.stream()))
                                            .responseFilter(ResponseFilters.replaceBody(BodyReplacers.stream()))
                                            .bodyFilter((contentType, body) -> "***")
                                            .build()
                                    )
                                );
                            //*/
                        })
                )
            )
            .build()
            .get()
            .uri("https://mirror.team-host.ru/ubuntu-cdimage/24.04.1/ubuntu-24.04.1-desktop-amd64.iso")
            .retrieve()
            .bodyToFlux(DataBuffer.class);

        Path path = Paths.get("large.dat");
        DataBufferUtils.write(dataBufferFlux, path)
            .block();

        return Files.size(path);
    }
...

Environment

Windows 11.
pom.xml

...
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.2.1</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <properties>
        <java.version>17</java.version>
        <maven.compiler.source>${java.version}</maven.compiler.source>
        <maven.compiler.target>${java.version}</maven.compiler.target>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.zalando</groupId>
                <artifactId>logbook-bom</artifactId>
                <version>3.10.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-tomcat</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-webflux</artifactId>
        </dependency>
        <dependency>
            <groupId>org.zalando</groupId>
            <artifactId>logbook-spring-boot-webflux-autoconfigure</artifactId>
        </dependency>
    </dependencies>
...

[d-a-gerashenko]

Perhaps is happens when there is a content-length header in streaming response.
https://github.com/zalando/logbook/blob/main/logbook-netty/src/main/java/org/zalando/logbook/netty/Offering.java#L24

[d-a-gerashenko]

This fixed my OutOfMemoryError problem:

Logbook.builder()
    .strategy(new WithoutBodyStrategy())
    .build()

But I'm not sure if it's the right decision to use a buffer size equal to the length of the content because it could be too long. If everything is ok, please close this issue.

[ChristianLohmann]

Hi, @d-a-gerashenko !
In your specific case you can write your own Strategy and add it to the Logbook builder section:

class WithoutIsoContentTypeStrategy implements Strategy {
        @Override
        public HttpResponse process(final HttpRequest request, final HttpResponse response) throws IOException {
            if(response.getContentType().equals("application/x-iso9660-image")) {
                return response.withoutBody();
            }
            return response.withBody();
        }
    }

Logbook.builder().strategy(new WithoutIsoContentTypeStrategy()).build()

That is, if that server returns this particular Content-Type. It should work analogous to that with application/octet-stream.

[d-a-gerashenko]

Hi, @ChristianLohmann
*.iso file was an example of a large file. I just pointed out the problem for any large file.
And I'm not sure how common is that type of use (it's about large files) because I faced it out during implementing an API-gateway which it not too common case I guess.

But it could be a solution to add a condition to use WithoutBodyStrategy in case of large files.

[d-a-gerashenko]

May be I'm wrong but in case of application/octet-stream everything will by alright if we have no content-length because in this case we get 2048 buffer size according to https://github.com/zalando/logbook/blob/main/logbook-netty/src/main/java/org/zalando/logbook/netty/Offering.java#L24

May be we just need an upper boundary for contentLength https://github.com/zalando/logbook/blob/main/logbook-netty/src/main/java/org/zalando/logbook/netty/Offering.java#L24