OOM in tests (Part 3)

[mschorsch]

Describe the bug

As stated in #42355 (comment), we continue to receive an OOM in our tests. I was able to create a reproducer for this.

Seems there is another memory/classloader leak.

Expected behavior

No response

Actual behavior

No response

How to Reproduce?

oom-reproducer3.zip

./gradlew test --console=plain

Output of uname -a or ver

Linux

Output of java -version

Java 21

Quarkus version or git rev

Quarkus 3.13.2

Build tool (ie. output of mvnw --version or gradlew --version)

Gradle 8.9

Additional information

No response

[gsmet]

That's awesome, thanks. I will have a closer look soon.

[gsmet]

I have been chasing them for a while but it all depends on the extensions being around. I hope we'll be able to get to a more sustainable situation thanks to you.

[gsmet]

OK so I can reproduce it and this is a bit different. The issue is mostly related to Testcontainers threads and things are not leaking like crazy as they use to do (at least in my experiments).

Now, our CL objects are very large and they also keep open a gazillion of zip file systems, which explains the problem (I think).

[mschorsch]

Unfortunately, it is not easy to create a reproducer. In our real project I can easily produce an OOM (without testcontainers) unfortunately I could not extract a simple reproducer so far. I was hoping that this reproducer would at least partially solve the problem.

Overall, one would expect that if each test class is executed individually and sequentially, the memory requirements would remain the same (at least after a full GC). This is not the case here.

[gsmet]

I pushed a first PR here ^.

It improves things a bit but doesn’t really fix the reproducer.

[gsmet]

Do you think we could maybe organize a Google Meet call together?

If you can get a heap dump and install Eclipse MAT, I could show you how to get some info and we could make progress without you sending me the heap dump?

My guess is that it’s all going to be about class loaders so I won’t see any privileged info on your screen.

Let me know if that could work for you. My email is open at gsmet at redhat dot com.

[mschorsch]

Hi @gsmet , sorry for the delay. Unfortunately this is currently not possible as I am on vacation. I will be back in September.

Nevertheless, thank you for your work and the first improvements in #42492 👍

[mschorsch]

@gsmet A positive message, I have checked your changes from #42492 again against our real application and I can confirm that the changes prevent an OOM from occurring 👍 .

[gsmet]

Yeah, it still fails with your reproducer 3 though. There are several other things to fix.

#42525 should help more and then there's the issue of Testcontainers leaking things like crazy that I'm trying to solve.