Skip to content

Lots of open kafka-producer-network-thread after RefreshRemoteApplicationEvent #3116

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
LysenkoPA opened this issue May 20, 2025 · 0 comments

Comments

@LysenkoPA
Copy link

Description
I've built a Spring Cloud Config Server integrated with Spring Cloud Bus via Kafka for dynamic property refreshing.
I've another application that consumes these properties and refreshes them dynamically.

Endpoint /monitor in Spring Cloud Config Server is used to update the service properties.
The service receives RefreshRemoteApplicationEvent in springCloudBus kafka topic.

After several properties updates, we started getting the error
java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached

I noticed that a lot of kafka-producer-network-thread threads remain open and their number grows after each update of the properties.
I see this on both an Spring Cloud Config Server and a configurable service.

I suspect this is due to improper closing of Kafka connections after getting RefreshRemoteApplicationEvent
24c1abe#diff-5e17c4149c35b951bd7e88675c6fab6589366819948092156fd0b70573f25632R370
During connection cache cleanup, the connection is not closed beforehand, and kafka-producer-network-thread threads continue to exist in the OS.

Version of the framework
org.springframework.boot:spring-boot-starter-parent:3.3.8
spring.cloud.dependencies 2023.0.5 -> org.springframework.cloud:spring-cloud-stream:jar:4.1.5

To Reproduce
Send several /monitor request to Spring Cloud Config Server or /busrefresh request to configurable service.

To view the threads, use the command
jcmd <PID> Thread.print | grep -E '"main"|"RMI"|"GC"|prio=' --color=auto
You can observe an increasing number of kafka-producer-network-thread threads.

Possible solution
Close kafka connections before cache cleanup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant