[Bug]: Redeploy Kafka broker when the operator cannot connect to it

[pbromb]

Bug Description

I have a Kubernetes cluster running on 10 physical servers, where the Strimzi operator is installed in the operators namespace using a Helm chart. Across 60 namespaces, I have deployed single Kafka brokers (KRaft) along with topic/user operators.
I have noticed an issue where, during control plane overload in Kubernetes, the operator strimzi logs display connection errors to the brokers:

2025-01-21 07:35:29 WARN  NetworkClient:1082 - [AdminClient clientId=adminclient-129] Error connecting to node kafka-cluster-kafka-bootstrap.development231.svc:9091 (id: -1 rack: null)
java.net.SocketException: Operation not permitted
	at sun.nio.ch.Net.connect0(Native Method) ~[?:?]
	at sun.nio.ch.Net.connect(Net.java:579) ~[?:?]
	at sun.nio.ch.Net.connect(Net.java:586) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:853) ~[?:?]
	at org.apache.kafka.common.network.Selector.doConnect(Selector.java:293) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.common.network.Selector.connect(Selector.java:271) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:1077) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:321) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.sendEligibleCalls(KafkaAdminClient.java:1270) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.processRequests(KafkaAdminClient.java:1530) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at org.apache.kafka.clients.admin.KafkaAdminClient$AdminClientRunnable.run(KafkaAdminClient.java:1473) ~[org.apache.kafka.kafka-clients-3.9.0.jar:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]

The operation of the Strimzi operator triggers a redeploy of Kafka brokers in such cases, which causes issues for applications relying on the Kafka broker.

Steps to reproduce

No response

Expected behavior

I would expect that in the event of network issues between the Strimzi operator and Kafka brokers, the Kafka brokers would not be redeployed.

Strimzi version

0.45.0

Kubernetes version

Kubernetes 1.29.10, Cilium 1.15.12

Installation method

Helm chart

Infrastructure

Bare-metal

Configuration files and logs

No response

Additional context

No response

[scholzj]

You should probably start by:

• Sharing the full logs and not just a single exception
• Explain what does during control plane overload in Kubernetes mean
• Explain what redeploy of Kafka brokers

Networking issues might cause the Kafka brokers to be rolled as they might lead to the brokers being considered not working. But if the cluster was redeployed, that suggests Kubernetes deleted it.