4 Out of 5 Brokers Crash Loop

[nChatziioannou]

The issue occurred 7 days after the initial deployment.

The brokers are deployed in Azure AKS.

Error:
java.nio.file.FileSystemException: /var/lib/kafka/data-0/kafka-log2/__consumer_offsets-24/00000000000000000000.log.cleaned: Operation not permitted
.
.
Shutdown broker because all log dirs in /var/lib/kafka/data-0/kafka-log0 have failed

The same error, in different consumer offsets, was thrown in the 4 different brokers. One out of the five had no issues.

After checking the file share, I was able to delete the files by hand. However, deleting did not fix the issue. The files were recreated and the error was raised.
Only, after deleting the whole directory and restarting the broker was able to start up.
For AKS storage I am using Azure storageAccount. Could this be the issue?
Also, I checked the storage space, and it was more than 80% empty.

KafkaNodePool

apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaNodePool metadata: name: controller labels: strimzi.io/cluster: kafkav2 spec: replicas: 3 roles: - controller storage: type: jbod volumes: - id: 0 type: persistent-claim size: 100Gi kraftMetadata: shared deleteClaim: false class: myCustomClass
Let me know, if you need more info about the configuration files.

[scholzj]

The Node Pools is unreadable, so you will need to fix it. You should also make sure you are using block storage and not file storage for your Kafka cluster as that tends to cause similar issues.