Strimzi error 409

[palpandy]

Hello,

I am facing weird issue in strimzi-operator because of that it couldn't reconcile the mm2's properly and ended up having issue with mm2 replication. We are using 2 replicas of strimzi-operator . Below is the error,

2024-11-27 18:07:28 INFO Main:346 - Health and metrics server is ready on port 8080)
2024-11-27 18:07:28 INFO Main:255 - Waiting to become a leader
2024-11-27 18:07:28 INFO LeaderElectionManager:69 - Starting the Leader Elector
2024-11-27 18:07:28 INFO LeaderElectionManager:121 - The new leader is strimzi-cluster-operator-app-ffc989754-vkt4k
2024-11-27 20:01:32 ERROR LeaderElector:181 - Exception occurred while acquiring lock 'LeaseLock: strimzi-operator - strimzi-cluster-operator (strimzi-cluster-operator-app-ffc989754-stspr) retrying...'
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.226.0.1:443/apis/coordination.k8s.io/v1/namespaces/strimzi-operator/leases/strimzi-cluster-operator. Message: Operation cannot be fulfilled on leases.coordination.k8s.io "strimzi-cluster-operator": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=coordination.k8s.io, kind=leases, name=strimzi-cluster-operator, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on leases.coordination.k8s.io "strimzi-cluster-operator": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:238) ~[io.fabric8.kubernetes-client-api-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:763) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:232) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:237) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:252) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:45) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.extension.ResourceAdapter.patch(ResourceAdapter.java:306) ~[io.fabric8.kubernetes-client-api-6.10.0.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.resourcelock.ResourceLock.update(ResourceLock.java:65) ~[io.fabric8.kubernetes-client-api-6.10.0.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.tryAcquireOrRenew(LeaderElector.java:236) ~[io.fabric8.kubernetes-client-api-6.10.0.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$acquire$4(LeaderElector.java:172) ~[io.fabric8.kubernetes-client-api-6.10.0.jar:?]
at io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector.lambda$loop$8(LeaderElector.java:282) ~[io.fabric8.kubernetes-client-api-6.10.0.jar:?]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: https://10.226.0.1:443/apis/coordination.k8s.io/v1/namespaces/strimzi-operator/leases/strimzi-cluster-operator. Message: Operation cannot be fulfilled on leases.coordination.k8s.io "strimzi-cluster-operator": the object has been modified; please apply your changes to the latest version and try again. Received status: Status(apiVersion=v1, code=409, details=StatusDetails(causes=[], group=coordination.k8s.io, kind=leases, name=strimzi-cluster-operator, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Operation cannot be fulfilled on leases.coordination.k8s.io "strimzi-cluster-operator": the object has been modified; please apply your changes to the latest version and try again, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Conflict, status=Failure, additionalProperties={}).
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:660) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:640) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:589) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549) ~[io.fabric8.kubernetes-client-6.10.0.jar:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:143) ~[io.fabric8.kubernetes-client-api-6.10.0.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2147) ~[?:?]
at io.fabric8.kubernetes.client.utils.AsyncUtils.lambda$retryWithExponentialBackoff$3(AsyncUtils.java:90) ~[io.fabric8.kubernetes-client-api-6.10.0.jar:?]
at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) ~[?:?]
at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
at java.util.concurrent.CompletableFuture.postFire(CompletableFuture.java:614) ~[?:?]
at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:844) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:482) ~[?:?]
... 1 more

So, strimzi gets blocked after this error sometimes.

Could you help us to understand this error and how to resolve this ? Also any possibility of catching this error using any alert ?

[scholzj]

From what you shared, it is not really clear what is happening nor what exactly is blocked. You should probably start by sharing what exactly you are running and full logs from all operator instances so that we get better idea.