spring-cloud-kubernetes-fabric8-leader - NullPointerException on pod termination

[1rrup7]

Hi, we are using Spring Boot 3.5.3 with Spring Cloud 2025.0.0 for leader election. However, on pod termination warning messages are logged. It seems that at the point the application is trying to modify the leader id in the config map the Kubernetes client is already destroyed by the Spring context. The corresponding handleRevokedEvent is never called, either.

Logs:

2025-07-21 03:30:30.874 | DEBUG : Checking leader state
2025-07-21 03:30:30.883 | DEBUG : retrieved leaderId: pod-77c56d477-nhxrc from leaderKey : pod-77c56d477-nhxrc
2025-07-21 03:30:30.890 | DEBUG : Leader is still : Leader{role='leader', id='pod-77c56d477-nhxrc'}
2025-07-21 03:30:58.947 | INFO : Leader election shutdown ordering lifecycle stopping
2025-07-21 03:31:03.067 | DEBUG : Leader initiator stopping
2025-07-21 03:31:03.069 | DEBUG : Stopping pod readiness watcher for :pod-77c56d477-vvmkx
2025-07-21 03:31:03.078 | DEBUG : Stopping leader record watcher
2025-07-21 03:31:03.084 | WARN : Failed to stop bean 'leaderInitiator'
2025-07-21 03:31:03.086 | DEBUG : Leader initiator stopping
2025-07-21 03:31:03.088 | WARN : Custom destroy method 'stop' on bean with name 'leaderInitiator' propagated an exception: java.lang.NullPointerException: Cannot invoke "java.util.concurrent.ScheduledExecutorService.shutdown()" because "this.scheduledExecutorService" is null

Stack Trace (Failed to stop bean 'leaderInitiator'):

java.lang.IllegalStateException: Client is closed
	at io.vertx.core.http.impl.HttpClientImpl.checkClosed(HttpClientImpl.java:405)
	at io.vertx.core.http.impl.HttpClientImpl.doRequest(HttpClientImpl.java:281)
	at io.vertx.core.http.impl.HttpClientImpl.request(HttpClientImpl.java:191)
	at io.fabric8.kubernetes.client.vertx.VertxHttpRequest.consumeBytes(VertxHttpRequest.java:87)
	at io.fabric8.kubernetes.client.vertx.VertxHttpClient.consumeBytesDirect(VertxHttpClient.java:131)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.consumeBytesOnce(StandardHttpClient.java:109)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$consumeBytes$1(StandardHttpClient.java:90)
	at io.fabric8.kubernetes.client.utils.AsyncUtils.retryWithExponentialBackoff(AsyncUtils.java:75)
	at io.fabric8.kubernetes.client.utils.AsyncUtils.retryWithExponentialBackoff(AsyncUtils.java:68)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.retryWithExponentialBackoff(StandardHttpClient.java:163)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.consumeBytes(StandardHttpClient.java:88)
	at io.fabric8.kubernetes.client.http.SendAsyncUtils.bytes(SendAsyncUtils.java:50)
	at io.fabric8.kubernetes.client.http.HttpResponse$SupportedResponses.sendAsync(HttpResponse.java:104)
	at io.fabric8.kubernetes.client.http.StandardHttpClient.sendAsync(StandardHttpClient.java:75)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:547)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524)
	at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleGet(OperationSupport.java:467)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handleGet(BaseOperation.java:792)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.requireFromServer(BaseOperation.java:193)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:149)
	at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.get(BaseOperation.java:98)
	at org.springframework.cloud.kubernetes.fabric8.leader.Fabric8LeadershipController.getConfigMap(Fabric8LeadershipController.java:155)
	at org.springframework.cloud.kubernetes.fabric8.leader.Fabric8LeadershipController.lambda$revoke$2(Fabric8LeadershipController.java:85)
	at org.springframework.cloud.kubernetes.commons.leader.LeaderUtils.guarded(LeaderUtils.java:51)
	at org.springframework.cloud.kubernetes.fabric8.leader.Fabric8LeadershipController.revoke(Fabric8LeadershipController.java:84)
	at org.springframework.cloud.kubernetes.commons.leader.LeaderInitiator.stop(LeaderInitiator.java:80)
	at org.springframework.cloud.kubernetes.commons.leader.LeaderInitiator.stop(LeaderInitiator.java:87)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:463)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:618)
	at java.base/java.lang.Iterable.forEach(Unknown Source)
	at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:432)
	at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:323)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1172)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:179)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1126)
	at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:147)
	at java.base/java.lang.Iterable.forEach(Unknown Source)
	at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:116)
	at java.base/java.lang.Thread.run(Unknown Source)

application.yml:

spring:
  cloud:
    kubernetes:
      leader:
        enabled: true
        config-map-name: leader-cm

kustomize:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - service-monitor.yaml
  - leader-role.yaml
  - leader-binding.yaml

configMapGenerator:
  - name: leader-cm
    behavior: create
    options:
      disableNameSuffixHash: true


apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: leader-role
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - watch
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      - leader-cm
    verbs:
      - watch
      - get
      - update

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: leader-rb
roleRef:
  apiGroup: ""
  kind: Role
  name: leader-role
subjects:
  - kind: ServiceAccount
    name: default

deployment.yaml (probes):

apiVersion: apps/v1
kind: Deployment
spec:
      containers:
          startupProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 5
            successThreshold: 1
            failureThreshold: 24
            timeoutSeconds: 3
          readinessProbe:
            httpGet:
              path: /readyz
              port: 8080
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
            timeoutSeconds: 5
          livenessProbe:
            httpGet:
              path: /livez
              port: 8080
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 6
            timeoutSeconds: 5

LeaderService:

@Service
public class LeaderElectionService {

    private volatile boolean isLeader = false;

    public boolean isLeader() {
        return isLeader;
    }

    @EventListener
    public void handleGrantedEvent(OnGrantedEvent event) {
        isLeader = true;
    }

    @EventListener
    public void handleRevokedEvent(OnRevokedEvent event) {
        isLeader = false;
    }
}

[wind57]

I will take a closer look at this next week, I hope. Thank you for the report, will get back

[wind57]

hmm, it seems like a bug and I think (not entirely sure, I'll do some testing) I know why it happens.

@ryanjbaxter can you add the bug label please?

[wind57]

interesting that we have not received this issue until now, I can easily reproduce it...

[wind57]

#1962

@ryanjbaxter can you please link the issue? thank you!

[wind57]

this can be closed now @ryanjbaxter . thx