How to handle lost event?

[bsch92]

We recently faced the problem that the OnRevokedEvent was lost, when using spring-cloud-kubernetes-fabric8-leader.

Our setup consists of 3 kubernetes pods with one leader. While our configuration is certainly not perfect (leadership is sometimes revoked while a pod is still running), it mostly works.

During the recent incident, the old leader was revoked because the readiness-check failed but the pod didn't shut down. Shortly after, another pod got elected as the new leader and started to do the leader-specific work. Meanwhile, the old leader didn't receive the OnRevokedEvent and kept on thinking that he was still the leader.

As a result, we had two leaders running until somebody noticed and manually shut down the "phantom-leader".

Sadly I can't reproduce the problem and the logs don't show any anomalies. Is this a known problem or how can I prevent this from happening?

[wind57]

hello and thank you for the report. This is by far not anything "known", in the sense that there are no issues like this raised, so far at least.

I am going to take a closer look at the code to see if according to your description something jumps in my eyes.

[wind57]

While our configuration is certainly not perfect (leadership is sometimes revoked while a pod is still running)

we treat that as a normal case btw. We have a readiness watcher that gets updated when pod readiness changes. I think that is fine, because if you depend on a DB as part of your work as leader for example, and the DB is down, we should revoke the leader.

---

I looked at the code and without logs in DEBUG mode, it is impossible for me to make even some guesses :( If you can provide those, I can take a look further.

@ryanjbaxter can you please add the feedback tag?

[bsch92]

Hello @wind57,
thank you for taking a look even though I am aware that the description is lacking. Unfortunately, we didn't have DEBUG logs enabled for the affected environment. Maybe something is wrong with our adaption:

LeaderService.java.txt

That lead to the following logs:

Time	Pod	Log
00:00	A	Doing leader-specific work
01:00	A	Doing leader-specific work
01:23	B	Leadership granted
02:00	A	Doing leader-specific work
02:00	B	Doing leader-specific work
02:05	B	Leadership revoked
02:05	C	Leadership granted
03:00	A	Doing leader-specific work
03:00	C	Doing leader-specific work

As you can see, leadership was never revoked for the original Pod A and it was still doing leader-specific work. For the other pods, leadership changed without a problem.

[wind57]

tbh, we are going to switch to a new implementation for the leader election and the one you are using now is becoming obsolete and will be removed in the future.

We will be using a more "native" way for leader election, build in the fabric8 client that we are using. here is the PR : #1658 and it should be going pretty soon in main.

The point is I am a bit reluctant to spend too much time here, especially since we do not have debug logs. So my suggestion is, you can turn debug logs on for certain packages only: org.springframework.cloud.kubernetes.fabric8.leader and org.springframework.cloud.kubernetes.commons.leader, that will help a lot.

If that is too much of a hassle for you, you can just wait for the new leader implementation and test against that one. In the long run, we will be supporting only the new implementation.

[bsch92]

That's the main problem, we can't reproduce the behaviour. We have had multiple environments running for several months but only once did we face the mentioned issue. Most of the other environments have debug logs enabled...

But if I understood you correctly, there is nothing fundamentally wrong with our usage.

Is there some kind of timeline for the new release or when can we approximately expect it?

[wind57]

But if I understood you correctly, there is nothing fundamentally wrong with our usage.

correct.

Is there some kind of timeline for the new release or when can we approximately expect it?

can't tell, I'm not a Spring employee... may be Ryan could.

[ryanjbaxter]

The new implementation with be part of spring cloud 2025.1.9 which is tentatively scheduled for November. You can always check release dates here https://github.com/spring-cloud/spring-cloud-release/milestones