When leader lease is lost applier manager is not restarted

[emosbaugh]

Before creating an issue, make sure you've checked the following:

• You are running the latest released version of k0s
• Make sure you've searched for existing issues, both open and closed
• Make sure you've searched for PRs too, a fix might've been merged already
• You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

No response

Version

v1.29.9+k0s

Sysinfo

`k0s sysinfo`

➡️ Please replace this text with the output of `k0s sysinfo`. ⬅️

What happened?

When a third controller is added the leader lease is somehow lost and when it is re-acquired the applier-manager is not restarted resulting in updates to manifests or stacks not being applied.

Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.386215    3978 leaderelection.go:285] failed to renew lease kube-node-lease/k0s-endpoint-reconciler: timed out waiting for the condition
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.386295    3978 leaderelection.go:285] failed to renew lease kube-node-lease/k0s-ctrl-node-e3a0d-00: timed out waiting for the condition
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Lost leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.391034    3978 leaderelection.go:250] attempting to acquire leader lease kube-node-lease/k0s-ctrl-node-e3a0d-00...
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Lost leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: I1015 16:55:27.391062    3978 leaderelection.go:250] attempting to acquire leader lease kube-node-lease/k0s-endpoint-reconciler...
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="lost leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=error msg="lost leader lease, this should not really happen!?!?!?" component=controllerlease

...

Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="manifest watcher done" component=applier-manager

...

Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="acquired leader lease" component=poolleaderelector
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="acquired leader lease" component=controllerlease
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=extensions_controller
Oct 15 16:55:27 node-e3a0d-00 k0s[3978]: time="2024-10-15 16:55:27" level=info msg="Acquired leader lease" component=extensions_controller

...

Steps to reproduce

Expected behavior

Changes to manifests dir will continue to be applied to the cluster

Actual behavior

Changes are no longer reflected in the cluster.

Screenshots and logs

k0scontroller-logs.txt
k0scontroller-logs.txt
k0scontroller-logs.txt

Additional context

No response

[emosbaugh]

Fixed by #5062 once merged

[github-actions]

The issue is marked as stale since no activity has been recorded in 30 days

[twz123]

Backports need to be approved.