[Features] Migrate cluster leader ConfigMap to Leases #9828
Description
If this is a new feature, please describe the motivation and goals.
Since cluster leader ConfigMaps do not store any actual configuration data, but instead just have annotations that are frequently updated, it might make sense long-term to migrate them to a resource specifically designed for leases: the Lease object in group coordination.k8s.io. The idea would be to reduce API churn on a frequently-watched object like ConfigMaps. Leases also get garbage collected by the kube apiserver 1 hour after expiring; free functionality.
Is your feature request related to a problem? Please describe.
I deploy Kubeblocks clusters into namespaces watched by ArgoCD. In production I currently have 80+ clusters running across my cluster. Since Kubeblocks updates its leader ConfigMaps so frequently, ArgoCD is constantly being notified of ConfigMap updates for each of the 80 clusters and must check if a reconciliation is necessary for the app in that namespace.
This leads to substantial CPU usage and apiserver event churn. I can decrease the CPU usage is greatly by annotating the ConfigMaps with argocd.argoproj.io/ignore-resource-updates: "true", (ArgoCD's official way to ignore updates to resources that are not tracked or deployed by ArgoCD). However because the leader configmaps are generated by Kubeblocks at runtime and can be deleted with e.g. cluster restarts, I'm forced to keep them labelled by cron job.
I believe migrating the cluster leader lease to a different kube resource would improve this situation. ArgoCD does not watch Leases, as they are meant to be created at runtime.
Describe alternatives you've considered
Perhaps there is another k8s resource besides Leases that would be more appropriate for Kubeblocks' use, but Leases seems to be the most appropriate to me.