Skip to content

Question: Flapping ArgoCD reconciliation when spec.remoteClusters is defined #8781

@kgorskowski

Description

@kgorskowski

ECK version: 2.14.0
Elasticsearch version: 8.18.3 Enterprise Orchestration License
GitOps tool: Argo CD
Observed behavior: Continuous flapping between Synced and OutOfSync in Argo CD when spec.remoteClusters is defined in the Elasticsearch manifest.

Problem Description
We are deploying Elasticsearch clusters using ECK and managing the manifests via GitOps with Argo CD.
We have a custom "helm chart" for our complete stack but we use helm template to generate a plain yaml manifest that we deploy with ArgoCD.
In a multi-cluster setup, we define the spec.remoteClusters block directly in our Git-managed manifests to establish cross-cluster trust.

While the trust setup works, we are experiencing persistent reconciliation flapping in Argo CD.
The elastic cluster is green and useable, the desired remote cluster is available, connected and green andall stack configuration policies applied.
The ArgoCD application constantly switches between Synced and OutOfSync, despite no visible differences being reported by argocd app diff with following messages:

Could not update remote clusters in Elasticsearch settings, re-queuing
Operation cannot be fulfilled on elasticsearches.elasticsearch.k8s.elastic.co "cluster-name": the object has been modified; please apply your changes to the latest version and try again

Already tried to ignore some of the specs I saw modified by the ECK Operator

resource.customizations: |
 elasticsearch.k8s.elastic.co/Elasticsearch:
   ignoreDifferences: |
     jsonPointers:
       - /spec/remoteClusters
       - /metadata/annotations/elasticsearch.k8s.elastic.co~1managed-remote-clusters

but to no avail

We are looking for one of the following:

Is anyone already running a conflict-free method to manage remoteClusters declaratively through Git?

If so, what resources/mutations can be ignored on the ArgoCD application side to break the OutOfSync loop?
A recommendation for how to structure ownership boundaries between ECK and GitOps tools like Argo CD when using trust configuration.
Any hints or inputs would be greatly appreciated!
Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions