Sharding application-controller by Application or a Project

[msobkowiak-olx]

Hey there !

Wanted to ask on possibilities to distribute load across multiple application-controllers, our setups usually consist of one ArgoCD instance per cluster, thus is not very beneficial for us to use default sharding option described here:
https://argo-cd.readthedocs.io/en/stable/operator-manual/high_availability/

I would like to ask if there is any plans to actually change this to shard per application name (using static or consistent (with crash support) hashing, just like on memcached clients or in loadbalancers), which should have the following benefits:

• best possible granularity of the load (with monorepos still being sort of an issue however it is still an issue even now)
• best resource ultilisation (achieving best possible "even" distribution, with exception for large monorepos)
• best possible scaling scenarios (supporting ArgoCD with one or multiple clusters)

Similar can be applied to a project. The hashing can simply take project or app name, calculate the hash and send it to proper slot for processing in the deployed statefulset. This should be possible because of the constraints ArgoCD puts on the Applications and Projects (unique naming, guaranteeing that each entity has unique key for hashing function, the distribution happens onto application-controller pods).

Please note I am aware of the queue tuning on operation and status processors, however we would like to avoid creating too large pods not fitting our workload nodes in the clusters hosting ArgoCD.

Let me know if it makes sense.

[awx-fuyuanchu]

Yeah. It sounds like a more general pain point for most users. One cluster could spawn hundreds of Applications and sharding the Applications in one cluster can help a lot for performance, efficiency, and scalability.

[saprette]

Hello, that is something that we would benefit from.

[andrii-korotkov-verkada]

I suggest to open an enhancement proposal for this. I don't know if this is easily solvable, since resources can move between applications and applications can move between projects.

[andrii-korotkov-verkada]

I think the problem is two things can maintain a different view of the same object in Kubernetes, so there might be conflicting changes.

[msobkowiak-olx]

Isn't this an issue even now on single shard ? I mean you can technically deploy two different applications with different name and exactly the same manifests (we used to hit collisions sometimes during migrations from helm).

[andrii-korotkov-verkada]

There's a cluster lock acquired by a number of places.

[msobkowiak-olx]

Thanks, I mean it would be crucial to understand what exactly this lock synchronizes in this case. Implementing a distributed lock would not be that easy (technically the locking for cross-shard implementation could be externalised to redis or so - that is actually not that hard but prone to process crashes without expiration implementation).

[andrii-korotkov-verkada]

I can bring this up during the contributors meeting.

[andrii-korotkov-verkada]

See #6125 for a similar thing.

[crenshaw-dev]

Most of the work that the Application controller does is cluster-scoped. Breaking it into Application- or Project-sized chunks is very high technical cost for potentially very low reward.

In my opinion, the time is better spent improving the general performance of the Application controller and tuning performance configuration to alleviate issues.

[stormmore]

It feels like you are making an assumption that people with multiple cluster run a single argocd architecture.

As someone who runs single argocd per cluster, I would argue that Application/Project chunks is very high reward, especially if I can shard critical application out from not so critical. The only sharding method we have that gives us any granularity is to run multiple argocds - one for each grouping - which is basically what our platform vendor recommends as well.

For the record, one of the clusters that I am an engineer of currently uses ArgoCD to manage 180 "applications" and we have just pushed its single controller up to 1.5 core / 6 GB request and 4 core / 12 GB due to OOMKills and slow sync speeds.

[crenshaw-dev]

If the main motivation is to mitigate "noisy neighbor" issues, I think a prod/pre-prod Argo CD instance split makes a lot of sense and is probably a better idea than having that split within just the application-controller. Presumably you'd also want to protect the repo-server, applicationset-controller, Dex, etc. from noisy pre-prod neighbors.

[luispabon]

Our use case is one cluster, one argocd instance managing hundreds of applications in that cluster. We've been forced to scale the application controller's node to very high levels (16 vCPUs per node) just to cope with the load created on a single pod.

Redis being available, can't it not be used as ephemeral state to coordinate which controller replica is working on what? Not knowing argo's internal design I can think of no reason this particular job can't be pub/subbed or some other similar mechanism for assigning work.

[chris-ng-1987]

Agree with @luispabon, I feel bad as ArgoCD is a CNCF project, but the workload of application controller is not supporting distribute single cluster applications workload by horizontal scaling, forcing vertical scaling without HA.

[7onn]

In a staging cluster, besides the apps themselves, I also provision preview environments where in each namespace, I have to render "my whole company"'s manifests; It is a lot to handle single-shardedly; and ArgoCD became really slow; sometimes taking nearly a minute to get ready to sync changes that people push on their pull requests.

It was a sad suprise when I enabled sharding just to find out all manifests were still falling into the same shard, leaving other two instances completely idle.

My current scale in the staging cluster is ~125 applications; but the preview ones (~90) can be very heavy, carrying from 26 to 80 resource's yaml per application! They are provisioned by pull request, depending on which apps in our monorepo are changing on a given PR.

So I can say it would be extremely rewarding for me if I could balance these apps' manifests across more ArgoCD app controller instances;