argocd error : Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = failed to list refs: Get "https://gitlab.xxxx.com/management.git/info/refs?service=git-upload-pack": context deadline exceeded

[lizzyliao]

Argocd version : v2.14.11
Gitlab version: v18.2.5
Scenario:
When four ArgoCD instances are accessing the same GitLab repository management.git , and running this command:
curl -k https://gitlab.xxxx.com/management.git/info/refs?service=git-upload-pack --output aaa
takes longer than 60 seconds, there is a chance that the following error message occurs:

Argocd error: Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = failed to list refs: Get "https://gitlab.xxxx.com/management.git/info/refs?service=git-upload-pack": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

When Argocd encounters issues, Gitlab functions properly (git pull/push, CI pipeline), with CPU and memory usage around 80%.

Argocd applications:
argocd-prd applications: 1049
argocd-qas applications: 824
argocd-dev applications: 901
argocd-app applications: 156

argocd yml:
reposerver.parallelism.limit: 0
timeout.reconciliation: 5m
timeout.reconciliation.jitter: "2m"
timeout.hard.reconciliation: 0s

Gitlab:
git show-ref | cut -d' ' -f2 | cut -d'/' -f2 | sort | uniq -c
1039 heads
873233 keep-around
8524 merge-requests

How can we fix this problem ? Thank you very much