Repetitive update of the reconciled VaultSecret status #38
Description
While preparing to test #37, another problem was discovered. It can be reproduced with both 1.1.2 and 1.2.1.
The scenario is:
- the Vault operator runs in
argocdnamespace and watches on its own namespace. It runs usingdefaultSA - the
git-githubVaultSecret is being created in the namespace. It should be synced with thegit-githubsecret
Everything works smoothly, but the following gets logged by the Vault operator
2022-03-11T15:00:56.732Z ERROR controller Reconciler error {"reconcilerGroup": "maupu.org", "reconcilerKind": "VaultSecret", "controller": "vaultsecret", "name": "git-github", "namespace": "argocd", "error": "Operation cannot be fulfilled on vaultsecrets.maupu.org \"git-github\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:237
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:209
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:188
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:90
Seems the operator attempts to reconcile the secret three times. The first two attempts start at nearly the same time and one of the threads gets into the conflict trying to update the status of the VaultSecret.
This is confirmed by the logs of the API server. it gets two requests to update /apis/maupu.org/v1beta1/namespaces/argocd/vaultsecrets/git-github/status from system:serviceaccount:argocd:default. The second update is rejected with "Conflict" 409.
❯ kubectl -n argocd logs vault-secret-7978575d49-88b77 | grep "Reconciling VaultSecret"
2022-03-11T15:00:41.050Z INFO vaultsecret-operator Reconciling VaultSecret {"Request.Namespace": "argocd", "Request.Name": "git"}
2022-03-11T15:00:41.244Z INFO vaultsecret-operator Reconciling VaultSecret {"Request.Namespace": "argocd", "Request.Name": "git-github-internal"}
2022-03-11T15:00:41.322Z INFO vaultsecret-operator Reconciling VaultSecret {"Request.Namespace": "argocd", "Request.Name": "argocd-secret"}
2022-03-11T15:00:41.387Z INFO vaultsecret-operator Reconciling VaultSecret {"Request.Namespace": "argocd", "Request.Name": "git"}
2022-03-11T15:00:56.574Z INFO vaultsecret-operator Reconciling VaultSecret {"Request.Namespace": "argocd", "Request.Name": "git-github"}
2022-03-11T15:00:56.668Z INFO vaultsecret-operator Reconciling VaultSecret {"Request.Namespace": "argocd", "Request.Name": "git-github"}
2022-03-11T15:00:57.732Z INFO vaultsecret-operator Reconciling VaultSecret {"Request.Namespace": "argocd", "Request.Name": "git-github"}