4.12.1 Helm chart upgrade through ArgoCD throws error on sync: runtime error: invalid memory address or nil pointer dereference #3923
Description
Describe the bug
We use ArgoCD v2.14.11 and deploy the SumoLogic k8s collector as a Helm release that is managed through an ArgoCD Application resource. We initially started using the k8s collector Helm chart version 4.11.0 which synced without any show stoppers and have updated the Helm chart version to 4.11.1 also without any problems.
Today, I did two things: enabled metrics-server and then upgraded the Helm chart version to the latest release 4.12.1. The sync after the metrics-server change was successful. When I attempt to sync after the Helm chart update, the ArgoCD sync immediately fails and this error is thrown in the UI:
runtime error: invalid memory address or nil pointer dereference
If I revert my Helm chart version change back to 4.11.1, the ArgoCD sync is successful again.
Logs
Error logged from the ArgoCD Application controller pod when I try to sync:
{"level":"error","msg":"Recovered from panic: runtime error: invalid memory address or nil pointer dereference\ngoroutine 235 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:26 +0x5e\ngithub.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem.func1()\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1605 +0x54\npanic({0x3e7e040?, 0x8240a20?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\ngithub.com/argoproj/argo-cd/v2/util/argo/managedfields.newTypedResults(0xc00270a688?, 0xc00270a6a8, 0x0)\n\t/go/src/github.com/argoproj/argo-cd/util/argo/managedfields/managed_fields.go:98 +0x2a\ngithub.com/argoproj/argo-cd/v2/util/argo/managedfields.Normalize(0xc00270a698, 0xc00270a688, {0xc00dc80aa0, 0x1, 0xc00f4617e8?}, 0x0)\n\t/go/src/github.com/argoproj/argo-cd/util/argo/managedfields/managed_fields.go:34 +0xab\ngithub.com/argoproj/argo-cd/v2/util/argo/diff.preDiffNormalize({0xc007930308, 0x6f, 0x2756b3147282?}, {0xc003e25b08, 0x6f, 0xc1fa1c46f506d61a?}, {0x5b8c168, 0xc005e8f080})\n\t/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:436 +0x571\ngithub.com/argoproj/argo-cd/v2/util/argo/diff.StateDiffs({0xc007930308?, 0xc0013c06e0?, 0x472bc0a?}, {0xc003e25b08?, 0xc000844bd0?, 0x9?}, {0x5b8c168, 0xc005e8f080})\n\t/go/src/github.com/argoproj/argo-cd/util/argo/diff/diff.go:300 +0x45\ngithub.com/argoproj/argo-cd/v2/controller.(*appStateManager).CompareAppState(0xc000ad8a50, 0xc005610408, 0xc004f0d688, {0xc00c95b6d0, 0x1, 0x1}, {0xc003e6c480, 0x1, 0x1}, 0x0, ...)\n\t/go/src/github.com/argoproj/argo-cd/controller/state.go:763 +0x3b7b\ngithub.com/argoproj/argo-cd/v2/controller.(*ApplicationController).processAppRefreshQueueItem(0xc000d3e5a0)\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:1726 +0x1696\ngithub.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run.func3()\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:900 +0x25\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x30?)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:226 +0x33\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc001191cb0, {0x5b19da0, 0xc0012722d0}, 0x1, 0xc0010b53b0)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:227 +0xaf\nk8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc001191cb0, 0x3b9aca00, 0x0, 0x1, 0xc0010b53b0)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:204 +0x7f\nk8s.io/apimachinery/pkg/util/wait.Until(...)\n\t/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/backoff.go:161\ncreated by github.com/argoproj/argo-cd/v2/controller.(*ApplicationController).Run in goroutine 101\n\t/go/src/github.com/argoproj/argo-cd/controller/appcontroller.go:899 +0x89b\n","time":"2025-04-22T20:18:35Z"}
Command used to install/upgrade Collection
ArgoCD runs Helm in the background to install and upgrade releases. The first upgrade from 4.11.0 to 4.11.1 was successful, but something in the 4.12.1 Helm chart changed that is causing ArgoCD to fail.
Configuration
My ArgoCD Application resource that manages the Sumo Logic k8s collector Helm release. The Helm value overrides are inline. The commented out sync options were originally enabled but I disabled them temporarily to see if it made any difference (it did not).
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: sumologic-kubernetes-collection
namespace: argocd
spec:
project: default
destination:
server: https://kubernetes.default.svc
namespace: sumologic
syncPolicy:
automated:
selfHeal: true
prune: true
syncOptions:
# - ServerSideApply=true
# - CreateNamespace=true
# - PruneLast=true
source:
chart: sumologic
repoURL: https://sumologic.github.io/sumologic-kubernetes-collection
targetRevision: 4.12.1
helm:
values: |
sumologic:
envFromSecret: sumo-api-secret
clusterName: eks-pci-prod-us-west-2
logs:
additionalFields:
- environment
collector:
fields:
environment: production
metrics-server:
enabled: true
To Reproduce Steps to reproduce the behavior.
- Configure ArgoCD application that manages sumologic-kubernetes-collection Helm chart version 4.11.0
- Sync application
- Upgrade chart version to 4.11.1
- Sync application
- Upgrade chart version to 4.12.1
- Observe error after sync attempt
Expected behavior A clear and concise description of what you expected to happen.
The Helm release is upgraded successfully
Environment (please complete the following information):
- Collection version (e.g.
helm ls -n sumologic):4.12.1 - Kubernetes version (e.g.
kubectl version):1.30 - ArgoCD version
v2.14.11 - Cloud provider:
AWS EKS - Others:
Anything else do we need to know Add any other context about the problem here, e.g. issue happens only occasionally.