This repository was archived by the owner on Dec 14, 2023. It is now read-only.
This repository was archived by the owner on Dec 14, 2023. It is now read-only.
Operator crashes upon reconcilation if AWS resources are removed #25
Description
Problem Statement
During my testing of this operator to ensure it's production readiness I encountered an issue where the operator would crash on reconcilation if the resources it had previously created were removed from AWS.
Steps to Reproduce
Create a resource:
apiVersion: irsa.voodoo.io/v1alpha1
kind: IamRoleServiceAccount
metadata:
name: s3-get-lister
spec:
policy:
statement:
- resource: "arn:aws:s3:::test-irsa-4gkut9fl"
action:
- "s3:Get*"
- "s3:List*"Ensure operator creates resource successfully:
2022-03-08T10:29:21.117Z INFO controllers.Aws successfully created trust role policy (irsa-op-irsa-controlller-default-s3-get-lister) on aws
2022-03-08T10:29:21.552Z INFO controllers.Aws successfully attached role (irsa-op-irsa-controlller-default-s3-get-lister) & policy (arn:aws:iam::703320575624:policy/irsa-operator/irsa-controlller/default/s3-get-lister/policy/irsa-op-irsa-controlller-default-s3-get-lister) on awsRemove the created resources (note: the below resource names will vary depending on your cluster, operator name and namespace):
aws iam detach-role-policy --policy-arn arn:aws:iam::<accountid>:policy/irsa-operator/irsa-controlller/default/s3-get-lister/policy/irsa-op-irsa-controlller-default-s3-get-lister --role-name irsa-op-irsa-controlller-default-s3-get-lister
aws iam delete-policy --policy-arn arn:aws:iam::<accountid>:policy/irsa-operator/irsa-controlller/default/s3-get-lister/policy/irsa-op-irsa-controlller-default-s3-get-lister
aws iam delete-role --role-name irsa-op-irsa-controlller-default-s3-get-listerNow either make a modification to the resource and apply it:
apiVersion: irsa.voodoo.io/v1alpha1
kind: IamRoleServiceAccount
metadata:
name: s3-get-lister
spec:
policy:
statement:
- resource: "arn:aws:s3:::test-irsa-4gkut9fl"
action:
- "s3:Get*"
- "s3:List*"
- "s3:Put*"or recreate the operator pod:
kubectl delete po -n irsa-controller irsa-controller-irsa-operator-7c87c75f4d-6lmtg
Expected Outcome
Operator pod should reconcile the changes and recreate the deleted resources. The operator pod logs should output the modifications made.
Actual Outcome
The operator is unable to reconcile the differences and the following output is constantly spammed by the pod:
2022-03-08T10:42:46.162Z INFO controllers.IamRoleServiceAccount [default/s3-get-lister] : Failed to create policy : Operation cannot be fulfilled on policies.irsa.voodoo.io "s3-get-lister": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:42:46.261Z INFO controllers.IamRoleServiceAccount [default/s3-get-lister] : Failed to create policy : Operation cannot be fulfilled on policies.irsa.voodoo.io "s3-get-lister": the object has been modified; please apply your changes to the latest version and try again
2022-03-08T10:42:46.361Z INFO controllers.IamRoleServiceAccount [default/s3-get-lister] : Failed to create policy : Operation cannot be fulfilled on policies.irsa.voodoo.io "s3-get-lister": the object has been modified; please apply your changes to the latest version and try again
When recreating the operator pod, it fails to start and outputs the following logs:
2022-03-08T10:47:37.386Z INFO controller-runtime.metrics metrics server is starting to listen {"addr": ":8080"}
2022-03-08T10:47:37.386Z INFO setup starting manager
I0308 10:47:37.386837 1 leaderelection.go:243] attempting to acquire leader lease irsa-controller/d8e70b98.voodoo.io...
2022-03-08T10:47:37.386Z INFO controller-runtime.manager starting metrics server {"path": "/metrics"}
I0308 10:47:54.856004 1 leaderelection.go:253] successfully acquired lease irsa-controller/d8e70b98.voodoo.io
2022-03-08T10:47:54.856Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"ConfigMap","namespace":"irsa-controller","name":"d8e70b98.voodoo.io","uid":"2555cf13-b658-4a04-9c96-ab7c7a35d8c9","apiVersion":"v1","resourceVersion":"1437116"}, "reason": "LeaderElection", "message": "irsa-controller-irsa-operator-7c87c75f4d-j9pnm_61841c80-e002-485c-aba7-346dc8f61461 became leader"}
2022-03-08T10:47:54.856Z INFO controller-runtime.manager.controller.role Starting EventSource {"reconciler group": "irsa.voodoo.io", "reconciler kind": "Role", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.856Z DEBUG controller-runtime.manager.events Normal {"object": {"kind":"Lease","namespace":"irsa-controller","name":"d8e70b98.voodoo.io","uid":"43ce5eba-2daf-49fd-8eb0-6f72c0b9887a","apiVersion":"coordination.k8s.io/v1","resourceVersion":"1437117"}, "reason": "LeaderElection", "message": "irsa-controller-irsa-operator-7c87c75f4d-j9pnm_61841c80-e002-485c-aba7-346dc8f61461 became leader"}
2022-03-08T10:47:54.856Z INFO controller-runtime.manager.controller.iamroleserviceaccount Starting EventSource {"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.856Z INFO controller-runtime.manager.controller.policy Starting EventSource {"reconciler group": "irsa.voodoo.io", "reconciler kind": "Policy", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.956Z INFO controller-runtime.manager.controller.role Starting Controller {"reconciler group": "irsa.voodoo.io", "reconciler kind": "Role"}
2022-03-08T10:47:54.956Z INFO controller-runtime.manager.controller.role Starting workers {"reconciler group": "irsa.voodoo.io", "reconciler kind": "Role", "worker count": 10}
2022-03-08T10:47:54.956Z INFO controller-runtime.manager.controller.policy Starting Controller {"reconciler group": "irsa.voodoo.io", "reconciler kind": "Policy"}
2022-03-08T10:47:54.957Z INFO controller-runtime.manager.controller.policy Starting workers {"reconciler group": "irsa.voodoo.io", "reconciler kind": "Policy", "worker count": 10}
2022-03-08T10:47:54.956Z INFO controller-runtime.manager.controller.iamroleserviceaccount Starting EventSource {"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.957Z INFO controller-runtime.manager.controller.iamroleserviceaccount Starting EventSource {"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "source": "kind source: /, Kind="}
2022-03-08T10:47:54.957Z INFO controller-runtime.manager.controller.iamroleserviceaccount Starting EventSource {"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "source": "kind source: /, Kind="}
2022-03-08T10:47:55.058Z INFO controller-runtime.manager.controller.iamroleserviceaccount Starting Controller {"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount"}
2022-03-08T10:47:55.058Z INFO controller-runtime.manager.controller.iamroleserviceaccount Starting workers {"reconciler group": "irsa.voodoo.io", "reconciler kind": "IamRoleServiceAccount", "worker count": 10}
2022-03-08T10:47:55.197Z INFO controllers.Aws role not found on aws
E0308 10:47:55.197815 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 290 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic(0x18eb100, 0x282d970)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:74 +0x95
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:48 +0x89
panic(0x18eb100, 0x282d970)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).attachPolicyToRoleIfNeeded(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc000132600, 0xc0004469c0)
/workspace/controllers/role_controller.go:192 +0x50a
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).reconcilerRoutine(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc000132600, 0x7, 0xc00048de50, 0xd, 0xc000132600)
/workspace/controllers/role_controller.go:146 +0x185
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).Reconcile(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc00048de60, 0x7, 0xc00048de50, 0xd, 0xc000657620, 0x40a1ff, 0xc000030000, ...)
/workspace/controllers/role_controller.go:87 +0x1a5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006af540, 0x1d8e680, 0xc0006ea100, 0x1952fe0, 0xc000449e00)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298 +0x317
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006af540, 0x1d8e680, 0xc0006ea100, 0x0)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x1d8e680, 0xc0006ea100)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000055750)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005c9f50, 0x1d538c0, 0xc0006575c0, 0xc0006ea101, 0xc00057dc80)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000055750, 0x3b9aca00, 0x0, 0x1, 0xc00057dc80)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1d8e680, 0xc0006ea100, 0xc0006f8440, 0x3b9aca00, 0x0, 0x1)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1d8e680, 0xc0006ea100, 0xc0006f8440, 0x3b9aca00)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:213 +0x45e
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x151a3ca]
goroutine 290 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/runtime/runtime.go:55 +0x10c
panic(0x18eb100, 0x282d970)
/usr/local/go/src/runtime/panic.go:969 +0x1b9
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).attachPolicyToRoleIfNeeded(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc000132600, 0xc0004469c0)
/workspace/controllers/role_controller.go:192 +0x50a
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).reconcilerRoutine(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc000132600, 0x7, 0xc00048de50, 0xd, 0xc000132600)
/workspace/controllers/role_controller.go:146 +0x185
github.com/VoodooTeam/irsa-operator/controllers.(*RoleReconciler).Reconcile(0xc0001bc930, 0x1d8e740, 0xc000657620, 0xc00048de60, 0x7, 0xc00048de50, 0xd, 0xc000657620, 0x40a1ff, 0xc000030000, ...)
/workspace/controllers/role_controller.go:87 +0x1a5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc0006af540, 0x1d8e680, 0xc0006ea100, 0x1952fe0, 0xc000449e00)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:298 +0x317
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc0006af540, 0x1d8e680, 0xc0006ea100, 0x0)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:253 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2(0x1d8e680, 0xc0006ea100)
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:216 +0x4a
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1()
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0x37
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0xc000055750)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:155 +0x5f
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005c9f50, 0x1d538c0, 0xc0006575c0, 0xc0006ea101, 0xc00057dc80)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:156 +0xad
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000055750, 0x3b9aca00, 0x0, 0x1, 0xc00057dc80)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:133 +0x98
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext(0x1d8e680, 0xc0006ea100, 0xc0006f8440, 0x3b9aca00, 0x0, 0x1)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:185 +0xa6
k8s.io/apimachinery/pkg/util/wait.UntilWithContext(0x1d8e680, 0xc0006ea100, 0xc0006f8440, 0x3b9aca00)
/go/pkg/mod/k8s.io/[email protected]/pkg/util/wait/wait.go:99 +0x57
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:213 +0x45e