[BUG] clickhouse keeper scalein hang: no template found for pod demo/ch-cluster-ch-keeper-1 #9833
Description
Describe the bug
This bug is happened on main branch, when scaleIn pod>=2.
e.g. clickhouse keeper 3->1, hang on 2 replica.
When scaling in from 3 → 1, the third node is terminated (k get pod disappears). However, when reconciling to statusReconciler, a problem occurs in isPodUpdated(): this function skips offloadInstances (i.e., the second node), but at this point, the second pod still exists.
Oct 30 11:36:39.000 |INFO| reconcile reconcileID=21650eb4-074d-4bc0-887e-8d537a6d6aa9
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=23f31a6c-2e92-4d2d-9b76-9c575a0e991f error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |ERRO| Reconciler error reconcileID=a20c13fd-5410-4b54-a654-653e0465dc87
Oct 30 11:36:39.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=c893e1c3-2eed-4a7f-95b8-544c4eec2605
Oct 30 11:36:39.000 |INFO| reconcile object *v1.ConfigMap with action UPDATE OK
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=73125ee5-bce8-4541-b18e-9e4c5d0bc122 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| reconcile object *v1.InstanceSet with action UPDATE OK namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=c893e1c3-2eed-4a7f-95b8-544c4eec2605
Oct 30 11:36:39.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false name=ch-cluster-ch-keeper Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=823f538e-7a98-43e1-aecd-87024d2f600e
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=3e42a3a3-3b7c-4ed6-a07e-881f6fd1e9b7 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| reconcile object *v1.ConfigMap with action UPDATE OK namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=c893e1c3-2eed-4a7f-95b8-544c4eec2605
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=7c722c65-ef5d-471d-a9ab-f77d5fc8ca5c
Oct 30 11:36:39.000 |INFO| execute error: Operation cannot be fulfilled on instancesets.workloads.kubeblocks.io "ch-cluster-ch-keeper": the object has been modified; please apply your changes to the latest version and try again controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=823f538e-7a98-43e1-aecd-87024d2f600e
Oct 30 11:36:39.000 |INFO| reconcile object *v1.Component with action STATUS OK name=ch-cluster-clickhouse-rsc Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=c893e1c3-2eed-4a7f-95b8-544c4eec2605
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=fc023738-b6ad-4b24-a785-662a35291121 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| reconcile namespace=demo name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=f1f4f8f2-42be-47f3-b993-8538a0e02e61
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=568373dd-1b39-4dc9-9535-f2f17576de8c error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| reconcile namespace=demo name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=eb0af09a-ee6c-498c-b1b6-d1dd6754c063
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=17b0ed26-5ab6-4651-9386-0a5b87a8f8bf error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |ERRO| Reconciler error reconcileID=e77b3f6b-0ffe-4397-907c-7f5dfc0599c8
Oct 30 11:36:39.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| reconcile object *v1.ConfigMap with action UPDATE OK
Oct 30 11:36:39.000 |INFO| status conditions, creating: false, its running: false, has failure: false, updating: false name=ch-cluster-clickhouse-rsc Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=52d0947e-8d7e-442b-bf7d-02c235d7a3d8
Oct 30 11:36:39.000 |INFO| reconcile object *v1.Component with action STATUS OK
Oct 30 11:36:39.000 |INFO| reconcile object *v1.InstanceSet with action UPDATE OK name=ch-cluster-ch-keeper Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| cluster component phase transition: -> Updating (cluster component ch-keeper is Updating) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster reconcileID=9582d05a-86ed-4d2a-90b1-a4cdde964298
Oct 30 11:36:39.000 |INFO| cluster sharding phase transition: -> Updating (cluster sharding clickhouse is Updating)
Oct 30 11:36:39.000 |INFO| reconcile object *v1.ConfigMap with action UPDATE OK controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| InstanceSet demo/ch-cluster-ch-keeper instances are not aligned controller=instanceset InstanceSet.namespace=demo InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=3b10c444-06ac-47e8-bafa-2d223dc8d9bd
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=dd1d320f-cdf8-4ac7-b4f3-1e2262700f4c
Oct 30 11:36:39.000 |INFO| reconcile object *v1.Component with action PATCH OK controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=39688056-be9d-40a9-9f75-a21e203ad83a
Oct 30 11:36:39.000 |INFO| reconcile object *v1.Component with action STATUS OK controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=61e11251-ed91-4169-bfdc-aebbd8b2848a
Oct 30 11:36:39.000 |INFO| cluster component phase transition: -> Failed (cluster component ch-keeper is Failed) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster reconcileID=37d0bab1-e00a-42e6-b83f-15ab9d9dc9f5
Oct 30 11:36:39.000 |INFO| cluster sharding phase transition: -> Updating (cluster sharding clickhouse is Updating)
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=db79b5a3-8178-4a92-af04-513678823037
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=6dfe5301-5c9c-4843-8359-4a1fa3494bc5 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:39.000 |INFO| cluster component phase transition: -> Failed (cluster component ch-keeper is Failed) namespace=demo name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster controllerGroup=apps.kubeblocks.io reconcileID=c70a0373-1ae4-41fd-b548-d25ddf79bd5d
Oct 30 11:36:39.000 |INFO| cluster sharding phase transition: -> Updating (cluster sharding clickhouse is Updating)
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=5ad7d9f4-547f-40df-b31a-2c7eb22a6d96
Oct 30 11:36:39.000 |INFO| cluster component phase transition: -> Failed (cluster component ch-keeper is Failed) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster controllerGroup=apps.kubeblocks.io reconcileID=5f98e759-9c15-4b40-bc33-a356d8f3ca5a
Oct 30 11:36:39.000 |INFO| cluster sharding phase transition: -> Updating (cluster sharding clickhouse is Updating)
Oct 30 11:36:39.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=9161834b-cd8f-481f-b975-97c375f77cb6
Oct 30 11:36:39.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=abf92a0d-4ace-49a7-864e-7ac00869cf7e error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=d5a9922e-5396-4e05-b958-92bcfc325331
Oct 30 11:36:40.000 |INFO| execute error: Operation cannot be fulfilled on components.apps.kubeblocks.io "ch-cluster-clickhouse-rsc": the object has been modified; please apply your changes to the latest version and try again
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=0d404486-7a7f-4467-9ccc-22a7b6660f76 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=64d52312-630c-447a-afe1-48d5a59f2599
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action PATCH OK
Oct 30 11:36:40.000 |INFO| execute error: Operation cannot be fulfilled on components.apps.kubeblocks.io "ch-cluster-ch-keeper": the object has been modified; please apply your changes to the latest version and try again
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=8b2fbc8a-366b-4052-bc3a-cea00312e91b error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=7ffe03ec-c056-4ed1-a173-62ec3f03471b
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK
Oct 30 11:36:40.000 |INFO| cluster component phase transition: -> Failed (cluster component ch-keeper is Failed) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster reconcileID=2c42a240-858f-4fb2-ba73-fe70fa62eedc
Oct 30 11:36:40.000 |INFO| cluster sharding phase transition: -> Running (cluster sharding clickhouse is Running)
Oct 30 11:36:40.000 |INFO| reconcile name=ch-scale-in controller=opsrequest OpsRequest.namespace=demo controllerKind=OpsRequest opsRequest.namespace=demo opsRequest.namespace=demo OpsRequest.name=ch-scale-in opsRequest.name=ch-scale-in opsRequest.name=ch-scale-in controllerGroup=operations.kubeblocks.io reconcileID=5c9e4268-d6b8-47e8-83a1-642d1547632d
Oct 30 11:36:40.000 |INFO| cluster component phase transition: -> Failed (cluster component ch-keeper is Failed) name=ch-cluster controller=cluster Cluster.namespace=demo cluster.namespace=demo controllerKind=Cluster Cluster.name=ch-cluster cluster.name=ch-cluster controllerGroup=apps.kubeblocks.io reconcileID=cff666aa-acfa-4a03-817f-4a01b4d4f577
Oct 30 11:36:40.000 |INFO| cluster sharding phase transition: -> Running (cluster sharding clickhouse is Running)
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=c6e5d57d-f5f5-4e4c-829d-a0219b6afcc6 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-ch-keeper controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=e644ecd3-58cb-4e71-ba73-cebc9b485e40
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=21f6b36a-ae47-459d-8ff9-01f34e701a35 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=681dd563-d69c-499a-a807-56c440692c67
Oct 30 11:36:40.000 |INFO| execute error: Operation cannot be fulfilled on components.apps.kubeblocks.io "ch-cluster-clickhouse-rsc": the object has been modified; please apply your changes to the latest version and try again
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: false, has failure: true, updating: false name=ch-cluster-ch-keeper Component.name=ch-cluster-ch-keeper component.name=ch-cluster-ch-keeper reconcileID=d7afbacb-d713-4465-975f-09edc0d412c7
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false name=ch-cluster-clickhouse-rsc Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=2306d427-c04c-4d9f-ab52-402c6dc9087d
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK
Oct 30 11:36:40.000 |INFO| status conditions, creating: false, its running: true, has failure: false, updating: false reconcileID=77df2453-73ae-4109-858a-683671dc69a3
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=3dd10a06-6d39-4e12-b3cf-3ae679ae2e3a error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:40.000 |INFO| reconcile object *v1.Component with action STATUS OK namespace=demo controller=component Component.namespace=demo component.namespace=demo controllerKind=Component name=ch-cluster-clickhouse-rsc controllerGroup=apps.kubeblocks.io Component.name=ch-cluster-clickhouse-rsc component.name=ch-cluster-clickhouse-rsc reconcileID=77df2453-73ae-4109-858a-683671dc69a3
Oct 30 11:36:40.000 |ERRO| Reconciler error namespace=demo controller=instanceset name=ch-cluster-ch-keeper InstanceSet.namespace=demo controllerKind=InstanceSet InstanceSet.name=ch-cluster-ch-keeper controllerGroup=workloads.kubeblocks.io reconcileID=115f53e7-6a33-41ff-b493-66c772460850 error=no template found for pod demo/ch-cluster-ch-keeper-1 stacktrace=sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/Users/loomt/gopath/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=227b61d4-46e1-473c-9664-92bd48e83cae
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=a9ed1170-cb72-48e4-b3e9-92279c5c47f6
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=3a50794e-a150-4b9a-beb2-b22ee7421b94
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=55acebb4-fc50-4edb-9364-456d1b53ccb9
Oct 30 11:36:41.000 |ERRO| Reconciler error reconcileID=60118cee-a554-407d-9d30-f01c306ed4fe
Oct 30 11:36:44.000 |ERRO| Reconciler error reconcileID=7a5e0f33-0685-45c9-9429-6ac535cfd11bTo Reproduce
Steps to reproduce the behavior:
- Go to '...'
- Click on '....'
- Scroll down to '....'
- See error
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
Additional context
Add any other context about the problem here.