Talos Cluster unstable #12267
Description
Bug Report
Hello guys, i have setup a talos cluster in my proxmox environnement.
Contains 3 nodes (1 cp and 2 workers)
Talos Version : 1.10.5
Description
The cluster is setup for homelab purpose, but pod keep restarting for unkown reason.
i've been trying to debug this issue with investigate in node, but didn't find error logs that are really helpful.
I have already the issue in a cluster with talos version 1.9.5, upgrading to 1.10.5 didn't fix.
You see the number of pod restart that are high.
Logs
pod kube-api-server
E1123 13:56:00.147794 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:00.149237 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.150382 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.151521 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.152789 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.899284ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/pks-share-wk2" result=null
E1123 13:56:00.173687 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:00.174993 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.176219 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.177357 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.178594 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.839672ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/pks-share-cp" result=null
E1123 13:56:00.271539 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:00.273006 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.274116 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.275257 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.276537 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.925904ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/pks-share-wk1" result=null
E1123 13:56:00.739054 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:00.740219 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.741336 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.742430 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:00.743642 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.552803ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/flux-system/leases/source-controller-leader-election" result=null
E1123 13:56:04.371033 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 6.89µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 13:56:04.371445 1 controller.go:195] "Failed to update lease" err="Timeout: request did not complete within requested timeout - context deadline exceeded"
E1123 13:56:04.732903 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:04.734103 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:04.735222 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:04.736350 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:04.737610 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.616114ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/external-secrets/leases/external-secrets-controller" result=null
E1123 13:56:05.467665 1 controller.go:163] "Unhandled Error" err="unable to sync kubernetes service: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:05.473234 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:09.024306 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 9.371µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 13:56:09.281106 1 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"apiserver-vlq2olu7fdhe7ulslvmeo2jdyi\": the object has been modified; please apply your changes to the latest version and try again"
I1123 13:56:09.979262 1 controller.go:115] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease"
E1123 13:56:22.063489 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:22.064891 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:22.066001 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:22.067322 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:22.068583 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.918595ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:22.988211 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:22.989412 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:22.990536 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:22.991631 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:22.992857 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.573313ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:23.341723 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:23.343398 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:23.344514 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:23.345603 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:23.346804 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="5.054818ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result=null
E1123 13:56:23.925064 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:23.925087 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 5.961µs, panicked: false, err: context canceled, panic-reason: <nil>" logger="UnhandledError"
E1123 13:56:23.926202 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:23.927301 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:23.928483 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.5103ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/external-secrets/leases/external-secrets-controller" result=null
E1123 13:56:24.339710 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:25.182968 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:25.184130 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:25.185248 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:25.186352 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:25.187570 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.559133ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager" result=null
E1123 13:56:25.681498 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:27.191878 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:27.357723 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:27.380477 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:27.528384 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:27.529931 1 controller.go:195] "Failed to update lease" err="etcdserver: request timed out"
E1123 13:56:28.606070 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:28.925359 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:28.926560 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.927689 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.928785 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.930099 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.695637ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/external-secrets/leases/external-secrets-controller" result=null
E1123 13:56:28.930729 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
E1123 13:56:28.930883 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.930906 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 4.19µs, panicked: false, err: context canceled, panic-reason: <nil>" logger="UnhandledError"
E1123 13:56:28.932024 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.932030 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.933141 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.933177 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.934268 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:28.934403 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.613424ms" method="PATCH" path="/api/v1/namespaces/flux-system/events/flux-system.187aa751e456d9a4" result=null
E1123 13:56:28.935469 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.794321ms" method="GET" path="/api/v1/namespaces/flux-system/secrets/flux-system" result=null
E1123 13:56:30.756052 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
E1123 13:56:30.757252 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:30.758358 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:30.759467 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:30.760783 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.717918ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:30.764862 1 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"apiserver-vlq2olu7fdhe7ulslvmeo2jdyi\": the object has been modified; please apply your changes to the latest version and try again"
E1123 13:56:41.629799 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:41.629858 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 6.72µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 13:56:41.630018 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 2.32µs, panicked: false, err: context canceled, panic-reason: <nil>" logger="UnhandledError"
E1123 13:56:41.630092 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:41.631136 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:41.631161 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:41.632243 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:41.632265 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:41.633453 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.542471ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:41.633540 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.80383ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:42.128489 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 6.12µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 13:56:42.129051 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:42.129468 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:42.130115 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:42.130721 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:42.131217 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:42.132317 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:42.132386 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.988585ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager" result=null
E1123 13:56:42.133429 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:42.134562 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="5.024007ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result=null
E1123 13:56:43.629826 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:47.128247 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
E1123 13:56:47.129428 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.129659 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
E1123 13:56:47.130589 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.130957 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.132508 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.132537 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.133921 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
E1123 13:56:47.134197 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.134276 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="6.0878ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-controller-manager" result=null
E1123 13:56:47.135362 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.136432 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.136541 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="6.803753ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result=null
E1123 13:56:47.137552 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:47.138687 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.795651ms" method="GET" path="/api/v1/namespaces/kube-system/serviceaccounts/validatingadmissionpolicy-status-controller" result=null
E1123 13:56:49.704959 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:49.706195 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:49.707321 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:49.708429 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:49.709664 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.595994ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:50.818282 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:50.819449 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:50.820565 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:50.821681 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:50.822898 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.574323ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:52.130837 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 4.32µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 13:56:52.130842 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:52.131943 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:52.133039 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:52.134243 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.451948ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result=null
E1123 13:56:58.064516 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:58.065712 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:58.066829 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:58.067942 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:58.069128 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.568523ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:59.098367 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:56:59.099554 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:59.100671 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:56:59.101782 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:56:59.102998 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.562962ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:56:59.246776 1 controller.go:163] "Unhandled Error" err="unable to sync kubernetes service: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:59.248855 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:59.250236 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:56:59.250278 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:57:25.183449 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"context canceled\"}: context canceled" logger="UnhandledError"
E1123 13:57:25.184629 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:57:25.185748 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:57:25.186863 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:57:25.188129 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.705937ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:57:27.933245 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:57:27.987961 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:57:27.989487 1 controller.go:195] "Failed to update lease" err="etcdserver: request timed out"
E1123 13:57:28.232195 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:57:28.336933 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:57:28.336946 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: rpctypes.EtcdError{code:0xe, desc:\"etcdserver: request timed out\"}: etcdserver: request timed out" logger="UnhandledError"
E1123 13:57:28.369288 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:57:28.369295 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 8.43µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 13:57:28.370398 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:57:28.371436 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:57:28.373704 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.646745ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
E1123 13:57:28.709832 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: context.deadlineExceededError{}: context deadline exceeded" logger="UnhandledError"
E1123 13:57:28.711491 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:57:28.712602 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 13:57:28.713705 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 13:57:28.714928 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="5.020027ms" method="GET" path="/apis/coordination.k8s.io/v1/namespaces/external-secrets/leases/external-secrets-controller" result=null
E1123 13:57:30.639667 1 controller.go:195] "Failed to update lease" err="Operation cannot be fulfilled on leases.coordination.k8s.io \"apiserver-vlq2olu7fdhe7ulslvmeo2jdyi\": the object has been modified; please apply your changes to the latest version and try again"
E1123 14:09:37.502924 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.502925 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 5.94µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 14:09:37.504056 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.505735 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.507002 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.104488ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/kube-scheduler" result=null
E1123 14:09:37.507039 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.507053 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 3.68µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 14:09:37.507137 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.507153 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 2.25µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 14:09:37.507947 1 writers.go:123] "Unhandled Error" err="apiserver was unable to write a JSON response: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.508015 1 finisher.go:175] "Unhandled Error" err="FinishRequest: post-timeout activity - time-elapsed: 2.71µs, panicked: false, err: context deadline exceeded, panic-reason: <nil>" logger="UnhandledError"
E1123 14:09:37.508160 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.508179 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.509125 1 status.go:71] "Unhandled Error" err="apiserver received an error that is not an metav1.Status: &errors.errorString{s:\"http: Handler timeout\"}: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.509308 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.509464 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.510476 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.392516ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/flux-system/leases/flux-operator" result=null
E1123 14:09:37.510557 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="3.600663ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/external-secrets/leases/external-secrets-controller" result=null
E1123 14:09:37.510961 1 writers.go:136] "Unhandled Error" err="apiserver was unable to write a fallback JSON response: http: Handler timeout" logger="UnhandledError"
E1123 14:09:37.512108 1 timeout.go:140] "Post-timeout activity" logger="UnhandledError" timeElapsed="4.140318ms" method="PUT" path="/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/cilium-operator-resource-lock" result=null
pod kube-controller-manager-pks-share-cp
│ I1123 14:02:15.333933 1 node_lifecycle_controller.go:1234] "Initializing eviction metric for zone" logger="node-lifecycle-controller" zone="" │
│ I1123 14:02:15.333977 1 range_allocator.go:183] "Starting range CIDR allocator" logger="node-ipam-controller" │
│ I1123 14:02:15.333979 1 node_lifecycle_controller.go:886] "Missing timestamp for Node. Assuming now as a timestamp" logger="node-lifecycle-controller" node="pks-share-wk2" │
│ I1123 14:02:15.333982 1 shared_informer.go:313] Waiting for caches to sync for cidrallocator │
│ I1123 14:02:15.333999 1 shared_informer.go:320] Caches are synced for cidrallocator │
│ I1123 14:02:15.334042 1 node_lifecycle_controller.go:886] "Missing timestamp for Node. Assuming now as a timestamp" logger="node-lifecycle-controller" node="pks-share-wk1" │
│ I1123 14:02:15.334051 1 node_lifecycle_controller.go:886] "Missing timestamp for Node. Assuming now as a timestamp" logger="node-lifecycle-controller" node="pks-share-cp" │
│ I1123 14:02:15.334073 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk2" │
│ I1123 14:02:15.334084 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-cp" │
│ I1123 14:02:15.334079 1 node_lifecycle_controller.go:1080] "Controller detected that zone is now in new state" logger="node-lifecycle-controller" zone="" newState="Normal" │
│ I1123 14:02:15.334101 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk1" │
│ I1123 14:02:15.335249 1 shared_informer.go:320] Caches are synced for resource quota │
│ I1123 14:02:15.337370 1 shared_informer.go:320] Caches are synced for expand │
│ I1123 14:02:15.338567 1 shared_informer.go:320] Caches are synced for ClusterRoleAggregator │
│ I1123 14:02:15.339807 1 shared_informer.go:320] Caches are synced for job │
│ I1123 14:02:15.342037 1 shared_informer.go:320] Caches are synced for certificate-csrsigning-kubelet-client │
│ I1123 14:02:15.342070 1 shared_informer.go:320] Caches are synced for certificate-csrsigning-kubelet-serving │
│ I1123 14:02:15.343140 1 shared_informer.go:320] Caches are synced for certificate-csrsigning-kube-apiserver-client │
│ I1123 14:02:15.343161 1 shared_informer.go:320] Caches are synced for certificate-csrsigning-legacy-unknown │
│ I1123 14:02:15.345320 1 shared_informer.go:320] Caches are synced for certificate-csrapproving │
│ I1123 14:02:15.351511 1 shared_informer.go:320] Caches are synced for HPA │
│ I1123 14:02:15.352643 1 shared_informer.go:320] Caches are synced for resource quota │
│ I1123 14:02:15.353677 1 shared_informer.go:320] Caches are synced for disruption │
│ I1123 14:02:15.355858 1 shared_informer.go:320] Caches are synced for legacy-service-account-token-cleaner │
│ I1123 14:02:15.356986 1 shared_informer.go:320] Caches are synced for endpoint │
│ I1123 14:02:15.363496 1 shared_informer.go:320] Caches are synced for garbage collector │
│ I1123 14:04:42.011668 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-cp" │
│ I1123 14:05:09.176375 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk2" │
│ I1123 14:05:11.890954 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk1" │
│ I1123 14:09:49.409012 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-cp" │
│ I1123 14:10:15.140638 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk2" │
│ I1123 14:10:19.720270 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk1" │
│ I1123 14:14:55.525373 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-cp" │
│ I1123 14:15:20.976962 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk2" │
│ I1123 14:15:25.453922 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk1" │
│ I1123 14:20:00.867847 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-cp" │
│ I1123 14:20:27.508014 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk2" │
│ I1123 14:20:31.180503 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk1" │
│ I1123 14:25:06.513713 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-cp" │
│ I1123 14:25:34.528075 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk2" │
│ I1123 14:25:37.234529 1 range_allocator.go:247] "Successfully synced" logger="node-ipam-controller" key="pks-share-wk1
pod kube-scheduler-pks-share-cp
I1123 14:02:00.100723 1 serving.go:386] Generated self-signed cert in-memory
I1123 14:02:00.553489 1 server.go:166] "Starting Kubernetes Scheduler" version="v1.32.0"
I1123 14:02:00.553506 1 server.go:168] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I1123 14:02:00.556904 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1123 14:02:00.556926 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1123 14:02:00.557021 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController
I1123 14:02:00.557215 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController
I1123 14:02:00.557435 1 secure_serving.go:213] Serving securely on 127.0.0.1:10259
I1123 14:02:00.557621 1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I1123 14:02:00.557437 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1123 14:02:00.557714 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1123 14:02:00.657141 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1123 14:02:00.657986 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1123 14:02:00.657985 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController
I1123 14:02:00.658110 1 leaderelection.go:257] attempting to acquire leader lease kube-system/kube-scheduler...
I1123 14:02:16.682855 1 leaderelection.go:271] successfully acquired lease kube-system/kube-scheduler
E1123 14:09:37.503221 1 leaderelection.go:429] Failed to update lock optimistically: the server was unable to return a response in the time allotted, but may still be processing the request (put leases.coordination.k8s.io kube-scheduler), falling back to slow path
Event
kube-scheduler-pks-share-cp
I1123 14:02:00.100723 1 serving.go:386] Generated self-signed cert in-memory
I1123 14:02:00.553489 1 server.go:166] "Starting Kubernetes Scheduler" version="v1.32.0"
I1123 14:02:00.553506 1 server.go:168] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I1123 14:02:00.556904 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I1123 14:02:00.556926 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1123 14:02:00.557021 1 requestheader_controller.go:180] Starting RequestHeaderAuthRequestController
I1123 14:02:00.557215 1 shared_informer.go:313] Waiting for caches to sync for RequestHeaderAuthRequestController
I1123 14:02:00.557435 1 secure_serving.go:213] Serving securely on 127.0.0.1:10259
I1123 14:02:00.557621 1 tlsconfig.go:243] "Starting DynamicServingCertificateController"
I1123 14:02:00.557437 1 configmap_cafile_content.go:205] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I1123 14:02:00.557714 1 shared_informer.go:313] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1123 14:02:00.657141 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1123 14:02:00.657986 1 shared_informer.go:320] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I1123 14:02:00.657985 1 shared_informer.go:320] Caches are synced for RequestHeaderAuthRequestController
I1123 14:02:00.658110 1 leaderelection.go:257] attempting to acquire leader lease kube-system/kube-scheduler...
I1123 14:02:16.682855 1 leaderelection.go:271] successfully acquired lease kube-system/kube-scheduler
E1123 14:09:37.503221 1 leaderelection.go:429] Failed to update lock optimistically: the server was unable to return a response in the time allotted, but may still be processing the request (put leases.coordination.k8s.io kube-scheduler), falling back to slow path
kube-controller-manager-pks-share-cp
│ Warning Failed 40m (x2 over 15h) kubelet Error: context deadline exceeded │
│ Warning Failed 39m (x3 over 40m) kubelet Error: failed to reserve container name "kube-controller-manager_kube-controller-manager-pks-share-cp_kube-system_44a3668a5a412b2e88dbfa21382f571c_44": name "kube-controller-manager_kub │
│ e-controller-manager-pks-share-cp_kube-system_44a3668a5a412b2e88dbfa21382f571c_44" is reserved for "79e87b29033916f3bf4726a6bfe01fb2e6a825ba688c58ae11ae710dfe253cd7" │
│ Warning Unhealthy 35m (x32 over 4d2h) kubelet Liveness probe failed: Get "https://localhost:10257/healthz": dial tcp [::1]:10257: connect: connection refused │
│ Warning BackOff 30m (x52 over 4d6h) kubelet Back-off restarting failed container kube-controller-manager in pod kube-controller-manager-pks-share-cp_kube-system(44a3668a5a412b2e88dbfa21382f571c) │
│ Normal Pulled 30m (x50 over 4d6h) kubelet Container image "registry.k8s.io/kube-controller-manager:v1.32.0" already present on machine │
│ Normal Created 30m (x45 over 4d6h) kubelet Created container: kube-controller-manager │
│ Normal Started 30m (x45 over 4d6h) kubelet Started container kube-controller-manager
Environment
- Talos Version : 1.10.5
- Kubernetes version: v1.32.0
- Platform: Proxmox Version : 8.2.4
Contact
If someone want to help me troubleshoot this issue, i can provider more information here or on discord