Root cause of high workqueue latency and slow scaling?

[kwngo]

Reaching out to the community for help and ideas 😄

We're running actions-runner-controller and recently started seeing degregaded scaling performance. We're running about 300-400 concurrent workflows at any given time. We're running against Github Enterprise and scaling on PercentageRunnersBusy

In the cluster, we're seeing pods not quickly reflecting change to desired in runnerreplicas, and so we can't scale up quickly enough to process our workflow queue. Metrics are reporting high reconciler latency and workqueue duration which we think may be the root cause. See below for controller_runtime_reconcile_time_seconds and workqueue_work_duration_seconds metric

Just to cover all bases, another error we see in the logs is runner is busy and issues with list runners in runner reconciler such as "failed to list runners: invalid character '<' looking for beginning of value"

2022-01-26T23:58:06.219Z	ERROR	controller-runtime.manager.controller.runner-controller	Reconciler error	{"reconciler group": "actions.summerwind.dev", "reconciler kind": "Runner", "name": "default-ci-runner-72w4z-4dpbp", "namespace": "actions-runner-system", "error": "runner is busy"}

2022-01-21T17:52:41.424Z	ERROR	controller-runtime.manager.controller.runner-controller	Reconciler error	{"reconciler group": "actions.summerwind.dev", "reconciler kind": "Runner", "name": "default-ci-runner-n6fzn-nprrq", "namespace": "actions-runner-system", "error": "failed to list runners: invalid character '<' looking for beginning of value"}

Another thing we noticed was when we are scaling in, it takes up to 30mins after the mutating webhook has already received the validation request to finally create the pod. From the controller logs we can see the following order of events for our runner

Any ideas much appreciated!

[cbui]

It looks like the sync period that was configured was too low.