Description
System that I use:
Kubernetes v1.27.13
Helm v3.14.4
Rancher v2.8.1 (chart rancher-2.8.1)
Rnacher Webhook v0.4.2 (chart rancher-webhook-103.0.1+up0.4.2)
Sometimes I got this error when doing helm upgrade command:
Error: UPGRADE FAILED: create: failed to create: Internal error occurred: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "[https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=30s](https://rancher-webhook.cattle-system.svc/v1/webhook/mutation/secrets?timeout=30s)": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
The default --timeout option from helm upgrade command is 5m (300 seconds). But the default mutatingwebhookconfigurations.admissionregistration.k8s.io for rancher (rancher.cattle.io) is 15s like below:
rancher.cattle.io Mutating Webhook
- admissionReviewVersions:
- v1
- v1beta1
clientConfig:
caBundle: <hidden>
service:
name: rancher-webhook
namespace: cattle-system
path: /v1/webhook/mutation/secrets
port: 443
failurePolicy: Fail
matchPolicy: Equivalent
name: rancher.cattle.io.secrets
namespaceSelector: {}
objectSelector: {}
reinvocationPolicy: Never
rules:
- apiGroups:
- ""
apiVersions:
- v1
operations:
- CREATE
- DELETE
resources:
- secrets
scope: Namespaced
sideEffects: NoneOnDryRun
timeoutSeconds: 15
That means rancher cancel the request because hit the timeout limit. Sometimes helm chart take longer when we have several helm release in one namespace, especially when one release have several history. the helm release history also stored in Kubernetes secret and I think this related with current case. I try to increase the timeout into 300, but the possible value is between 1 and 30.
The MutatingWebhookConfiguration "rancher.cattle.io" is invalid: webhooks[4].timeoutSeconds: Invalid value: 300: the timeout value must be between 1 and 30 seconds
The question is, it's there a another way to solve my case ?. Currently the options to solve this is delete that rancher.cattle.io Mutating Webhook. But in several time, that will automatically created again.