K3s never load containers

[wviana]

I got this in my log, last messages keep repeating for ever.

# journalctl -u k3s full output

systemd[1]: Started Lightweight Kubernetes.
s[1859051]: I0607 1859051 tlsconfig.go:240] "Starting DynamicServingCertificateController
s[1859051]: I0607 1859051 autoregister_controller.go:141] Starting autoregister controller
s[1859051]: I0607 1859051 controller.go:83] Starting OpenAPI AggregationController
s[1859051]: I0607 1859051 apf_controller.go:317] Starting API Priority and Fairness config controller
s[1859051]: I0607 1859051 dynamic_serving_content.go:131] "Starting controller" name="aggregator-proxy-cert::/var/lib/rancher/k3s/server/tls/client-auth-proxy.crt::/var/lib/rancher/k3s/server/tls/client-auth-proxy.key
s[1859051]: I0607 1859051 cache.go:32] Waiting for caches to sync for autoregister controller
s[1859051]: I0607 1859051 apiservice_controller.go:97] Starting APIServiceRegistrationController
s[1859051]: I0607 1859051 cache.go:32] Waiting for caches to sync for APIServiceRegistrationController controller
s[1859051]: I0607 1859051 available_controller.go:491] Starting AvailableConditionController
s[1859051]: I0607 1859051 cache.go:32] Waiting for caches to sync for AvailableConditionController controller
s[1859051]: I0607 1859051 cluster_authentication_trust_controller.go:440] Starting cluster_authentication_trust_controller controller
s[1859051]: I0607 1859051 shared_informer.go:240] Waiting for caches to sync for cluster_authentication_trust_controller
s[1859051]: I0607 1859051 customresource_discovery_controller.go:209] Starting DiscoveryController
...
...
s[1859051]: certificate CN=system:node:<my-hostname>,O=system:nodes signed by CN=k3s-client-ca@1653867140: notBefore=<recent_date> notAfter=<recent_date>
s[1859051]: Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: no such device
s[1859051]: Waiting for control-plane node <my-hostname> startup: nodes \"<my-hostname>\" not found
s[1859051]: certificate CN=<my-hostname> signed by CN=k3s-server-ca@1653867140: notBefore=<recent_date> notAfter=<recent_date>
s[1859051]: Waiting for control-plane node <my-hostname> startup: nodes \"<my-hostname>\" not found
s[1859051]: certificate CN=system:node:<my-hostname>,O=system:nodes signed by CN=k3s-client-ca@1653867140: notBefore=<recent_date> notAfter=<recent_date>
s[1859051]: Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: no such device
s[1859051]: Waiting for control-plane node <my-hostname> startup: nodes \"<my-hostname>\" not found
s[1859051]: W0607 1859051 handler_proxy.go:104] no RequestInfo found in the context
s[1859051]: E0607 1859051 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
s[1859051]: , Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
s[1859051]: I0607 1859051 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
s[1859051]: certificate CN=<my-hostname> signed by CN=k3s-server-ca@1653867140: notBefore=<recent_date> notAfter=<recent_date>
s[1859051]: certificate CN=system:node:<my-hostname>,O=system:nodes signed by CN=k3s-client-ca@1653867140: notBefore=<recent_date> notAfter=<recent_date>
s[1859051]: Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: no such device
s[1859051]: Waiting for control-plane node <my-hostname> startup: nodes \"<my-hostname>\" not found

Don't matter how much I wait for, I always get this same output to listing kubernet things.
From # kubectl get all -A

NAMESPACE     NAME                                          READY   STATUS    RESTARTS   AGE
kube-system   pod/helm-install-traefik-crd-8dffs            0/1     Pending   0          12d
kube-system   pod/helm-install-traefik-hph8c                0/1     Pending   0          12d
kube-system   pod/metrics-server-7cd5fcb6b7-m74qr           0/1     Pending   0          12d
kube-system   pod/local-path-provisioner-6c79684f77-mb5nq   0/1     Pending   0          12d
kube-system   pod/coredns-d76bd69b-p76ds                    0/1     Pending   0          12d

NAMESPACE     NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                  AGE
default       service/kubernetes       ClusterIP   10.43.0.1      <none>        443/TCP                  12d
kube-system   service/kube-dns         ClusterIP   10.43.0.10     <none>        53/UDP,53/TCP,9153/TCP   12d
kube-system   service/metrics-server   ClusterIP   10.43.106.41   <none>        443/TCP                  12d

NAMESPACE     NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
kube-system   deployment.apps/coredns                  0/1     1            0           12d
kube-system   deployment.apps/local-path-provisioner   0/1     1            0           12d
kube-system   deployment.apps/metrics-server           0/1     1            0           12d

NAMESPACE     NAME                                                DESIRED   CURRENT   READY   AGE
kube-system   replicaset.apps/metrics-server-7cd5fcb6b7           1         1         0       12d
kube-system   replicaset.apps/coredns-d76bd69b                    1         1         0       12d
kube-system   replicaset.apps/local-path-provisioner-6c79684f77   1         1         0       12d

NAMESPACE     NAME                                 COMPLETIONS   DURATION   AGE
kube-system   job.batch/helm-install-traefik-crd   0/1           12d        12d
kube-system   job.batch/helm-install-traefik       0/1           12d        12d

Also tried to install containerd and lxc in host machine.

# pacman -S lxc containerd

[brandond]

s[1859051]: Waiting to retrieve agent configuration; server is not ready: \"overlayfs\" snapshotter cannot be enabled for \"/var/lib/rancher/k3s/agent/containerd\", try using \"fuse-overlayfs\" or \"native\": failed to mount overlay: no such device

What Linux distribution is this on? Are you missing features from your kernel? Does the output of k3s check-config show anything missing?

[ChristianCiach]

How did you install k3s? Did you use a package from the AUR?

[ChristianCiach]

This may be a long shot, but I've seen this happening: Did you make sure to reboot your system after your latest system update? I've seen similar errors after updating the kernel or kernel modules when I forgot to reboot.

[wviana]

Did you make sure to reboot your system after your latest system update? I've seen similar errors after updating the kernel or kernel modules when I forgot to reboot.

Probably that's it. As this machine is encrypted, think after finishing its setup I've never restarted it. As to reboot I would need to connect by provider panel VNC to type disk encryption password.

How did you install k3s? Did you use a package from the AUR?

No, from that script you pipe from curl into sh.

[wviana]

Hey @ChristianCiach ,
That was it. After restart it's working.
For coincidence I had a similar problem with my workstation some days ago. Calibre was not detecting my Kindle. This after long time without restarting, also having more than one kernel update between.

This made me wonder, is there a kernel "hot swap"? Would it be a feature of microservices kernel like Hurd?
Actually, as it didn't change current memory loaded kernel, why does it stops to work property?

Please don't mind about last questions, just taking notes on some thoughts.

So, thank you very much for the help. 😄

[ChristianCiach]

This made me wonder, is there a kernel "hot swap"?

There are some projects that enable live patching of the Linux Kernel, but these come with a ton of caveats.

Actually, as it didn't change current memory loaded kernel, why does it stops to work property?

I am not an expert on this, but as far as I know, after updating the kernel and its modules (but before a reboot) modprobe thinks you are already using the new kernel and thus tries to load the newer version of the module, which is incompatible with the currently loaded kernel.

At first I didn't understand why you were even able to start K3s at all, since K3s tries to load the overlay-module on startup. I think this is because of these lines in the k3s.service file:

ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay

According to https://www.freedesktop.org/software/systemd/man/systemd.service.html:

If the executable path is prefixed with "-", an exit code of the command normally considered a failure (i.e. non-zero exit status or abnormal exit due to signal) is recorded, but has no further effect and is considered equivalent to success.

So, modprobe probably already failed at this point, but thanks to the -, K3s continued to startup anyway.

[wviana]

Just to keep the answer easier if someone find this discussion.

@ChristianCiach reply

Did you make sure to reboot your system after your latest system update? I've seen similar errors after updating the kernel or kernel modules when I forgot to reboot.

That was it, I hadn't restart after kernel updates for a long time.