Using K3s with CRI-O as runtime leads to frequent Interface veth* Mac doesn't match: * not found errors #11870

adelton · 2025-03-03T14:36:55Z

adelton
Mar 3, 2025

Environmental Info:
K3s Version:

k3s version v1.31.6+k3s1 (6ab750f9)
go version go1.22.12

Node(s) CPU architecture, OS, and Version:

Linux test6-ubuntu-24-04.example.com 6.8.0-48-generic #48-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 14:04:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:

Single node, control-plane,master.

Describe the bug:

I try to setup K3s with CRI-O, for CI purposes in a GitHub Actions runner. I follow the spirit of https://devopstales.github.io/kubernetes/k3s-crio/ (referenced from #2741 (comment)), although many things need to be done differently.

The containers gets setup properly and the node is shown as Ready, but attempts to run a Pod (sometimes even Pods like coredns*) often take many attempts to start, producing errors like

k3s[2200]: E0303 14:28:35.396649    2200 kuberuntime_manager.go:1170] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx_default_1f421ebb-ead5-4d9c-919a-ad74d892b926_0(772ddfa9f7ea0c6929b4a0a10a3609bacb640bd23aa20d8fe66c779b6ba890a8): error checking pod default_nginx for CNI network \"crio\": Interface vethe09c95bd Mac doesn't match: 86:39:98:f3:83:77 not found" pod="default/nginx"

Steps To Reproduce:

Have an fresh Ubuntu 24.04 machine (VM).
export CRIO_VERSION=v1.31
curl -fsSL https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/Release.key | gpg --dearmor | sudo tee /etc/apt/keyrings/cri-o-apt-keyring.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/cri-o-apt-keyring.gpg] https://pkgs.k8s.io/addons:/cri-o:/stable:/$CRIO_VERSION/deb/ /" | sudo tee /etc/apt/sources.list.d/cri-o.list
sudo apt update
sudo apt install -y cri-o
sudo cp /etc/cni/net.d/10-crio-bridge.conflist.disabled /etc/cni/net.d/10-crio-bridge.conflist
sudo vi /etc/cni/net.d/10-crio-bridge.conflist and edit the file to contain

{
  "cniVersion": "1.0.0",
  "name": "crio",
  "plugins": [
    {
      "type": "bridge",
      "bridge": "cni0",
      "isGateway": true,
      "ipMasq": true,
      "hairpinMode": true,
      "ipam": {
        "type": "host-local",
        "routes": [
            { "dst": "0.0.0.0/0" }
        ],
        "ranges": [
            [{ "subnet": "10.244.0.0/16" }]
        ]
      }
    }
  ]
}

sudo apt install -y containernetworking-plugins
( echo '[crio.network]' ; echo 'plugin_dir = "/usr/lib/cni"' ) | sudo tee /etc/crio/crio.conf.d/20-cni.conf
sudo systemctl start crio.service
sudo modprobe br_netfilter
sudo sysctl -w net.ipv4.ip_forward=1
export K3S_KUBECONFIG_MODE=644
export INSTALL_K3S_EXEC=" --container-runtime-endpoint /var/run/crio/crio.sock --disable servicelb --disable traefik
curl -sfL https://get.k3s.io | sh -
Check that kubectl get nodes shows the node as Ready.
kubectl run nginx --image=nginx
Observe kubectl get pod/nginx shows it as ContainerCreating for quite a long time.
sudo journalctl -fl | grep -E 'veth|cni'

Expected behavior:

No errors in the journal, Pod Running immediatelly (especially when doing it the second time when the image has already been pulled from the registry).

Actual behavior:

kubectl get pod/nginx shows the Pod as ContainerCreating for quite a long time.

Journal showing messages like

systemd-networkd[512]: vethe09c95bd: Link UP
kernel: cni0: port 4(vethe09c95bd) entered blocking state
kernel: cni0: port 4(vethe09c95bd) entered disabled state
kernel: vethe09c95bd: entered allmulticast mode
kernel: vethe09c95bd: entered promiscuous mode
kernel: cni0: port 4(vethe09c95bd) entered blocking state
kernel: cni0: port 4(vethe09c95bd) entered forwarding state
systemd-networkd[512]: vethe09c95bd: Gained carrier
crio[1881]: time="2025-03-03 14:28:35.339078300Z" level=error msg="Error checking network: Interface vethe09c95bd Mac doesn't match: 86:39:98:f3:83:77 not found"
kernel: cni0: port 4(vethe09c95bd) entered disabled state
systemd-networkd[512]: vethe09c95bd: Link DOWN
systemd-networkd[512]: vethe09c95bd: Lost carrier
kernel: vethe09c95bd (unregistering): left allmulticast mode
kernel: vethe09c95bd (unregistering): left promiscuous mode
kernel: cni0: port 4(vethe09c95bd) entered disabled state
k3s[2200]: E0303 14:28:35.396576    2200 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx_default_1f421ebb-ead5-4d9c-919a-ad74d892b926_0(772ddfa9f7ea0c6929b4a0a10a3609bacb640bd23aa20d8fe66c779b6ba890a8): error checking pod default_nginx for CNI network \"crio\": Interface vethe09c95bd Mac doesn't match: 86:39:98:f3:83:77 not found"
k3s[2200]: E0303 14:28:35.396631    2200 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx_default_1f421ebb-ead5-4d9c-919a-ad74d892b926_0(772ddfa9f7ea0c6929b4a0a10a3609bacb640bd23aa20d8fe66c779b6ba890a8): error checking pod default_nginx for CNI network \"crio\": Interface vethe09c95bd Mac doesn't match: 86:39:98:f3:83:77 not found" pod="default/nginx"
k3s[2200]: E0303 14:28:35.396649    2200 kuberuntime_manager.go:1170] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx_default_1f421ebb-ead5-4d9c-919a-ad74d892b926_0(772ddfa9f7ea0c6929b4a0a10a3609bacb640bd23aa20d8fe66c779b6ba890a8): error checking pod default_nginx for CNI network \"crio\": Interface vethe09c95bd Mac doesn't match: 86:39:98:f3:83:77 not found" pod="default/nginx"
k3s[2200]: E0303 14:28:35.396689    2200 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"nginx_default(1f421ebb-ead5-4d9c-919a-ad74d892b926)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"nginx_default(1f421ebb-ead5-4d9c-919a-ad74d892b926)\\\": rpc error: code = Unknown desc = failed to get network status for pod sandbox k8s_nginx_default_1f421ebb-ead5-4d9c-919a-ad74d892b926_0(772ddfa9f7ea0c6929b4a0a10a3609bacb640bd23aa20d8fe66c779b6ba890a8): error checking pod default_nginx for CNI network \\\"crio\\\": Interface vethe09c95bd Mac doesn't match: 86:39:98:f3:83:77 not found\"" pod="default/nginx" podUID="1f421ebb-ead5-4d9c-919a-ad74d892b926"

every cca 11 seconds, before settling down with

systemd-networkd[512]: veth6fd32e41: Link UP
kernel: cni0: port 4(veth6fd32e41) entered blocking state
kernel: cni0: port 4(veth6fd32e41) entered disabled state
kernel: veth6fd32e41: entered allmulticast mode
kernel: veth6fd32e41: entered promiscuous mode
kernel: cni0: port 4(veth6fd32e41) entered blocking state
kernel: cni0: port 4(veth6fd32e41) entered forwarding state
systemd-networkd[512]: veth6fd32e41: Gained carrier
systemd-networkd[512]: veth6fd32e41: Gained IPv6LL

and the Pod finally gets Running.

Additional context / logs:

I'm not sure if the ranges subnet in the /etc/cni/net.d/10-crio-bridge.conflist should be matched with --cluster-cidr but even if I do that, the result is still the same.

adelton · 2025-03-03T16:25:37Z

adelton
Mar 3, 2025
Author

It seems the problem is the containernetworking-plugins package which got installed from the stock Ubuntu 24 repository in version 1.1.1+ds1-3ubuntu0.24.04.2.

When I install kubernetes-cni instead (version 1.5.1-1.1), things seem to work fine.

Is there a plan to have a page describing the necessary prerequisites / minimal setup for using K3s with CRI-O? That https://devopstales.github.io/kubernetes/k3s-crio/ seems to come from https://github.com/devopstales/devopstales.github.io but looking at the open issues there like devopstales/devopstales.github.io#30, it does not seem to be actively maintained ...

9 replies

brandond Mar 3, 2025
Collaborator

I don't see why you need to provide your own cni-plugins, can't you tell cri-o to use the ones packaged with k3s?

adelton Mar 3, 2025
Author

That is indeed something I did not consider -- let me try that.

brandond Mar 3, 2025
Collaborator

The binaries are in /var/lib/rancher/k3s/data/cni if that helps any. See https://docs.k3s.io/networking/multus-ipams

adelton Mar 3, 2025
Author

Nod, something like

( echo '[crio.network]' ; echo 'plugin_dirs = [ "/var/lib/rancher/k3s/data/cni" ]' ) | sudo tee /etc/crio/crio.conf.d/20-cni.conf
sudo systemctl restart crio.service

seems to do the trick. I will spend some more time to make it stable -- the timing seems important as CRI-O keeps reporting that it couldn't find the plugin even if in the meantime, after K3s installation and setup, it appeared in the declared directory.

adelton Mar 14, 2025
Author

We now include the K3s with CRI-O setup in GitHub Actions workflow of https://github.com/freeipa/freeipa-container, via freeipa/freeipa-container@32e1d49. We use the
/var/lib/rancher/k3s/data/cni plugins and they work fine: https://github.com/freeipa/freeipa-container/actions/runs/13856319050/job/38774031678

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using K3s with CRI-O as runtime leads to frequent Interface veth* Mac doesn't match: * not found errors #11870

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 9 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Using K3s with CRI-O as runtime leads to frequent Interface veth* Mac doesn't match: * not found errors #11870

adelton Mar 3, 2025

Replies: 1 comment · 9 replies

adelton Mar 3, 2025 Author

brandond Mar 3, 2025 Collaborator

adelton Mar 3, 2025 Author

brandond Mar 3, 2025 Collaborator

adelton Mar 3, 2025 Author

adelton Mar 14, 2025 Author

adelton
Mar 3, 2025

Replies: 1 comment 9 replies

adelton
Mar 3, 2025
Author

brandond Mar 3, 2025
Collaborator

adelton Mar 3, 2025
Author

brandond Mar 3, 2025
Collaborator

adelton Mar 3, 2025
Author

adelton Mar 14, 2025
Author