Liqo Network blocked when liqo-controller-manager pod restart then worker node reboot

[huntkalio]

Is there an existing issue for this?

• I have searched the existing issues

Version

1.0.1

What happened?

I have two k8s cluster(cluser A, cluster B).Both CNI is flannel(Kernel : 6.1.0-32-amd64).I use liqo to peer two cluster.Now cluster A pod can connect cluster B pod.A cluster as consumer cluster,B cluster as remote provider cluster.
Then I reboot some nodes in cluster A , after a moment , wait all pod restart.Then restart nodes pod can't connect cluster B pod but below show all things right

 liqoctl info peer 
┌─ Peer cluster info ──────────────────────────────────────────────────────────────┐
|  Cluster ID: old-v5                                                              |
|  Role:       Provider                                                            |
└──────────────────────────────────────────────────────────────────────────────────┘
┌─ Network ────────────────────────────────────────────────────────────────────────┐
|  Status: Healthy                                                                 |
|  CIDR                                                                            |
|      Remote                                                                      |
|          Pod CIDR:      172.22.0.0/16 → Remapped to 172.22.0.0/16                |
|          External CIDR: 10.70.0.0/16 → Remapped to 10.71.0.0/16                  |
|  Gateway                                                                         |
|      Role:    Server                                                             |
|      Address: 172.30.97.106                                                      |
|      Port:    32359                                                              |
└──────────────────────────────────────────────────────────────────────────────────┘
┌─ Authentication ─────────────────────────────────────────────────────────────────┐
|  Status:     Healthy                                                             |
|  API server: https://172.30.97.105:6443                                          |
|  Resource slices                                                                 |
|      old-v5                                                                      |
|          Action: Consuming                                                       |
|          Resource slice accepted                                                 |
|          Resources                                                               |
|              pods:              110                                              |
|              cpu:               4                                                |
|              ephemeral-storage: 20Gi                                             |
|              memory:            8Gi                                              |
└──────────────────────────────────────────────────────────────────────────────────┘
┌─ Offloading ─────────────────────────────────────────────────────────────────────┐
|  Status: Healthy                                                                 |
|  Virtual nodes                                                                   |
|      old-v5                                                                      |
|          Status:         Healthy                                                 |
|          Secret:         kubeconfig-resourceslice-old-v5                         |
|          Resource slice: old-v5                                                  |
|          Resources                                                               |
|              pods:              110                                              |
|              cpu:               4                                                |
|              ephemeral-storage: 20Gi                                             |
|              memory:            8Gi                                              |
└──────────────────────────────────────────────────────────────────────────────────┘


 kubectl get connection -A
NAMESPACE            NAME        TYPE     STATUS      AGE
liqo-tenant-old-v5   gw-old-v5   Client   Connected   2d

Then I use https://github.com/liqotech/liqo/blob/master/docs/faq/faq.md to Sniff the traffic inside the gateway, And found that:
Cluster A(consumer cluster): gateway pod can't receive the ping ( when cluster A pod ping cluster B pod)
Cluster B(provider cluster): gateway pod can receive the ping ( when cluster B pod ping cluster A pod)

If I kill cluster A gateway pod wait another gateway pod start, The network will recovery.cluster A restart node pod can connect cluster B pod

Relevant log output

How can we reproduce the issue?

1.create cluster A and B
2.use liqo to peer two cluser
3.restart worker node in cluater A.

Provider or distribution

rancher

CNI version

flannel v1.4.1-rancher1

Kernel Version

6.1.0-32

Kubernetes Version

1.30

Code of Conduct

• I agree to follow this project's Code of Conduct

[huntkalio]

And when cluster A restart pod ping cluster B pod,tcpdump on gateway pod:

tcpdump -i any -nn "udp port 6091" -vvv -e
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
03:37:13.059639 eth0  In  ifindex 2 66:bf:db:78:35:87 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 33356, offset 0, flags [none], proto UDP (17), length 78)
    172.23.0.64.35472 > 172.23.1.135.6091: [no cksum] UDP, length 50
03:37:14.095102 eth0  In  ifindex 2 66:bf:db:78:35:87 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 33458, offset 0, flags [none], proto UDP (17), length 78)
    172.23.0.64.35472 > 172.23.1.135.6091: [no cksum] UDP, length 50
03:37:15.119098 eth0  In  ifindex 2 66:bf:db:78:35:87 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 33613, offset 0, flags [none], proto UDP (17), length 78)
    172.23.0.64.35472 > 172.23.1.135.6091: [no cksum] UDP, length 50
03:37:16.143184 eth0  In  ifindex 2 66:bf:db:78:35:87 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 33833, offset 0, flags [none], proto UDP (17), length 78)
    172.23.0.64.35472 > 172.23.1.135.6091: [no cksum] UDP, length 50
03:37:17.167127 eth0  In  ifindex 2 66:bf:db:78:35:87 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 33994, offset 0, flags [none], proto UDP (17), length 78)
    172.23.0.64.35472 > 172.23.1.135.6091: [no cksum] UDP, length 50
03:37:18.191060 eth0  In  ifindex 2 66:bf:db:78:35:87 ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 34024, offset 0, flags [none], proto UDP (17), length 78)
    172.23.0.64.35472 > 172.23.1.135.6091: [no cksum] UDP, length 50

geneve can receive packet

 tcpdump -i liqo-tunnel -nn
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on liqo-tunnel, link-type RAW (Raw IP), snapshot length 262144 bytes
03:38:09.795334 IP 169.254.18.2.12345 > 169.254.18.1.12345: UDP, length 84
03:38:09.798058 IP 169.254.18.1.12345 > 169.254.18.2.12345: UDP, length 84
03:38:10.768428 IP 169.254.18.1.12345 > 169.254.18.2.12345: UDP, length 84
03:38:10.768501 IP 169.254.18.2.12345 > 169.254.18.1.12345: UDP, length 84

liqo-tunnel can not receive packet

when the network is block ,below is the link on cluser A gateway pod:

gw-old-v5-57747c49db-nxw45:/# ip -d link show dev liqo.tmgzqkd7lp
5: liqo.tmgzqkd7lp: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1340 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether aa:bd:a5:13:10:3c brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65485 
    geneve id 5 remote 172.23.0.128 ttl auto dstport 6091 noudpcsum udp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 
gw-old-v5-57747c49db-nxw45:/# ip -d link show dev liqo.zrfk4bkq6w
4: liqo.zrfk4bkq6w: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1340 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether a2:6c:64:b4:1e:18 brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65485 
    geneve id 4 remote 172.23.0.64 ttl auto dstport 6091 noudpcsum udp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 
gw-old-v5-57747c49db-nxw45:/# ip -d link show dev liqo.85jvs44mcg
6: liqo.85jvs44mcg: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1340 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 7e:3e:2f:73:dc:5e brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65485 
    geneve id 6 remote 172.23.1.129 ttl auto dstport 6091 noudpcsum udp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 

when I kill the gatway pod , then network recovery ,below is the link on cluser A new gateway pod:

gw-old-v5-57747c49db-wkfhg:/# ip -d link show dev liqo.zrfk4bkq6w
4: liqo.zrfk4bkq6w: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1340 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 6a:86:67:30:1b:4d brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65485 
    geneve id 1 remote 172.23.0.64 ttl auto dstport 6091 noudpcsum udp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 
gw-old-v5-57747c49db-wkfhg:/# ip -d link show dev liqo.tmgzqkd7lp
5: liqo.tmgzqkd7lp: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1340 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 56:af:64:b1:50:cd brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65485 
    geneve id 2 remote 172.23.0.128 ttl auto dstport 6091 noudpcsum udp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 
gw-old-v5-57747c49db-wkfhg:/# ip -d link show dev liqo.85jvs44mcg
6: liqo.85jvs44mcg: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1340 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ce:12:c0:ba:71:18 brd ff:ff:ff:ff:ff:ff promiscuity 0 allmulti 0 minmtu 68 maxmtu 65485 
    geneve id 7 remote 172.23.1.129 ttl auto dstport 6091 noudpcsum udp6zerocsumrx numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536 

When network blocked, liked gateway pod geneve id have wrong old value

liqo/pkg/gateway/fabric/geneve/internalnode_controller.go

Line 97 in f085da4

if err := geneve.EnsureGeneveInterfacePresence(

liqo/pkg/fabric/internalfabric_controller.go

Line 111 in f085da4

if err := geneve.EnsureGeneveInterfacePresence(

liqo/pkg/utils/network/geneve/netlink.go

Line 80 in f085da4

if !geneveLink.Remote.Equal(remote) || geneveLink.MTU != mtu || geneveLink.Dport != port {

above code not check the geneve id ,,remoteIP,gatewaIP is same or right

[huntkalio]

I also found when node restart, sometimes internalnodes.networking.liqo.io nodeIP remote wrong,like below:

kubectl get internalnodes.networking.liqo.io -A
NAME       NODE IP LOCAL   NODE IP REMOTE   AGE
test-502   172.23.0.65     172.30.127.93    2d19h
test-503   172.23.0.129    172.23.0.128     2d19h
test-507   172.23.1.129    172.23.1.128     18h

test-502 machine is reboot

ifconfig
cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.23.0.65  netmask 255.255.255.192  broadcast 172.23.0.127
        ether f6:1d:7b:07:e9:a6  txqueuelen 1000  (Ethernet)
        RX packets 10539  bytes 1908415 (1.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10381  bytes 7115250 (6.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.30.127.93  netmask 255.255.240.0  broadcast 172.30.127.255
        ether 00:16:3e:23:ec:7f  txqueuelen 1000  (Ethernet)
        RX packets 26293  bytes 13098028 (12.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22325  bytes 4081396 (3.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 172.23.0.64  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 96:fb:d7:6c:70:78  txqueuelen 0  (Ethernet)
        RX packets 789  bytes 194942 (190.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 841  bytes 150750 (147.2 KiB)
        TX errors 0  dropped 5 overruns 0  carrier 0  collisions 0

liqo.x8mbqrgc6r: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1340
        inet 10.80.0.2  netmask 255.255.255.255  broadcast 0.0.0.0
        ether 4a:33:e3:1d:34:cf  txqueuelen 1000  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 575 (575.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 11448  bytes 3338391 (3.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 11448  bytes 3338391 (3.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

test-502 machine remote ip use eth0 ip when restart, but other use flannel.1 IP.
This may because when restart ,flannel.1 is ready after liqo-fabric-xxxx

ifconfig
docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.222.1  netmask 255.255.255.0  broadcast 192.168.222.255
        ether 6a:10:47:a9:da:e4  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.30.127.93  netmask 255.255.240.0  broadcast 172.30.127.255
        ether 00:16:3e:23:ec:7f  txqueuelen 1000  (Ethernet)
        RX packets 2640  bytes 1678149 (1.6 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2292  bytes 340238 (332.2 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 1390  bytes 1140304 (1.0 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1390  bytes 1140304 (1.0 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

liqo/pkg/fabric/source-detector/gateway_controller.go

Line 92 in f085da4

src, err := GetSrcIPFromDstIP(pod.Status.PodIP)

So, Does we need add some check method to ensure farbic use the right interface(flannel.1 not eth0)?
like pinging gateway pod , and retry until success ,then we can update the remote ip.

[huntkalio]

1.When liqo-controller-manager pod restart, it seems all GeneveTunnels "id" will change, but fabric pod and gateway pod interface geneve id won't change. Why we need chang GeneveTunnels id?

liqo/pkg/liqo-controller-manager/networking/internal-network/id/genevetunnel.go

Line 41 in f085da4

err = geneveTunnelManager.Configure(tunnel.Name, tunnel.Spec.ID)

here like need change to

geneveTunnelManager.Configure(client.ObjectKeyFromObject(tunnel).String(), tunnel.Spec.ID)

2. we can add telnet gateway health check pod to check wether use the right interface.

	src, err := GetSrcIPFromDstIP(pod.Status.PodIP)
	if err != nil {
		return ctrl.Result{}, err
	}

	if !probeTCPPort(pod.Status.PodIP, 8083, 3*time.Second) {
		klog.Infof("TCP probe to %s:8083 failed(src: %s), requeueing", pod.Status.PodIP, src)
		return ctrl.Result{RequeueAfter: time.Second * 2}, nil
	}

[jv4n5e]

I am experiencing a similar behavior; when worker nodes are removed from the cluster and new ones come in, peered connections stop working. Restarting the gateway pods resolves this for me.