Receiving inbound unicast UDP traffic to Multus network interface

[dockerised]

Hello,

Firstly thank you for all the development efforts on Multus!

I was wondering if someone could help me, I've been stuck on an issue for quite a while.

The high level objective is to send/receive Multicast and Unicast UDP traffic on a pod using a multus network interface.

I deploy a Multus network interface to a pod and am looking to direct UDP unicast traffic via my Nginx ingress controller (using NodePort/HostPort).

Outbound unicast UDP traffic from inside the Pod to outside the k8s cluster works just fine, the current issue i'm facing is with ingress Unicast traffic to my pod via a nodePort or HostPort. I tried assigning Multus networks to my ingress deployments but the iptables on the worker node still show the default eth0 routes.

Please see below for details:

Kubernetes distro: Tried Kubeadm and Microk8s
Kubernetes version: 1.29
OS version: Ubuntu 20.04
Kernel version: 5.15.0-102-generic

net-attach-def.yaml

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: unicast-ipvlan
  namespace: default 
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "ipvlan",
      "master": "ens38",
      "mode": "l3",
      "ipam": {
        "type": "host-local",
        "subnet": "10.10.5.0/24",
        "rangeStart": "10.10.5.2",
        "rangeEnd": "10.10.5.80",
        "gateway": "10.10.5.1",
        "routes": [
          {
            "dst": "0.0.0.0/0"
          },
    	    { "dst": "192.168.33.0/24", "gw": "10.10.5.1" }
        ]
      }
    }'

pod-and-service.yaml

apiVersion: v1
kind: Pod
metadata:
  name: ipvlan-test
  namespace: default
  annotations:
    k8s.v1.cni.cncf.io/networks: unicast-ipvlan
  labels:
    app: net-tools
spec:
  containers:
  - image: george7522/net-tools:ubuntu
    command: [ "/bin/sh", "-c" ]
    args: ["sleep 1000000"]
    imagePullPolicy: Always
    name: net-tools
    securityContext:
      privileged: true
    securityContext:
      privileged: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  name: net-tools-test
  namespace: default
spec:
  type: NodePort
  ports:
    - port: 30003
      targetPort: 30003
      protocol: UDP
      nodePort: 30003
  selector:
    app: net-tools

K8s host ip tables output, showing only route table entries for default k8s network on 10.96 where 10.96.5.114 is eth0 of ipvlan-test pod

sudo iptables -t nat -L -n -v | grep -i 30003
    1    60 KUBE-EXT-SK3HBFWQJ4PCWJT2  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/net-tools-test */ tcp dpt:30003
    0     0 KUBE-SVC-SK3HBFWQJ4PCWJT2  tcp  --  *      *       0.0.0.0/0            10.99.130.27         /* default/net-tools-test cluster IP */ tcp dpt:30003
    0     0 KUBE-MARK-MASQ  tcp  --  *      *      !10.96.0.0/12         10.99.130.27         /* default/net-tools-test cluster IP */ tcp dpt:30003
    1    60 KUBE-SEP-LZIC3MXJHT4ASY4P  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/net-tools-test -> 10.96.5.114:30003 */

Any help would be greatly appreciated!!

Kind regards,
George

[george-starfish]

Update:

I managed to find a way to correct the iptable entry on the host using a custom Endpoint, shown below, still no traffic coming through.

apiVersion: v1
kind: Service
metadata:
  name: sriov-test
  namespace: hpsample
spec:
  type: NodePort
  ports:
    - port: 30003
      targetPort: 30003
      protocol: UDP
      nodePort: 30003
---
apiVersion: v1
kind: Endpoints
metadata:
  name: sriov-test
  namespace: hpsample
subsets:
- addresses:
  - ip: "10.10.5.2"  # This is IP of the secondary multus interface
  ports:
  - port: 30003
    protocol: UDP

[george-starfish]

This is now working for me, the final solution was to remove the Endpoints, use a selector in the Service, add route tables entries on 192.168.33.111 (UDP sender) and 192.168.33.163 (K8s worker host)

• Changes carried out on 192.168.33.163 (K8s worker host)

sudo ip route del 10.10.5.0/24
sudo ip route add 10.10.5.0/24 dev ens2f0np0

192.168.33.111 (UDP sender)

sudo ip route del 10.10.5.0/24 via 192.168.33.163
sudo ip route add 10.10.5.0/24 via 192.168.33.163

• Changes to the Multus setup:

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: unicast-ipvlan
  namespace: hpsample 
spec:
  config: '{
      "cniVersion": "0.3.0",
      "type": "ipvlan",
      "master": "ens2f0np0",
      "mode": "l2",
      "ipam": {
        "type": "host-local",
        "subnet": "10.10.5.0/24",
        "rangeStart": "10.10.5.2",
        "rangeEnd": "10.10.5.80",
        "gateway": "10.10.5.1",
        "routes": [
          {
            "dst": "0.0.0.0/0",
            "gw": "10.10.5.1"
          }
        ]
      }
    }'
---
apiVersion: v1
kind: Pod
metadata:
  name: sriov-test
  namespace: hpsample
  annotations:
    # k8s.v1.cni.cncf.io/networks: unicast-ipvlan
    k8s.v1.cni.cncf.io/networks: '[{ "name": "unicast-ipvlan", "ips": [ "10.10.5.10/32" ] }]'
  labels:
    app: net-tools
spec:
  containers:
  - image: george7522/net-tools:ubuntu
    command: [ "/bin/sh", "-c" ]
    args: ["sleep 1000000"]
    imagePullPolicy: Always
    name: ffplay
    # resources:
    #   limits:
    #     intel.com/sriov_device_vmain: "1"
    #   requests:
    #     intel.com/sriov_device_vmain: "1"
    securityContext:
      privileged: true
    securityContext:
      privileged: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always

---
apiVersion: v1
kind: Service
metadata:
  name: sriov-test
  namespace: hpsample
spec:
  type: NodePort
  ports:
    - port: 30003
      targetPort: 30003
      protocol: UDP
      nodePort: 30003
  selector:
    app: net-tools
---