how to use

How to use multus-cni?

Prerequisite

Kubelet must be configured to run with the CNI network plugin. Edit /etc/kubernetes/kubelet file and add --network-plugin=cni flags in KUBELET\_OPTS as shown below:

KUBELET_OPTS="...
--network-plugin-dir=/etc/cni/net.d
--network-plugin=cni
"

Install multus

You could copy binary directory or could use daemonset yaml in multus repository.

copy multus binary

Copy multus binary into CNI binary directory, usually /opt/cni/bin.

$ cp multus /opt/cni/bin

use daemonset

As Quickstart, you could apply as such:

$ cat ./images/{multus-daemonset.yml,flannel-daemonset.yml} | kubectl apply -f -

Set up conf file in /etc/cni/net.d/

You put CNI config file in /etc/cni/net.d. Kubernetes CNI runtime uses the alphabetically first file in the directory. XXX mention that you need to do at all nodes

$ mkdir -p /etc/cni/net.d
$ cat >/etc/cni/net.d/30-multus.conf <<EOF
{
  "name": "multus-cni-network",
  "type": "multus",
  "readinessindicatorfile": "/var/run/flannel/subnet.env",
  "delegates": [
    {
      "XXX_note": "This is example, wrote your CNI config in delegates",
      "XXX_note": "If you use flannel, you also need to run flannel daemonset before!",
      "type": "flannel",
      "name": "flannel.1",
      "delegate": {
        "isDefaultGateway": true
      }
    }
  ],
  "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig"
}
EOF

You can use "clusterNetwork"/"defaultNetworks" instead of "delegates", see for the detail

You should set "kubeconfig" in the config file for NetworkAttachmentDefinition(CRD). For the detail, please take a look into Configuration Reference

XXX mention that you need to do at all nodes

Create ServiceAccount, ClusterRole and its binding

Create resources for multus to access CRD objects as folloiwng command:

$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multus
  namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: multus
rules:
  - apiGroups: ["k8s.cni.cncf.io"]
    resources:
      - '*'
    verbs:
      - '*'
  - apiGroups:
      - ""
    resources:
      - pods
      - pods/status
    verbs:
      - get
      - update
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: multus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: multus
subjects:
- kind: ServiceAccount
  name: multus
  namespace: kube-system
EOF

Set up kubeconfig file

Create kubeconfig at master node as following commands:

$ mkdir -p /etc/cni/multus.d
$ SERVICEACCOUNT_CA=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data."ca.crt"')
$ SERVICEACCOUNT_TOKEN=$(kubectl get secrets -n=kube-system -o json | jq -r '.items[]|select(.metadata.annotations."kubernetes.io/service-account.name"=="multus")| .data.token')
$ KUBERNETES_SERVICE_PROTO=$(kubectl get all -o json | jq -r .items[0].spec.ports[0].name)
$ KUBERNETES_SERVICE_HOST=$(kubectl get all -o json | jq -r .items[0].spec.clusterIP)
$ KUBERNETES_SERVICE_PORT=$(kubectl get all -o json | jq -r .items[0].spec.ports[0].port)
$ cat > /etc/cni/net.d/multus.d/multus.kubeconfig <<EOF
# Kubeconfig file for Multus CNI plugin.
apiVersion: v1
kind: Config
clusters:
- name: local
  cluster:
    server: ${KUBERNETES_SERVICE_PROTOCOL:-https}://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}
    certificate-authority-data: ${SERVICEACCOUNT_CA}
users:
- name: multus
  user:
    token: "${SERVICEACCOUNT_TOKEN}"
contexts:
- name: multus-context
  context:
    cluster: local
    user: multus
current-context: multus-context
EOF

Copy /etc/cni/net.d/multus.d/multus.kubeconfig into other kubernetes nodes

$ scp /etc/cni/net.d/multus.d/multus.kubeconfig ...

Setup CRDs (daemonset automatically does)

If you use daemonset to install multus, skip this section and go to "Create network attachment"

In this section, you need to configure following Kubernetes resources for multus:

NetworkAttachmentDefinition
ServiceAccount
ClusterRole
ClusterRoleBinding

Create NetworkAttachmentDefiniton

Create CRD definition in kubernetes as following command at master node:

cat <<EOF | kubectl create -f -
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: network-attachment-definitions.k8s.cni.cncf.io
spec:
  group: k8s.cni.cncf.io
  version: v1
  scope: Namespaced
  names:
    plural: network-attachment-definitions
    singular: network-attachment-definition
    kind: NetworkAttachmentDefinition
    shortNames:
    - net-attach-def
  validation:
    openAPIV3Schema:
      properties:
        spec:
          properties:
            config:
                 type: string
EOF

Create network attachment definition

NetworkAttachmentDefinition with json CNI config:

cat <<EOF | kubectl create -f -
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: macvlan-conf-1
spec:
  config: '{
            "cniVersion": "0.3.0",
            "type": "macvlan",
            "master": "eth1",
            "mode": "bridge",
            "ipam": {
                "type": "host-local",
                "ranges": [
                    [ {
		         "subnet": "10.10.0.0/16",
                         "rangeStart": "10.10.1.20",
                         "rangeEnd": "10.10.3.50",
                         "gateway": "10.10.0.254"
                    } ]
                ]
            }
        }'
EOF

- net-attach-def with config
- net-attach-def with config file name in /etc/cni/multus.d
- thick plugin case (need to write now?)

Run pod with network annotation

text annotation

- text annotation with additional info (ifname)
- json annotation
- json annotation with additional info (ifname)

XXX: TBD

Verifying pod network

XXX: TBD

Run ifconfig command in Pod:

# kubectl exec -it multus-multi-net-poc -- ifconfig
eth0      Link encap:Ethernet  HWaddr C6:43:7C:09:B4:9C
          inet addr:10.128.0.4  Bcast:0.0.0.0  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:8 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:648 (648.0 B)  TX bytes:42 (42.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

net0      Link encap:Ethernet  HWaddr 06:21:91:2D:74:B9
          inet addr:192.168.42.3  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::421:91ff:fe2d:74b9/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:648 (648.0 B)

net1      Link encap:Ethernet  HWaddr D2:94:98:82:00:00
          inet addr:10.56.217.171  Bcast:0.0.0.0  Mask:255.255.255.0
          inet6 addr: fe80::d094:98ff:fe82:0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:120 (120.0 B)  TX bytes:648 (648.0 B)

north     Link encap:Ethernet  HWaddr BE:F2:48:42:83:12
          inet6 addr: fe80::bcf2:48ff:fe42:8312/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1276 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:95956 (93.7 KiB)  TX bytes:82200 (80.2 KiB)

Interface name	Description
lo	loopback
eth0	weave network interface
net0	Flannel network tap interface
net1	VF0 of NIC 1 assigned to the container by Intel - SR-IOV CNI plugin
north	VF0 of NIC 2 assigned with VLAN ID 210 to the container by SR-IOV CNI plugin

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

how to use

How to use multus-cni?

Prerequisite

Install multus

Set up conf file in /etc/cni/net.d/

Create ServiceAccount, ClusterRole and its binding

Set up kubeconfig file

Setup CRDs (daemonset automatically does)

Create NetworkAttachmentDefiniton

Create network attachment definition

NetworkAttachmentDefinition with json CNI config:

Run pod with network annotation

text annotation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally