failed to find plugin "multi-config" in path [/opt/cni/bin /opt/cni/bin]

[cyclinder]

• checked https://foundation-model-stack.github.io/multi-nic-cni/user_guide/troubleshooting/.
• titled with the bug issue (if applicable).
• provided corresponding information regarding the troubleshooting guidelines (CR list/detail, multi-nic cni controller and/or daemon status/log).

Describe the bug
A clear and concise description of what the bug is.

  Warning  FailedSync              <invalid> (x2 over <invalid>)    kubelet            error determining status: rpc error: code = Unknown desc = Error: No such container: 53289028945b59db07e6d499f7aa62b2c72149d590218423b59a6ea1cb3a462f
  Warning  FailedCreatePodSandBox  <invalid> (x448 over <invalid>)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "30bff6b9dbb3aa9c081cc463503bc04cb259f0f0554e2f17ec6177ff097610c6" network for pod "dao-2048-6fcc685d6f-njk6w": networkPlugin cni failed to set up pod "dao-2048-6fcc685d6f-njk6w_default" network: [default/dao-2048-6fcc685d6f-njk6w/:multinic-multiconfig]: error adding container to network "multinic-multiconfig": failed to find plugin "multi-config" in path [/opt/cni/bin /opt/cni/bin]

To Reproduce
Steps to reproduce the behavior:

1. create a multinetwork:

apiVersion: multinic.fms.io/v1
kind: MultiNicNetwork
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"multinic.fms.io/v1","kind":"MultiNicNetwork","metadata":{"annotations":{},"name":"multinic-multiconfig"},"spec":{"ipam":"{\n  \"type\": \"multi-config\",\n  \"ipam_type\": \"whereabouts\",\n  \"args\": {\n    \"p0\": {\n      \"range\": \"192.168.0.0/18\"\n    },\n    \"p1\": {\n      \"range\": \"192.168.64.0/18\"\n    }\n  }\n}\n","multiNICIPAM":false,"plugin":{"args":{"mode":"l2"},"cniVersion":"0.3.0","type":"ipvlan"},"subnet":""}}
  creationTimestamp: "2024-05-22T07:47:30Z"
  finalizers:
  - finalizers.multinicnetwork.multinic.fms.io
  generation: 2
  name: multinic-multiconfig
  resourceVersion: "259834695"
  uid: 10435885-5348-4b12-96f5-42fd15c500f1
spec:
  attachPolicy:
    strategy: ""
  ipam: |
    {
      "type": "multi-config",
      "ipam_type": "whereabouts",
      "args": {
        "p0": {
          "range": "192.168.0.0/18"
        },
        "p1": {
          "range": "192.168.64.0/18"
        }
      }
    }
  plugin:
    args:
      mode: l2
    cniVersion: 0.3.0
    dns: {}
    type: ipvlan

2. start a pod with the multus annotaions

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

• manager container of controller and multi-nicd DS status:
• multinicnetwork CR:
• hostinterface list/CR:
• cidr CR (multiNICIPAM: true):
• ippools CR (multiNICIPAM: true):
• log of manager container:
• log of failed multi-nicd pod:

Environment (please complete the following information):

• platform: [e.g. self-managed k8s, self-managed OpenShift, EKS, IKS, AKS]
• node profile:
• operator version :
• cluster scale (number of nodes, pods, interfaces):

Additional context
Add any other context about the problem here.

[cyclinder]

docs say: the feature is already available, but I can't find the binary. Could you guys help me?

[sunya-ch]

@cyclinder I greatly apologize for very late reply.
Could you please provide the version that you have been installed for both controller and multi-nicd?
The error reports multi-config is not recognized. It should be available after v1.2.2 version: https://github.com/foundation-model-stack/multi-nic-cni/blob/main/document/docs/release/alpha.md#v122.