[BUG] fast ranges not working out of the box

[dougbtv]

Describe the bug
Using fast ranges results

Expected behavior
The pod should come up, however, it fails with:

  Warning  FailedCreatePodSandBox  7s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_samplepod_default_514048e8-7b94-4cf7-8711-17adb72f0ac7_0(c88719557a2b8b049c0013ae7c62a98a37841d5961b6ab87e87b2b7c2b578969): error adding pod default_samplepod to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"c88719557a2b8b049c0013ae7c62a98a37841d5961b6ab87e87b2b7c2b578969" Netns:"/var/run/netns/e21fd2c7-625b-41fd-80ec-1d23c18c2ef3" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=default;K8S_POD_NAME=samplepod;K8S_POD_INFRA_CONTAINER_ID=c88719557a2b8b049c0013ae7c62a98a37841d5961b6ab87e87b2b7c2b578969;K8S_POD_UID=514048e8-7b94-4cf7-8711-17adb72f0ac7" Path:"" ERRORED: error configuring pod [default/samplepod] networking: [default/samplepod/514048e8-7b94-4cf7-8711-17adb72f0ac7:whereaboutsexample]: error adding container to network "whereaboutsexample": netplugin failed: "2024-11-27T05:09:21+09:00 [debug] Used defaults from parsed flat file config @ /etc/cni/net.d/whereabouts.d/whereabouts.conf\n2024-11-27T05:09:21+09:00 [debug] ADD - IPAM configuration successfully read: {Name:whereaboutsexample Type:whereabouts Routes:[] Addresses:[] IPRanges:[{OmitRanges:[] Range:192.168.2.0/24 RangeStart:192.168.2.0 RangeEnd:<nil>}] OmitRanges:[] DNS:{Nameservers:[] Domain: Search:[] Options:[]} Range: NodeSliceSize:/22 RangeStart:<nil> RangeEnd:<nil> GatewayStr: LeaderLeaseDuration:1500 LeaderRenewDeadline:1000 LeaderRetryPeriod:500 LogFile:/var/log/whereabouts.log LogLevel:debug ReconcilerCronExpression:30 4 * * * OverlappingRanges:true SleepForRace:0 Gateway:<nil> Kubernetes:{KubeConfigPath:/etc/cni/net.d/whereabouts.d/whereabouts.kubeconfig K8sAPIRoot:} ConfigurationPath: PodName:samplepod PodNamespace:default NetworkName:donkeykongjr}\n2024-11-27T05:09:21+09:00 [debug] Beginning IPAM for ContainerID: \"c88719557a2b8b049c0013ae7c62a98a37841d5961b6ab87e87b2b7c2b578969\" - podRef: \"default/samplepod\" - ifName: \"net1\"\n2024-11-27T05:09:21+09:00 [debug] discovered current hostname as: labkubedualhost-node-1\n2024-11-27T05:09:21+09:00 [debug] ipam namespace is kube-system\n2024-11-27T05:09:21+09:00 [error] error getting node slice kube-system/donkeykongjr nodeslicepools.whereabouts.cni.cncf.io \"donkeykongjr\" not found\n2024-11-27T05:09:21+09:00 [error] Failed to create leader elector: nodeslicepools.whereabouts.cni.cncf.io \"donkeykongjr\" not found\n2024-11-27T05:09:21+09:00 [debug] Started leader election\nE1127 05:09:21.599030  105556 runtime.go:79] Observed a panic: \"invalid memory address or nil pointer dereference\" (runtime error: invalid memory address or nil pointer dereference)\ngoroutine 41 [running]:\nk8s.io/apimachinery/pkg/util/runtime.logPanic({0x1636d20, 0x2642050})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:75 +0x85\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00050a380?})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:49 +0x6b\npanic({0x1636d20?, 0x2642050?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0x0, {0x1a88a68, 0xc0004fe2d0})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:206 +0x43\ngithub.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2.1()\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:496 +0x68\ncreated by github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2 in goroutine 40\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:494 +0x109\npanic: runtime error: invalid memory address or nil pointer dereference [recovered]\n\tpanic: runtime error: invalid memory address or nil pointer dereference\n[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x1495103]\n\ngoroutine 41 [running]:\nk8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc00050a380?})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:56 +0xcd\npanic({0x1636d20?, 0x2642050?})\n\t/usr/local/go/src/runtime/panic.go:785 +0x132\nk8s.io/client-go/tools/leaderelection.(*LeaderElector).Run(0x0, {0x1a88a68, 0xc0004fe2d0})\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/vendor/k8s.io/client-go/tools/leaderelection/leaderelection.go:206 +0x43\ngithub.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2.1()\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:496 +0x68\ncreated by github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2 in goroutine 40\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:494 +0x109\n"

This looks like a leader election issue at a glance, with a shorter version being:

Failed to create leader elector: nodeslicepools.whereabouts.cni.cncf.io \"donkeykongjr\" not found

And then panics with:

Observed a panic: \"invalid memory address or nil pointer dereference\"

Which seems to be associated with:

github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes.IPManagement.func2.1()\n\t/go/src/github.com/k8snetworkplumbingwg/whereabouts/pkg/storage/kubernetes/ipam.go:496

which is found here @ https://github.com/k8snetworkplumbingwg/whereabouts/blob/master/pkg/storage/kubernetes/ipam.go#L496

To Reproduce

Install whereabouts with:

kubectl apply     -f doc/crds/daemonset-install.yaml     -f doc/crds/whereabouts.cni.cncf.io_ippools.yaml     -f doc/crds/whereabouts.cni.cncf.io_overlappingrangeipreservations.yaml     -f doc/crds/whereabouts.cni.cncf.io_nodeslicepools.yaml     -f doc/crds/node-slice-controller.yaml

Apply this net-attach-def

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: whereabouts-fast-ipam
spec:
  config: '{
    "cniVersion": "0.3.0",
    "name": "whereaboutsexample",
    "type": "macvlan",
    "master": "eth0",
    "mode": "bridge",
    "ipam": {
      "type": "whereabouts",
      "range": "192.168.2.0/24",
      "node_slice_size": "/22",
      "network_name": "donkeykongjr",
      "leader_lease_duration": 1500,
      "leader_renew_deadline": 1000,
      "leader_retry_period": 500,
      "log_file": "/var/log/whereabouts.log",
      "log_level": "debug"
    }
  }'

Create this pod:

apiVersion: v1
kind: Pod
metadata:
  name: samplepod
  annotations:
    k8s.v1.cni.cncf.io/networks: whereabouts-fast-ipam
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine

Environment:

• Whereabouts version : Latest (I also tried one from october 17th which also exhibited the same issue)
• Kubernetes version (use kubectl version): v1.30.7
• Network-attachment-definition: (above)
• Whereabouts configuration (on the host): default
• OS (e.g. from /etc/os-release): fedora
• Kernel (e.g. uname -a): 6.8.5-301.fc40.x86_64

[lubronzhan]

Run into the same issue. Do I need to create a nodeslicepools object?

[lubronzhan]

E0516 18:29:02.199379       1 controller.go:290] "Unhandled Error" err="error syncing 'default/macvlan-config-with-fast-ipam': found IPAM conf mismatch for network-attachment-definitions with same network name, requeuing" logger="UnhandledError"

Looks like error inside the controller. I have two NAD, after I removed one of them it's ok now

[lubronzhan]

E0516 20:23:11.760723       1 controller.go:409] "failed to create nodeslicepool" err="NodeSlicePool.whereabouts.cni.cncf.io \"\" is invalid: metadata.name: Required value: name or generateName is required" resourceName="default/macvlan-config-with-fast-ipam"

Ok looks like network_name is required for fast node ipam

[lubronzhan]

The log shows the nodeslicepool was created

I0516 20:32:07.115693       1 controller.go:358] "node slice pool does not exist, creating" resourceName="default/macvlan-config-with-fast-ipam"
I0516 20:32:07.115847       1 controller.go:382] "node slice: &{{NodeSlicePool whereabouts.cni.cncf.io/v1alpha1} {macvlan-config-with-fast-ipam  kube-system    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [{k8s.cni.cncf.io/v1 NetworkAttachmentDefinition macvlan-config-with-fast-ipam 41251b97-7c64-4bdd-9d36-a41f43111fd7 0xc000119c69 0xc000119c68}] [] []} {192.168.30.0/24 /31} {[]}}" resourceName="default/macvlan-config-with-fast-ipam"

...


I0516 20:33:46.698100       1 controller.go:400] "assigning node to slice: wl-calico-controlplane-zz2r4-h5pc4" resourceName="default/macvlan-config-with-fast-ipam"
I0516 20:33:46.698166       1 controller.go:400] "assigning node to slice: wl-calico-md-0-bqqxv-lwqvw-vq9t8" resourceName="default/macvlan-config-with-fast-ipam"
I0516 20:33:46.698512       1 controller.go:406] "final allocations: [{wl-calico-controlplane-zz2r4-h5pc4 192.168.30.0/31} {wl-calico-md-0-bqqxv-lwqvw-vq9t8 192.168.30.2/31} { 192.168.30.4/31} { 192.168.30.6/31} { 192.168.30.8/31} { 192.168.30.10/31} { 192.168.30.12/31} { 192.168.30.14/31} { 192.168.30.16/31} { 192.168.30.18/31} { 192.168.30.20/31} { 192.168.30.22/31} { 192.168.30.24/31} { 192.168.30.26/31} { 192.168.30.28/31} { 192.168.30.30/31} { 192.168.30.32/31} { 192.168.30.34/31} { 192.168.30.36/31} { 192.168.30.38/31} { 192.168.30.40/31} { 192.168.30.42/31} { 192.168.30.44/31} { 192.168.30.46/31} { 192.168.30.48/31} { 192.168.30.50/31} { 192.168.30.52/31} { 192.168.30.54/31} { 192.168.30.56/31} { 192.168.30.58/31} { 192.168.30.60/31} { 192.168.30.62/31} { 192.168.30.64/31} { 192.168.30.66/31} { 192.168.30.68/31} { 192.168.30.70/31} { 192.168.30.72/31} { 192.168.30.74/31} { 192.168.30.76/31} { 192.168.30.78/31} { 192.168.30.80/31} { 192.168.30.82/31} { 192.168.30.84/31} { 192.168.30.86/31} { 192.168.30.88/31} { 192.168.30.90/31} { 192.168.30.92/31} { 192.168.30.94/31} { 192.168.30.96/31} { 192.168.30.98/31} { 192.168.30.100/31} { 192.168.30.102/31} { 192.168.30.104/31} { 192.168.30.106/31} { 192.168.30.108/31} { 192.168.30.110/31} { 192.168.30.112/31} { 192.168.30.114/31} { 192.168.30.116/31} { 192.168.30.118/31} { 192.168.30.120/31} { 192.168.30.122/31} { 192.168.30.124/31} { 192.168.30.126/31} { 192.168.30.128/31} { 192.168.30.130/31} { 192.168.30.132/31} { 192.168.30.134/31} { 192.168.30.136/31} { 192.168.30.138/31} { 192.168.30.140/31} { 192.168.30.142/31} { 192.168.30.144/31} { 192.168.30.146/31} { 192.168.30.148/31} { 192.168.30.150/31} { 192.168.30.152/31} { 192.168.30.154/31} { 192.168.30.156/31} { 192.168.30.158/31} { 192.168.30.160/31} { 192.168.30.162/31} { 192.168.30.164/31} { 192.168.30.166/31} { 192.168.30.168/31} { 192.168.30.170/31} { 192.168.30.172/31} { 192.168.30.174/31} { 192.168.30.176/31} { 192.168.30.178/31} { 192.168.30.180/31} { 192.168.30.182/31} { 192.168.30.184/31} { 192.168.30.186/31} { 192.168.30.188/31} { 192.168.30.190/31} { 192.168.30.192/31} { 192.168.30.194/31} { 192.168.30.196/31} { 192.168.30.198/31} { 192.168.30.200/31} { 192.168.30.202/31} { 192.168.30.204/31} { 192.168.30.206/31} { 192.168.30.208/31} { 192.168.30.210/31} { 192.168.30.212/31} { 192.168.30.214/31} { 192.168.30.216/31} { 192.168.30.218/31} { 192.168.30.220/31} { 192.168.30.222/31} { 192.168.30.224/31} { 192.168.30.226/31} { 192.168.30.228/31} { 192.168.30.230/31} { 192.168.30.232/31} { 192.168.30.234/31} { 192.168.30.236/31} { 192.168.30.238/31} { 192.168.30.240/31} { 192.168.30.242/31} { 192.168.30.244/31} { 192.168.30.246/31} { 192.168.30.248/31} { 192.168.30.250/31} { 192.168.30.252/31} { 192.168.30.254/31}]" resourceName="default/macvlan-config-with-fast-ipam"
I0516 20:33:46.719738       1 controller.go:285] "Successfully synced" resourceName="default/macvlan-config-with-fast-ipam"

But I couldn't find it

 kubectl get nodeslicepools -A
No resources found

 [error] error getting node slice kube-system/macvlan-config-with-fast-ipam nodeslicepools.whereabouts.cni.cncf.io \"macvlan-config-with-fast-ipam\" not found\n2025-05-16T20:32:37Z [error] Failed to create leader elector: nodeslicepools.whereabouts.cni.cncf.io \"macvlan-config-with-fast-ipam\" not found

[lubronzhan]

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"12dd2428-5632-4115-940f-f72797836a74","stage":"ResponseComplete","requestURI":"/apis/whereabouts.cni.cncf.io/v1alpha1/namespaces/kube-system/nodeslicepools","verb":"create","user":{"username":"system:serviceaccount:kube-system:whereabouts","uid":"2c1cf829-758e-4dc6-a0da-8153edc6b2a5","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["whereabouts-controller-79c45c85b6-qf8xg"],"authentication.kubernetes.io/pod-uid":["e81dd709-9b89-4d2d-8bc9-271375d45e7a"]}},"sourceIPs":["10.161.91.68"],"userAgent":"node-slice-controller/v0.0.0 (linux/amd64) kubernetes/$Format","objectRef":{"resource":"nodeslicepools","namespace":"kube-system","name":"macvlan-config-with-fast-ipam","apiGroup":"whereabouts.cni.cncf.io","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"code":201},"requestReceivedTimestamp":"2025-05-16T22:17:04.294102Z","stageTimestamp":"2025-05-16T22:17:04.319298Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"whereabouts\" of ClusterRole \"whereabouts-cni\" to ServiceAccount \"whereabouts/kube-system\""}}

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"60961b37-227b-42c2-b0a4-54f204452efe","stage":"ResponseComplete","requestURI":"/apis/whereabouts.cni.cncf.io/v1alpha1/namespaces/kube-system/nodeslicepools/macvlan-config-with-fast-ipam","verb":"delete","user":{"username":"system:serviceaccount:kube-system:generic-garbage-collector","uid":"9986418b-b1a3-461b-b86e-08df354da978","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"sourceIPs":["10.161.89.174"],"userAgent":"kube-controller-manager/v1.29.13+vmware.3 (linux/amd64) kubernetes/9a58e93/system:serviceaccount:kube-system:generic-garbage-collector","objectRef":{"resource":"nodeslicepools","namespace":"kube-system","name":"macvlan-config-with-fast-ipam","apiGroup":"whereabouts.cni.cncf.io","apiVersion":"v1alpha1"},"responseStatus":{"metadata":{},"status":"Success","details":{"name":"macvlan-config-with-fast-ipam","group":"whereabouts.cni.cncf.io","kind":"nodeslicepools","uid":"ea5834c8-fa92-4103-b053-fb51252cced3"},"code":200},"requestReceivedTimestamp":"2025-05-16T22:17:04.326180Z","stageTimestamp":"2025-05-16T22:17:04.342004Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding \"system:controller:generic-garbage-collector\" of ClusterRole \"system:controller:generic-garbage-collector\" to ServiceAccount \"generic-garbage-collector/kube-system\""}}

I noticed in audit log, there is an immediate 'delete' event conducted by garbage collector system:controller:generic-garbage-collector

[lubronzhan]

Ok looking at the code

ObjectMeta: metav1.ObjectMeta{
				Name:      getSliceName(ipamConf),
				Namespace: c.whereaboutsNamespace,
				OwnerReferences: []metav1.OwnerReference{
					*metav1.NewControllerRef(nad, cncfV1.SchemeGroupVersion.WithKind("NetworkAttachmentDefinition")),
				},
			},

It's expecting the NAD in the namespace where the whereabouts pod is located. So the ownerRef points to an non existence NAD

I0516 22:17:04.157346       1 controller.go:353] "About to update node slices for network-attachment-definition" resourceName="default/macvlan-config-with-fast-ipam" network-attachment-definition="default/macvlan-config-with-fast-ipam"

But the log previously is happy about that NAD.
Do you think the namespace should be updated to the current one?