Failed to assign an IP address to container

[maxsxu]

What happened:

Pod event

"aws-cni" failed (add): add cmd: failed to assign an IP address to container

Attach logs

{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"rpc/rpc.pb.go:1105","msg":"AddNetworkRequest: K8S_POD_NAME:\"myapp-7dbfd86df4-gsnsn\" K8S_POD_NAMESPACE:\"streamnative\" K8S_POD_INFRA_CONTAINER_ID:\"23296d3b8d9673814fd2039c316826aa531272bccb6d04a4ef6d910e20640b23\" ContainerID:\"23296d3b8d9673814fd2039c316826aa531272bccb6d04a4ef6d910e20640b23\" IfName:\"eth0\" NetworkName:\"aws-cni\" Netns:\"/var/run/netns/cni-b7d93dd1-57f9-6c58-4f56-f70ae8b09678\""}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: IP address pool stats: total 64, assigned 64"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.50.62.224/fffffff0"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.50.62.224/fffffff0"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.50.59.144/fffffff0"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.50.59.144/fffffff0"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.50.60.96/fffffff0"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.50.60.96/fffffff0"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 10.50.63.96/fffffff0"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 10.50.63.96/fffffff0"}
{"level":"debug","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: ENI eni-08b920c5d24225fd1 does not have available addresses"}
{"level":"error","ts":"2025-07-15T09:32:42.904Z","caller":"datastore/data_store.go:607","msg":"DataStore has no available IP/Prefix addresses"}

What you expected to happen:

Pod running successfully.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

By running curl http://localhost:61679/v1/enis on the Node, found the assignedIPs is 64 but actually there is only 12 Pod on this Node:

ENI info

{
  "TotalIPs": 64,
  "AssignedIPs": 64,
  "ENIs": {
    "eni-08b920c5d24225fd1": {
      "ID": "eni-08b920c5d24225fd1",
      "IsPrimary": true,
      "IsTrunk": false,
      "IsEFA": false,
      "DeviceNumber": 0,
      "AvailableIPv4Cidrs": {
        "10.50.59.144/28": {
          "Cidr": {
            "IP": "10.50.59.144",
            "Mask": "////8A=="
          },
          "IPAddresses": {
            "10.50.59.144": {
              "Address": "10.50.59.144",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "d39767022c53680b37389b43dde41c00b4461335bcab1fa9c812ca4fb4fa72c7",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "kube-system",
                "k8sPodName": "ebs-csi-node-mg5th"
              },
              "AssignedTime": "2025-07-15T03:11:28.923685168Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.145": {
              "Address": "10.50.59.145",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "8c8b84feb892c74e006e8aa6d2e596f2dfe261570af4912df7395dd07ecbb690",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-vklvj",
                "k8sPodName": "safari-production-bk-4"
              },
              "AssignedTime": "2025-07-15T03:11:43.416119661Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.146": {
              "Address": "10.50.59.146",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "85b59d62d2b992f422db891463cf3b3a1c2fcd670efca86ed97ae27e621b875c",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-cni-node-9g7f6"
              },
              "AssignedTime": "2025-07-15T03:11:43.624747619Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.147": {
              "Address": "10.50.59.147",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "8180e326fbfaeb6ca53c976b736d96d54b62a920c0c0340645ef0f72dfdac5c7",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "pulsar-filebeat-f8b6v"
              },
              "AssignedTime": "2025-07-15T03:11:43.721102863Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.148": {
              "Address": "10.50.59.148",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "94211403b0fbb8eb78d1df9df544d9255161bcbbbb48d2115e8ab5cbb0a76b13",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-ingressgateway-597b8c6d7d-xf8jk"
              },
              "AssignedTime": "2025-07-15T03:11:56.570216525Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.149": {
              "Address": "10.50.59.149",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "6bba458d93472b00d26a8d0572e3acc5704eb171195ffe200347bb3684b666d9",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "kyverno",
                "k8sPodName": "kyverno-background-controller-67d5fbcdff-85l7f"
              },
              "AssignedTime": "2025-07-15T03:11:56.678996726Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.150": {
              "Address": "10.50.59.150",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "083beba412ac8fe930ab61131ac393a218d0ee6066ead49fff2fbc5b41d810fb",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-apikeys-9cc458d77-ct5ls"
              },
              "AssignedTime": "2025-07-15T03:11:56.830093156Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.151": {
              "Address": "10.50.59.151",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "96947e0e4d6c4d217783836b886623d9f0bd918c9f52ad116c8688fd30b091ec",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "kube-system",
                "k8sPodName": "prometheus-adapter-84fc765c7f-pdgct"
              },
              "AssignedTime": "2025-07-15T03:11:57.694807528Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.152": {
              "Address": "10.50.59.152",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "a5eeb835dd1b3ccc3e52bc9ec431f0bcf7fd41dc46fe55a43d5b24187d617151",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-proxy-0"
              },
              "AssignedTime": "2025-07-15T03:12:01.557550211Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.153": {
              "Address": "10.50.59.153",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "c5de224b3dfa656501fd360aa57de620ef3621cc78222a46253d1b055358c541",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "dev-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:12:22.646677877Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.154": {
              "Address": "10.50.59.154",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "2b5498cb77978298f158820ae371bd92b3dbe61055c7a7accf24954006306aee",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-47mk6",
                "k8sPodName": "n-amer-1-broker-1"
              },
              "AssignedTime": "2025-07-15T03:12:28.178479117Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.155": {
              "Address": "10.50.59.155",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "60a8ef5c2ecc9240ec570fcc87d675952a3fd815f56bc2a3b845c2689daf88fb",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-broker-1"
              },
              "AssignedTime": "2025-07-15T03:12:28.958480015Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.156": {
              "Address": "10.50.59.156",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "ec36f85f17f40395320031eca6df7d687264c2a9a13cbc4f677c40ccefd99781",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-px2ri",
                "k8sPodName": "sbg-prod-cluster-zk-0"
              },
              "AssignedTime": "2025-07-15T03:12:32.645569896Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.157": {
              "Address": "10.50.59.157",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "3b24dc6d38065f6412d778094277cddd3dc483c5847b3da3fe11669206d29a60",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-x1dfi",
                "k8sPodName": "contextual-qa-bk-1"
              },
              "AssignedTime": "2025-07-15T03:12:32.846614046Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.158": {
              "Address": "10.50.59.158",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "1b22a27b2b31cf6d85121f9c131c928f0ce7f4e0867fdfb7033a1f7360779d80",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-47mk6",
                "k8sPodName": "n-amer-1-zk-0"
              },
              "AssignedTime": "2025-07-15T03:12:42.726078847Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.159": {
              "Address": "10.50.59.159",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "156de4c2cf9b61263ff54cc420bd1168bf461cb8cfbeffa26949797074e7a02e",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:12:43.627264186Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            }
          },
          "IsPrefix": true,
          "AddressFamily": ""
        },
        "10.50.60.96/28": {
          "Cidr": {
            "IP": "10.50.60.96",
            "Mask": "////8A=="
          },
          "IPAddresses": {
            "10.50.60.100": {
              "Address": "10.50.60.100",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "f78b694c0daa61fd7567eade9cf96692000e95b1976c9721fc95e8632e7470c2",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-vklvj",
                "k8sPodName": "safari-production-bk-4"
              },
              "AssignedTime": "2025-07-15T03:12:44.229422635Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.101": {
              "Address": "10.50.60.101",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "815730493d0116d2c12412d5f2beaf672b55dfd1fed169ab36e4154216c06d33",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "alloy-9x6j8"
              },
              "AssignedTime": "2025-07-15T03:12:44.234200386Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.102": {
              "Address": "10.50.60.102",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "e69ff13dd101877d970ebde3f814d7a716e4cba684fbaa2af7baf5a53403d47c",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "aws-use2-production-snci-pool-kid-vmagent-1"
              },
              "AssignedTime": "2025-07-15T03:12:45.268303529Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.103": {
              "Address": "10.50.60.103",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "72f2e4d6a4ca91e33962d0cf34694b654eda3691f31ac3b91b5a5774b6bb0d15",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-ingressgateway-597b8c6d7d-xf8jk"
              },
              "AssignedTime": "2025-07-15T03:12:57.255051428Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.104": {
              "Address": "10.50.60.104",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "ffd4c79cc6cd5d1de9b2528b7a7044d9717c958bc329265eafe4ad21f9b0b780",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "kyverno",
                "k8sPodName": "kyverno-background-controller-67d5fbcdff-85l7f"
              },
              "AssignedTime": "2025-07-15T03:12:57.267629732Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.105": {
              "Address": "10.50.60.105",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "bb7bd859990fe4f688215b6472ff90a4018c688ab75b6a9a6674da488b9912a6",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "kube-system",
                "k8sPodName": "prometheus-adapter-84fc765c7f-pdgct"
              },
              "AssignedTime": "2025-07-15T03:12:58.241921699Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.106": {
              "Address": "10.50.60.106",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "f5d2b8fbf230d6c1f802cde55fab6d42b990a3fba06f31e9291589d905131561",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-proxy-0"
              },
              "AssignedTime": "2025-07-15T03:13:02.253782605Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.107": {
              "Address": "10.50.60.107",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "2df96c602843e3a29462639133b5bb27629815f0da2c8635552fbc78434e55db",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-vklvj",
                "k8sPodName": "udx-cluster-zk-0"
              },
              "AssignedTime": "2025-07-15T03:13:04.1246837Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.108": {
              "Address": "10.50.60.108",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "f099a8c65c93063a10a6e46d1afb8342b027bf3750702a045f787c6cdc0d5ba6",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-c5xl1",
                "k8sPodName": "data-ingress-bk-0"
              },
              "AssignedTime": "2025-07-15T03:13:23.055147005Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.109": {
              "Address": "10.50.60.109",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "b07c16a836cd63973d8bec83155b1997ecec95b42d3707e71a7e168c455411f3",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "dev-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:13:23.293634508Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.110": {
              "Address": "10.50.60.110",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "ef27cb9b91da9702a9e958f93850ecb033c26cc61998a386dc327cdf17022855",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-47mk6",
                "k8sPodName": "n-amer-1-bk-2"
              },
              "AssignedTime": "2025-07-15T03:13:24.092128466Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.111": {
              "Address": "10.50.60.111",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "861235afc8b50b544ba5b70218e7b7fcd4bc69910e7f65b942fc872eeff6266d",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-47mk6",
                "k8sPodName": "n-amer-1-broker-1"
              },
              "AssignedTime": "2025-07-15T03:13:29.327885046Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.96": {
              "Address": "10.50.60.96",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "5d613d083f291aa0f95828f3f8cd696cae466d7a87bfb0d057920db5d8685f80",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "alloy-9x6j8"
              },
              "AssignedTime": "2025-07-15T03:11:43.619440373Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.97": {
              "Address": "10.50.60.97",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "324ff24520695a9c1c7162ca2748068f529f461702e0070e8e16dc6a5db215ee",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-x1dfi",
                "k8sPodName": "contextual-qa-apikeys-67bdffcf56-7jm4h"
              },
              "AssignedTime": "2025-07-15T03:11:56.972206012Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.98": {
              "Address": "10.50.60.98",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "0e415af836c1ac078dc6f9758b0437cf44b254895af2cbaac62f5d465578fe7f",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "pulsar-filebeat-f8b6v"
              },
              "AssignedTime": "2025-07-15T03:12:44.223313978Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.99": {
              "Address": "10.50.60.99",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "c8a6c34b4242e13705f59569a0a92195ba4671d48c2e9688072091436ad5fde3",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-cni-node-9g7f6"
              },
              "AssignedTime": "2025-07-15T03:12:44.225888249Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            }
          },
          "IsPrefix": true,
          "AddressFamily": ""
        },
        "10.50.62.224/28": {
          "Cidr": {
            "IP": "10.50.62.224",
            "Mask": "////8A=="
          },
          "IPAddresses": {
            "10.50.62.224": {
              "Address": "10.50.62.224",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "3170ea134bd658ef39f8f4d813dea8479e2ff4fbb04b6ebe7b60555aed41227e",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-vklvj",
                "k8sPodName": "safari-production-bk-4"
              },
              "AssignedTime": "2025-07-15T03:13:45.396289721Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.225": {
              "Address": "10.50.62.225",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "56a4f2c1bc61a01ad0ba616899e4df4c48dc80c889aa7cd30c47b587b722cfdd",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "pulsar-filebeat-f8b6v"
              },
              "AssignedTime": "2025-07-15T03:13:45.402019143Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.226": {
              "Address": "10.50.62.226",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "217f4d219c722fc7a920ad44a3a35a32b90c92699c4b5ec412bed138f958f3c0",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "aws-use2-production-snci-pool-kid-vmagent-1"
              },
              "AssignedTime": "2025-07-15T03:13:46.363643467Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.227": {
              "Address": "10.50.62.227",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "829e619a5f72bf62e5aac838c8c070a6739e2e279efdfb760f237de1269687b1",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-proxy-0"
              },
              "AssignedTime": "2025-07-15T03:14:03.404799809Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.228": {
              "Address": "10.50.62.228",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "962333b60d5ebd9a658f80b6d85a181c7222caa122546a89dfd0b99a6a7c63ab",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-vklvj",
                "k8sPodName": "udx-cluster-zk-0"
              },
              "AssignedTime": "2025-07-15T03:14:04.414048973Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.229": {
              "Address": "10.50.62.229",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "6e401f26b54cdbb81381fdc44bef55ba49083e3fc5bc7cf165497251eafeee53",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "aws-use2-production-snci-pool-kid-vmagent-1"
              },
              "AssignedTime": "2025-07-15T03:15:52.272586147Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.230": {
              "Address": "10.50.62.230",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "9cb26a12257b4e7282ed1e38dc758ab0e861aaf12af70c0179291db279d41a8e",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-2s9gy",
                "k8sPodName": "lz-non-prod-broker-1"
              },
              "AssignedTime": "2025-07-15T06:52:49.914574859Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.231": {
              "Address": "10.50.62.231",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "69a8e5f51286b317b7f386b13cc7546c64b0390be150d98572b5db67cd980bc7",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-broker-1"
              },
              "AssignedTime": "2025-07-15T03:15:52.294132431Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.232": {
              "Address": "10.50.62.232",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "ac757f0e1f6ef3493a8a88c815965ff416b0f93f788622510dc391c025f553c4",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "streamnative",
                "k8sPodName": "myapp-7dbfd86df4-9hljr"
              },
              "AssignedTime": "2025-07-15T09:31:15.913827911Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.233": {
              "Address": "10.50.62.233",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "974295d8059822f73f41c0bd262cee1ccfbd80143043958aeafeb0748c63b000",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "dev-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:15:52.303508525Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.234": {
              "Address": "10.50.62.234",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "139b93ef2b2896d17c0720d8ad802b44820932216b4e0f5e8c75843cd85e03db",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "pulsar-filebeat-mw28n"
              },
              "AssignedTime": "2025-07-15T09:04:06.673831114Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.235": {
              "Address": "10.50.62.235",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "112de150cd3293c736a5c29e98e11f0ae3df78aad50ea27057215fd8aa13e8da",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-47mk6",
                "k8sPodName": "n-amer-1-bk-2"
              },
              "AssignedTime": "2025-07-15T03:15:52.310143254Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.236": {
              "Address": "10.50.62.236",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "947ee8ae7280e76ae2b0645cce6d0cb09fc003881b9c9b251d0687310137625c",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-px2ri",
                "k8sPodName": "sbg-prod-cluster-zk-0"
              },
              "AssignedTime": "2025-07-15T03:16:30.245040527Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.237": {
              "Address": "10.50.62.237",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "c18117f85e84636d172aad7f8f73f564704c8cc9a557ad95c44a93eddd30a51d",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:14:45.515467295Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.238": {
              "Address": "10.50.62.238",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "d29a99afe6ce9a65dbe39add82bd65383e08325d4f23770eaa036ba113e1bcb9",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "alloy-9x6j8"
              },
              "AssignedTime": "2025-07-15T03:14:46.551619101Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.239": {
              "Address": "10.50.62.239",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "3d9c8716bf76088cf80935a71657aaaee1c69d447e6a92f93b78db9c49331f98",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-cni-node-9g7f6"
              },
              "AssignedTime": "2025-07-15T03:14:46.557082644Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            }
          },
          "IsPrefix": true,
          "AddressFamily": ""
        },
        "10.50.63.96/28": {
          "Cidr": {
            "IP": "10.50.63.96",
            "Mask": "////8A=="
          },
          "IPAddresses": {
            "10.50.63.100": {
              "Address": "10.50.63.100",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "3c74b3884f323172efdb6c922f4d301e3fa65987d6e1b98bc49052d2b3062a33",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-x1dfi",
                "k8sPodName": "contextual-qa-bk-1"
              },
              "AssignedTime": "2025-07-15T03:13:33.33804249Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.101": {
              "Address": "10.50.63.101",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "03942dfbe27fb3cdd5daf914d021ae680e706311c819441b14293fe4cafed611",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-px2ri",
                "k8sPodName": "sbg-prod-cluster-zk-0"
              },
              "AssignedTime": "2025-07-15T03:13:33.342242716Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.102": {
              "Address": "10.50.63.102",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "f8031a1e2d0f0d61893b608c5a0a99eb97466b46124264f6aaddb8b356786c21",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-47mk6",
                "k8sPodName": "n-amer-1-zk-0"
              },
              "AssignedTime": "2025-07-15T03:13:43.350257344Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.103": {
              "Address": "10.50.63.103",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "389ca75f19074cba018dc539619767c9f00077ded19fdad144fcb970471c1bf3",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:13:44.353822803Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.104": {
              "Address": "10.50.63.104",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "e6dce4ef9279d532a55a7b7a47051ba51b09dab64c1221e287a5eb658d8749fd",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-cni-node-9g7f6"
              },
              "AssignedTime": "2025-07-15T03:13:45.388768888Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.105": {
              "Address": "10.50.63.105",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "406245f8fa97a297524413aea2e81c279d4bb2276f2b173c21dd68846f609b2f",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "alloy-9x6j8"
              },
              "AssignedTime": "2025-07-15T03:13:45.402933178Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.106": {
              "Address": "10.50.63.106",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "cc6e8f4f4bec547058b215b2f103113cb0acf4523be6b9fc5a7fc05fe7889178",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-vklvj",
                "k8sPodName": "safari-production-zk-1"
              },
              "AssignedTime": "2025-07-15T03:13:54.380533479Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.107": {
              "Address": "10.50.63.107",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "4d3fc483553ab5e2fda16c614dab3630ef9d7ad30c52c278e8fd39969b5f7aa6",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "kyverno",
                "k8sPodName": "kyverno-background-controller-67d5fbcdff-85l7f"
              },
              "AssignedTime": "2025-07-15T03:13:58.435487603Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.108": {
              "Address": "10.50.63.108",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "c554f8031e54a086aa790fbd22bac1dddd17276e4e86ef9bd815da12a61aa413",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-apikeys-9cc458d77-ct5ls"
              },
              "AssignedTime": "2025-07-15T03:13:58.454671806Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.109": {
              "Address": "10.50.63.109",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "1ada51af6f2a4b751f6bc778c28c622eb49bcfda03bae57319840a0dcc5244db",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-ingressgateway-597b8c6d7d-xf8jk"
              },
              "AssignedTime": "2025-07-15T03:13:58.457391662Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.110": {
              "Address": "10.50.63.110",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "1ffd3803d5319505c2034448e70ce618a46cded954b22a70cdac27c5a90f85e5",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-x1dfi",
                "k8sPodName": "contextual-qa-apikeys-67bdffcf56-7jm4h"
              },
              "AssignedTime": "2025-07-15T03:13:58.46095269Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.111": {
              "Address": "10.50.63.111",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "ece5fad595c254441aac6144fa2a0a508273ee7b7e589b4555d5c15e88dafe79",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "kube-system",
                "k8sPodName": "prometheus-adapter-84fc765c7f-pdgct"
              },
              "AssignedTime": "2025-07-15T03:13:59.398515995Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.96": {
              "Address": "10.50.63.96",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "620c28030103b853c9495b92168816cc69043a85481b331261e1da85c8a7096a",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-vklvj",
                "k8sPodName": "safari-production-zk-1"
              },
              "AssignedTime": "2025-07-15T03:12:53.938617143Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.97": {
              "Address": "10.50.63.97",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "f5d5cfa5d6313e0c27fd88acb8d2373c249452ca16d8f304132ab1c3a5441b8c",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-apikeys-9cc458d77-ct5ls"
              },
              "AssignedTime": "2025-07-15T03:12:57.268762613Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.98": {
              "Address": "10.50.63.98",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "a9a6fe6169ba86f7952a44e321f04f63040b29b1282bbf41dc018608c31faae2",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-x1dfi",
                "k8sPodName": "contextual-qa-apikeys-67bdffcf56-7jm4h"
              },
              "AssignedTime": "2025-07-15T03:12:57.277650376Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.99": {
              "Address": "10.50.63.99",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "30ff9558c1e85c6fe18751cdf886d69810fca49fef83861450e15edf9e7c0eff",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-broker-1"
              },
              "AssignedTime": "2025-07-15T03:13:29.330080297Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            }
          },
          "IsPrefix": true,
          "AddressFamily": ""
        }
      },
      "IPv6Cidrs": {}
    }
  }
}

After restart the aws-node Pod, the stuck Pods get running successfully, and checking the ENI info again, the AssignedIPs number decreased to 28, but still bigger than actual Pods number.

ENI info after aws-node restart

{
  "TotalIPs": 64,
  "AssignedIPs": 28,
  "ENIs": {
    "eni-08b920c5d24225fd1": {
      "ID": "eni-08b920c5d24225fd1",
      "IsPrimary": true,
      "IsTrunk": false,
      "IsEFA": false,
      "DeviceNumber": 0,
      "AvailableIPv4Cidrs": {
        "10.50.59.144/28": {
          "Cidr": {
            "IP": "10.50.59.144",
            "Mask": "////8A=="
          },
          "IPAddresses": {
            "10.50.59.144": {
              "Address": "10.50.59.144",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "d39767022c53680b37389b43dde41c00b4461335bcab1fa9c812ca4fb4fa72c7",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "kube-system",
                "k8sPodName": "ebs-csi-node-mg5th"
              },
              "AssignedTime": "2025-07-15T03:11:28.923685168Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.146": {
              "Address": "10.50.59.146",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "85b59d62d2b992f422db891463cf3b3a1c2fcd670efca86ed97ae27e621b875c",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-cni-node-9g7f6"
              },
              "AssignedTime": "2025-07-15T03:11:43.624747619Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.153": {
              "Address": "10.50.59.153",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "c5de224b3dfa656501fd360aa57de620ef3621cc78222a46253d1b055358c541",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "dev-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:12:22.646677877Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.155": {
              "Address": "10.50.59.155",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "60a8ef5c2ecc9240ec570fcc87d675952a3fd815f56bc2a3b845c2689daf88fb",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-broker-1"
              },
              "AssignedTime": "2025-07-15T03:12:28.958480015Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.156": {
              "Address": "10.50.59.156",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "ec36f85f17f40395320031eca6df7d687264c2a9a13cbc4f677c40ccefd99781",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-px2ri",
                "k8sPodName": "sbg-prod-cluster-zk-0"
              },
              "AssignedTime": "2025-07-15T03:12:32.645569896Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.59.159": {
              "Address": "10.50.59.159",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "156de4c2cf9b61263ff54cc420bd1168bf461cb8cfbeffa26949797074e7a02e",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:12:43.627264186Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            }
          },
          "IsPrefix": true,
          "AddressFamily": ""
        },
        "10.50.60.96/28": {
          "Cidr": {
            "IP": "10.50.60.96",
            "Mask": "////8A=="
          },
          "IPAddresses": {
            "10.50.60.101": {
              "Address": "10.50.60.101",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "815730493d0116d2c12412d5f2beaf672b55dfd1fed169ab36e4154216c06d33",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "alloy-9x6j8"
              },
              "AssignedTime": "2025-07-15T03:12:44.234200386Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.102": {
              "Address": "10.50.60.102",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "e69ff13dd101877d970ebde3f814d7a716e4cba684fbaa2af7baf5a53403d47c",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "aws-use2-production-snci-pool-kid-vmagent-1"
              },
              "AssignedTime": "2025-07-15T03:12:45.268303529Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.109": {
              "Address": "10.50.60.109",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "b07c16a836cd63973d8bec83155b1997ecec95b42d3707e71a7e168c455411f3",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "dev-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:13:23.293634508Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.110": {
              "Address": "10.50.60.110",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "ef27cb9b91da9702a9e958f93850ecb033c26cc61998a386dc327cdf17022855",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-47mk6",
                "k8sPodName": "n-amer-1-bk-2"
              },
              "AssignedTime": "2025-07-15T03:13:24.092128466Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.96": {
              "Address": "10.50.60.96",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "5d613d083f291aa0f95828f3f8cd696cae466d7a87bfb0d057920db5d8685f80",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "alloy-9x6j8"
              },
              "AssignedTime": "2025-07-15T03:11:43.619440373Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.60.97": {
              "Address": "10.50.60.97",
              "IPAMKey": {
                "networkName": "",
                "containerID": "",
                "ifName": ""
              },
              "IPAMMetadata": {},
              "AssignedTime": "2025-07-15T10:22:19.913975065Z",
              "UnassignedTime": "2025-07-15T10:27:55.735890827Z"
            },
            "10.50.60.99": {
              "Address": "10.50.60.99",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "c8a6c34b4242e13705f59569a0a92195ba4671d48c2e9688072091436ad5fde3",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-cni-node-9g7f6"
              },
              "AssignedTime": "2025-07-15T03:12:44.225888249Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            }
          },
          "IsPrefix": true,
          "AddressFamily": ""
        },
        "10.50.62.224/28": {
          "Cidr": {
            "IP": "10.50.62.224",
            "Mask": "////8A=="
          },
          "IPAddresses": {
            "10.50.62.224": {
              "Address": "10.50.62.224",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "12415314a39fc01add8d4f35731ef5ff04fc43d4f8cb83ade3f283b25265d462",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "pulsar-filebeat-wwlzm"
              },
              "AssignedTime": "2025-07-15T10:22:15.918029626Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.226": {
              "Address": "10.50.62.226",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "217f4d219c722fc7a920ad44a3a35a32b90c92699c4b5ec412bed138f958f3c0",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "aws-use2-production-snci-pool-kid-vmagent-1"
              },
              "AssignedTime": "2025-07-15T03:13:46.363643467Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.229": {
              "Address": "10.50.62.229",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "6e401f26b54cdbb81381fdc44bef55ba49083e3fc5bc7cf165497251eafeee53",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "aws-use2-production-snci-pool-kid-vmagent-1"
              },
              "AssignedTime": "2025-07-15T03:15:52.272586147Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.230": {
              "Address": "10.50.62.230",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "9cb26a12257b4e7282ed1e38dc758ab0e861aaf12af70c0179291db279d41a8e",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-2s9gy",
                "k8sPodName": "lz-non-prod-broker-1"
              },
              "AssignedTime": "2025-07-15T06:52:49.914574859Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.231": {
              "Address": "10.50.62.231",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "69a8e5f51286b317b7f386b13cc7546c64b0390be150d98572b5db67cd980bc7",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-broker-1"
              },
              "AssignedTime": "2025-07-15T03:15:52.294132431Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.232": {
              "Address": "10.50.62.232",
              "IPAMKey": {
                "networkName": "",
                "containerID": "",
                "ifName": ""
              },
              "IPAMMetadata": {},
              "AssignedTime": "2025-07-15T09:31:15.913827911Z",
              "UnassignedTime": "2025-07-15T10:27:55.875808217Z"
            },
            "10.50.62.233": {
              "Address": "10.50.62.233",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "974295d8059822f73f41c0bd262cee1ccfbd80143043958aeafeb0748c63b000",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "dev-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:15:52.303508525Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.234": {
              "Address": "10.50.62.234",
              "IPAMKey": {
                "networkName": "",
                "containerID": "",
                "ifName": ""
              },
              "IPAMMetadata": {},
              "AssignedTime": "2025-07-15T10:04:36.914284637Z",
              "UnassignedTime": "2025-07-15T10:27:55.883553057Z"
            },
            "10.50.62.235": {
              "Address": "10.50.62.235",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "112de150cd3293c736a5c29e98e11f0ae3df78aad50ea27057215fd8aa13e8da",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-47mk6",
                "k8sPodName": "n-amer-1-bk-2"
              },
              "AssignedTime": "2025-07-15T03:15:52.310143254Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.236": {
              "Address": "10.50.62.236",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "947ee8ae7280e76ae2b0645cce6d0cb09fc003881b9c9b251d0687310137625c",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-px2ri",
                "k8sPodName": "sbg-prod-cluster-zk-0"
              },
              "AssignedTime": "2025-07-15T03:16:30.245040527Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.237": {
              "Address": "10.50.62.237",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "c18117f85e84636d172aad7f8f73f564704c8cc9a557ad95c44a93eddd30a51d",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:14:45.515467295Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.238": {
              "Address": "10.50.62.238",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "d29a99afe6ce9a65dbe39add82bd65383e08325d4f23770eaa036ba113e1bcb9",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "alloy-9x6j8"
              },
              "AssignedTime": "2025-07-15T03:14:46.551619101Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.62.239": {
              "Address": "10.50.62.239",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "3d9c8716bf76088cf80935a71657aaaee1c69d447e6a92f93b78db9c49331f98",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-cni-node-9g7f6"
              },
              "AssignedTime": "2025-07-15T03:14:46.557082644Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            }
          },
          "IsPrefix": true,
          "AddressFamily": ""
        },
        "10.50.63.96/28": {
          "Cidr": {
            "IP": "10.50.63.96",
            "Mask": "////8A=="
          },
          "IPAddresses": {
            "10.50.63.101": {
              "Address": "10.50.63.101",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "03942dfbe27fb3cdd5daf914d021ae680e706311c819441b14293fe4cafed611",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-px2ri",
                "k8sPodName": "sbg-prod-cluster-zk-0"
              },
              "AssignedTime": "2025-07-15T03:13:33.342242716Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.103": {
              "Address": "10.50.63.103",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "389ca75f19074cba018dc539619767c9f00077ded19fdad144fcb970471c1bf3",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "ssi",
                "k8sPodName": "prod-us-storage-bk-2"
              },
              "AssignedTime": "2025-07-15T03:13:44.353822803Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.104": {
              "Address": "10.50.63.104",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "e6dce4ef9279d532a55a7b7a47051ba51b09dab64c1221e287a5eb658d8749fd",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "istio-system",
                "k8sPodName": "istio-cni-node-9g7f6"
              },
              "AssignedTime": "2025-07-15T03:13:45.388768888Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.105": {
              "Address": "10.50.63.105",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "406245f8fa97a297524413aea2e81c279d4bb2276f2b173c21dd68846f609b2f",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "sn-system",
                "k8sPodName": "alloy-9x6j8"
              },
              "AssignedTime": "2025-07-15T03:13:45.402933178Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            },
            "10.50.63.99": {
              "Address": "10.50.63.99",
              "IPAMKey": {
                "networkName": "aws-cni",
                "containerID": "30ff9558c1e85c6fe18751cdf886d69810fca49fef83861450e15edf9e7c0eff",
                "ifName": "eth0"
              },
              "IPAMMetadata": {
                "k8sPodNamespace": "o-xqpg6",
                "k8sPodName": "stage-broker-1"
              },
              "AssignedTime": "2025-07-15T03:13:29.330080297Z",
              "UnassignedTime": "0001-01-01T00:00:00Z"
            }
          },
          "IsPrefix": true,
          "AddressFamily": ""
        }
      },
      "IPv6Cidrs": {}
    }
  }
}

So I guess the ipamd didn't refresh the ENI info and calculate the wrong Pod and assigned IP info by some reason.

Environment:

• Kubernetes version (use kubectl version): v1.31.9-eks-5d4a308
• CNI Version: v1.19.4-eksbuild.1
• OS (e.g: cat /etc/os-release): Amazon Linux 2023.6.20250303"
• Kernel (e.g. uname -a): 6.1.129-138.220.amzn2023.x86_64

[haouc]

Hey would you be able to provide a node log if the node is still there? Also are you using prefix delegation?

[jaydeokar]

If you have node logs please email it to k8s-awscni-triage@amazon.com to investigate further into this. Thanks

[maxsxu]

Yes, I'm using the prefix delegation, and I've Email the logs bundle to k8s-awscni-triage@amazon.com at Jul 15, 2025.

[haouc]

Thanks for sharing the logs. I am seeing these logs from IPAMD. Looks like the subnet has ran out of prefixes. Do you have multiple subnets with tag kubernetes.io/role/cni=1 added (https://aws.amazon.com/about-aws/whats-new/2024/04/amazon-vpc-cni-automatic-subnet-discovery/)? Have you checked if subnets provided still have prefixes available? Please be aware that secondary IP availability is not same as prefix availability in a subnet since fragmentation can make prefix not available when you still have plenty of secondary IPs.

{"level":"debug","ts":"2025-07-15T09:37:02.914Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - xxx"}
{"level":"debug","ts":"2025-07-15T09:37:02.914Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - xxx"}
{"level":"debug","ts":"2025-07-15T09:37:02.914Z","caller":"datastore/data_store.go:607","msg":"AssignPodIPv4Address: ENI eni-xxx does not have available addresses"}
{"level":"error","ts":"2025-07-15T09:37:02.914Z","caller":"datastore/data_store.go:607","msg":"DataStore has no available IP/Prefix addresses"}

[jaydeokar]

Please let us know if this helps understand the behavior of CNI.