[BUG] Race when deleting a pod and subnet with multiple IPs the same subnet bound to the same pod

[zsxsoft]

Kube-OVN Version

v1.12.28

Kubernetes Version

/

Operation-system/Kernel Version

/

Description

Race A

When a pod is in the process of being created, but before its IP allocation is complete—if you delete both the pod and the subnet now, the following issues occur:

• Both the Pod and Subnet are deleted.
• The IP address remains and cannot be deleted using kubectl delete ip.
• The only way to delete the lingering IP is by running: kubectl patch ips --type=merge -p '{"metadata":{"finalizers":[]}}'

[root@vm-master-1 ~]# kubectl delete ip vm-1-0.vm
ip.kubeovn.io "vm-1-0.vm" deleted
^C[root@vm-master-1 ~]# kubectl get ip | grep vm-1 | head -n9
vm-1-0.vm                           10.XXXX           XXXX   node-XX       net-c13XXXX
vm-1-0.vm.nad-10.vm.ovn             10.XXXX           XXXX                 net-c13XXXX
vm-1-0.vm.nad-11.vm.ovn             10.XXXX           XXXX                 net-c13XXXX
vm-1-0.vm.nad-12.vm.ovn             10.XXXX           XXXX                 net-c13XXXX
vm-1-0.vm.nad-13.vm.ovn             10.XXXX           XXXX                 net-c13XXXX
vm-1-0.vm.nad-14.vm.ovn             10.XXXX           XXXX                 net-c13XXXX
vm-1-0.vm.nad-15.vm.ovn             10.XXXX           XXXX                 net-c13XXXX
vm-1-0.vm.nad-16.vm.ovn             10.XXXX           XXXX                 net-c13XXXX
vm-1-0.vm.nad-17.vm.ovn             10.XXXX           XXXX                 net-c13XXXX
[root@vm-master-1 ~]# 
[root@vm-master-1 ~]# kubectl delete ip vm-1-0.vm
ip.kubeovn.io "vm-1-0.vm" deleted
^C[root@vm-master-1 ~]# kubectl get ip | grep vm-1 | head -n1
vm-1-0.vm                           10.XXXX           XXXX   node-XX       net-c13XXXX
[root@vm-master-1 ~]# kubectl get pods -n vm | grep vm-1
[root@vm-master-1 ~]# kubectl get subnet | grep net-c13XXX

Race B

I'm unsure about the triggering conditions, but the final test results are:

• The Pod was deleted, but both the IP and Subnet remained undeleted.
• The IP can only be deleted by patching the finalizer; it cannot be deleted directly.
• Even after deleting all IPs (as shown below, with V4USED equal to 0), the Subnet cannot be deleted and can only be removed by patching the finalizer.

This is quite common in my production cluster, with about a 10% probability of recurrence, but I'm still unsure of the underlying cause.

[root@vm-master-1 ~]# kubectl get subnet net-net-edaXXXXX
NAME                                   PROVIDER   VPC           PROTOCOL   CIDR            PRIVATE   NAT    DEFAULT   GATEWAYTYPE   V4USED   V4AVAILABLE   V6USED   V6AVAILABLE   EXCLUDEIPS       U2OINTERCONNECTIONIP
net-net-edaXXXXX   ovn        ovn-cluster   IPv4       10.XX.XX.0/24   false     true   false     distributed   0        253           0        0             ["10.XX.XX.1"]   
[root@vm-master-1 ~]# kubectl delete subnet net-net-edaXXXXX
subnet.kubeovn.io "net-net-edaXXXXX" deleted
^C[root@vm-master-1 ~]# kubectl edit subnet net-net-edaXXXXX
subnet.kubeovn.io/net-net-edaXXXXX edited
[root@vm-master-1 ~]# kubectl get subnet net-net-edaXXXXX

Steps To Reproduce

Subnet YAML: #4822 (comment)
Pod YAML:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    k8s.v1.cni.cncf.io/networks: vm/nad-3,vm/nad-4,vm/nad-5,vm/nad-6,vm/nad-7,vm/nad-8,vm/nad-9,vm/nad-10,vm/nad-11,vm/nad-12,vm/nad-13,vm/nad-14,vm/nad-15
    nad-3.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-3.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-3.vm.ovn.kubernetes.io/ip_address: 10.123.123.194
    nad-3.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-3.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-3.vm.ovn.kubernetes.io/mac_address: e8:4f:e0:e3:4e:b3
    nad-3.vm.ovn.kubernetes.io/port_security: "true"
    nad-4.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-4.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-4.vm.ovn.kubernetes.io/ip_address: 10.123.123.3
    nad-4.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-4.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-4.vm.ovn.kubernetes.io/mac_address: 10:71:ec:53:e2:89
    nad-4.vm.ovn.kubernetes.io/port_security: "true"
    nad-5.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-5.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-5.vm.ovn.kubernetes.io/ip_address: 10.123.123.245
    nad-5.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-5.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-5.vm.ovn.kubernetes.io/mac_address: 24:c3:df:f5:2f:fe
    nad-5.vm.ovn.kubernetes.io/port_security: "true"
    nad-6.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-6.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-6.vm.ovn.kubernetes.io/ip_address: 10.123.123.152
    nad-6.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-6.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-6.vm.ovn.kubernetes.io/mac_address: 48:79:2b:7a:e0:91
    nad-6.vm.ovn.kubernetes.io/port_security: "true"
    nad-7.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-7.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-7.vm.ovn.kubernetes.io/ip_address: 10.123.123.21
    nad-7.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-7.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-7.vm.ovn.kubernetes.io/mac_address: 00:6f:72:df:67:7b
    nad-7.vm.ovn.kubernetes.io/port_security: "true"
    nad-8.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-8.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-8.vm.ovn.kubernetes.io/ip_address: 10.123.123.191
    nad-8.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-8.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-8.vm.ovn.kubernetes.io/mac_address: 00:13:bb:f1:04:6b
    nad-8.vm.ovn.kubernetes.io/port_security: "true"
    nad-9.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-9.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-9.vm.ovn.kubernetes.io/ip_address: 10.123.123.121
    nad-9.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-9.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-9.vm.ovn.kubernetes.io/mac_address: 14:ac:d3:09:6b:10
    nad-9.vm.ovn.kubernetes.io/port_security: "true"
    nad-10.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-10.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-10.vm.ovn.kubernetes.io/ip_address: 10.123.123.25
    nad-10.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-10.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-10.vm.ovn.kubernetes.io/mac_address: 40:ec:54:03:f1:27
    nad-10.vm.ovn.kubernetes.io/port_security: "true"
    nad-11.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-11.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-11.vm.ovn.kubernetes.io/ip_address: 10.123.123.9
    nad-11.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-11.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-11.vm.ovn.kubernetes.io/mac_address: 64:c2:cd:a3:b2:93
    nad-11.vm.ovn.kubernetes.io/port_security: "true"
    nad-12.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-12.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-12.vm.ovn.kubernetes.io/ip_address: 10.123.123.160
    nad-12.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-12.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-12.vm.ovn.kubernetes.io/mac_address: 78:4f:a3:a7:e1:69
    nad-12.vm.ovn.kubernetes.io/port_security: "true"
    nad-13.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-13.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-13.vm.ovn.kubernetes.io/ip_address: 10.123.123.197
    nad-13.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-13.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-13.vm.ovn.kubernetes.io/mac_address: 40:ce:03:28:e7:08
    nad-13.vm.ovn.kubernetes.io/port_security: "true"
    nad-14.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-14.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-14.vm.ovn.kubernetes.io/ip_address: 10.123.123.222
    nad-14.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-14.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-14.vm.ovn.kubernetes.io/mac_address: 10:3d:bf:ef:2c:85
    nad-14.vm.ovn.kubernetes.io/port_security: "true"
    nad-15.vm.ovn.kubernetes.io/cidr: 10.123.123.0/24
    nad-15.vm.ovn.kubernetes.io/gateway: 10.123.123.1
    nad-15.vm.ovn.kubernetes.io/ip_address: 10.123.123.157
    nad-15.vm.ovn.kubernetes.io/logical_router: ovn-cluster
    nad-15.vm.ovn.kubernetes.io/logical_switch: net-netbbb
    nad-15.vm.ovn.kubernetes.io/mac_address: 28:89:fd:85:ca:17
    nad-15.vm.ovn.kubernetes.io/port_security: "true"
    ovn.kubernetes.io/cidr: 10.10.10.0/24
    ovn.kubernetes.io/gateway: 10.10.10.1
    ovn.kubernetes.io/ip_address: 10.10.10.52
    ovn.kubernetes.io/logical_router: ovn-cluster
    ovn.kubernetes.io/logical_switch: net-netaaa
    ovn.kubernetes.io/port_security: "true"
    prometheus.io/path: /metrics
    prometheus.io/scrape: "true"
  name: vm-aaa-0
  namespace: vm
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx
  dnsPolicy: ClusterFirst

Current Behavior

/

Expected Behavior

/

[dosubot]

The issues you're encountering with race conditions during the deletion of pods and subnets in Kube-OVN are related to the management of finalizers. Finalizers are used to ensure that certain cleanup operations are completed before a resource is fully deleted. In your case, the lingering IPs and subnets that cannot be deleted without manually patching the finalizers indicate that the cleanup process is not being completed as expected.

For Race A, the problem arises when a pod is being created, but its IP allocation isn't complete. If you delete both the pod and the subnet at this stage, the IP address remains because the finalizer prevents its deletion until the IP allocation process is fully resolved.

For Race B, the issue is that even after the pod is deleted, the IP and subnet remain due to the finalizer not being removed. This can happen if the system doesn't recognize that all IPs are no longer in use, preventing the subnet from being deleted.

To address these issues, you might consider the following:

1. Ensure Proper Finalizer Management: Make sure that finalizers are correctly added and removed during the lifecycle of IP and subnet resources. This involves ensuring that the cleanup operations are completed before the finalizer is removed.

2. Check for IP Usage: Before attempting to delete a subnet, verify that all associated IPs are no longer in use. This is crucial for allowing the finalizer to be removed and the subnet to be deleted.

3. Review Cleanup Logic: Investigate the logic that handles the deletion of IPs and subnets to ensure that it correctly identifies when resources are no longer in use and can be safely deleted.

4. Patch Finalizers if Necessary: As a temporary workaround, you can manually patch the finalizers to remove them and allow the deletion to proceed. However, this should be done with caution and ideally automated within the system to prevent manual intervention.

These steps should help mitigate the race conditions you're experiencing and ensure that resources are properly cleaned up when a pod or subnet is deleted.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[zbb88888]

could you please attach the log kubectl logs -n kube-system kube-ovn-controllerxxx | grep vm-aaa-0?

I want to see if the pod deletion triggers IP deletion, and, if IP deletion trigger subnet update.

[oilbeater]

For IP CR bind to VM, the lifecycle is bind to the VM CR not the Pod. Does the VM CR still exists?

[zsxsoft]

@oilbeater I'm not using KubeVirt, all VMs are StatefulSets. I confirmed pods and sfs are gone.

[zsxsoft]

@bobz965
For case A:

I0107 12:30:07.889537       7 subnet.go:350] format subnet net-aaa, changed true
I0107 12:30:07.901360       7 security_group.go:247] handle add/update security group sg-net-aaa
W0107 12:30:07.904692       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:07.908350       7 security_group.go:306] ingress need update, sg:sg-net-aaa
I0107 12:30:07.908451       7 security_group.go:311] egress need update, sg:sg-net-aaa
I0107 12:30:07.908465       7 security_group.go:317] both ingress && egress need update, sg:sg-net-aaa
I0107 12:30:08.163319       7 security_group.go:433] sync lsp for security group sg-net-aaa
I0107 12:30:08.190277       7 security_group.go:247] handle add/update security group kubeovn_deny_all
I0107 12:30:08.906900       7 event.go:377] Event(v1.ObjectReference{Kind:"Subnet", Namespace:"", Name:"net-aaa", UID:"9c7461ee-587a-4210-a077-add4c28de6a8", APIVersion:"kubeovn.io/v1", ResourceVersion:"13009863", FieldPath:""}): type: 'Normal' reason: 'ValidateLogicalSwitchSuccess' 
I0107 12:30:08.915207       7 ipam.go:311] adding new subnet net-aaa
I0107 12:30:09.032756       7 subnet.go:2378] add common policy route for router: ovn-cluster, match ip4.dst == 10.144.39.0/24, action allow, externalID map[subnet:net-aaa vendor:kube-ovn]
I0107 12:30:09.140560       7 subnet.go:2524] add policy route for router: ovn-cluster, match ip4.src == $net.aaa.node.XXX_ip4, action reroute, extrenalID map[node:node-72 subnet:net-aaa vendor:kube-ovn]
I0107 12:30:19.800032       7 event.go:377] Event(v1.ObjectReference{Kind:"Subnet", Namespace:"", Name:"net-aaa", UID:"9c7461ee-587a-4210-a077-add4c28de6a8", APIVersion:"kubeovn.io/v1", ResourceVersion:"13009863", FieldPath:""}): type: 'Normal' reason: 'ResetLogicalSwitchAclSuccess' 
I0107 12:30:19.850460       7 subnet.go:350] format subnet net-aaa, changed false
I0107 12:30:19.850574       7 vpc.go:147] handle status update for vpc ovn-cluster
I0107 12:30:19.850742       7 event.go:377] Event(v1.ObjectReference{Kind:"Subnet", Namespace:"", Name:"net-aaa", UID:"9c7461ee-587a-4210-a077-add4c28de6a8", APIVersion:"kubeovn.io/v1", ResourceVersion:"13009905", FieldPath:""}): type: 'Normal' reason: 'ValidateLogicalSwitchSuccess' 
I0107 12:30:19.933985       7 subnet.go:2378] add common policy route for router: ovn-cluster, match ip4.dst == 10.XXX/24, action allow, externalID map[subnet:net-aaa vendor:kube-ovn]
I0107 12:30:23.328905       7 pod.go:224] enqueue add pod vm/vm-aaa-0
I0107 12:30:23.328968       7 pod.go:519] handle add/update pod vm/vm-aaa-0
I0107 12:30:23.744171       7 pod.go:570] sync pod vm/vm-aaa-0 allocated
I0107 12:30:23.765953       7 pod.go:347] enqueue update pod vm/vm-aaa-0
I0107 12:30:23.787226       7 ipam.go:72] allocating static ip 10.144.39.213 from subnet net-aaa
I0107 12:30:23.787291       7 ipam.go:102] allocate v4 10.144.39.213, mac 70:dd:80:f9:0a:0f for vm/vm-aaa-0 from subnet net-aaa
I0107 12:30:23.795357       7 security_group.go:433] sync lsp for security group sg-net-aaa
W0107 12:30:23.795395       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:23.806065       7 ipam.go:72] allocating static ip 10.144.39.233 from subnet net-aaa
I0107 12:30:23.806141       7 ipam.go:102] allocate v4 10.144.39.233, mac d0:04:72:cb:3c:02 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:23.828094       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:23.841775       7 ipam.go:72] allocating static ip 10.144.39.158 from subnet net-aaa
I0107 12:30:23.841837       7 ipam.go:102] allocate v4 10.144.39.158, mac d4:17:cb:05:92:27 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:23.849843       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:23.850787       7 security_group.go:433] sync lsp for security group allow-internet-and-some-others
I0107 12:30:23.893909       7 ipam.go:72] allocating static ip 10.144.39.117 from subnet net-aaa
I0107 12:30:23.893975       7 ipam.go:102] allocate v4 10.144.39.117, mac a8:ec:30:ca:67:e4 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:23.902060       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:23.929348       7 ipam.go:72] allocating static ip 10.144.39.55 from subnet net-aaa
I0107 12:30:23.929408       7 ipam.go:102] allocate v4 10.144.39.55, mac 24:fb:ea:2e:e7:21 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:23.935702       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:23.944139       7 ipam.go:72] allocating static ip 10.144.39.247 from subnet net-aaa
I0107 12:30:23.944199       7 ipam.go:102] allocate v4 10.144.39.247, mac c4:66:57:81:39:32 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:23.952357       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:23.995488       7 ipam.go:72] allocating static ip 10.144.39.25 from subnet net-aaa
I0107 12:30:23.995552       7 ipam.go:102] allocate v4 10.144.39.25, mac b8:57:2e:26:0c:0a for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.001812       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.038526       7 ipam.go:72] allocating static ip 10.144.39.121 from subnet net-aaa
I0107 12:30:24.039191       7 ipam.go:102] allocate v4 10.144.39.121, mac 84:ea:35:29:dc:3c for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.047859       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.086262       7 ipam.go:72] allocating static ip 10.144.39.10 from subnet net-aaa
I0107 12:30:24.086365       7 ipam.go:102] allocate v4 10.144.39.10, mac 30:5b:cb:44:a3:2b for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.094244       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.134230       7 ipam.go:72] allocating static ip 10.144.39.50 from subnet net-aaa
I0107 12:30:24.134316       7 ipam.go:102] allocate v4 10.144.39.50, mac 70:96:bc:bc:7e:1e for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.142800       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
W0107 12:30:24.229347       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.229640       7 ipam.go:72] allocating static ip 10.144.39.249 from subnet net-aaa
I0107 12:30:24.229715       7 ipam.go:102] allocate v4 10.144.39.249, mac 50:c1:15:6f:49:cc for vm/vm-aaa-0 from subnet net-aaa
I0107 12:30:24.258147       7 pod.go:347] enqueue update pod vm/vm-aaa-0
I0107 12:30:24.274990       7 ipam.go:72] allocating static ip 10.144.39.186 from subnet net-aaa
I0107 12:30:24.275062       7 ipam.go:102] allocate v4 10.144.39.186, mac 90:3a:c0:45:9e:77 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.287119       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.301674       7 ipam.go:72] allocating static ip 10.144.39.35 from subnet net-aaa
I0107 12:30:24.301747       7 ipam.go:102] allocate v4 10.144.39.35, mac 34:52:7a:76:d3:bf for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.311303       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.328804       7 ipam.go:72] allocating static ip 10.144.39.114 from subnet net-aaa
I0107 12:30:24.329061       7 ipam.go:102] allocate v4 10.144.39.114, mac 04:39:02:0c:c7:b9 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.336742       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.349698       7 ipam.go:72] allocating static ip 10.144.39.104 from subnet net-aaa
I0107 12:30:24.349769       7 ipam.go:102] allocate v4 10.144.39.104, mac b0:7a:67:68:02:b5 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.356774       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.363634       7 ipam.go:72] allocating static ip 10.144.39.47 from subnet net-aaa
I0107 12:30:24.364226       7 ipam.go:102] allocate v4 10.144.39.47, mac 34:9a:57:c1:68:9a for vm/vm-aaa-0 from subnet net-aaa
I0107 12:30:24.387369       7 ipam.go:72] allocating static ip 10.144.39.115 from subnet net-aaa
W0107 12:30:24.387420       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.387819       7 ipam.go:102] allocate v4 10.144.39.115, mac 4c:b9:b0:d2:8c:23 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.396659       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.409034       7 ipam.go:72] allocating static ip 10.144.39.11 from subnet net-aaa
I0107 12:30:24.409548       7 ipam.go:102] allocate v4 10.144.39.11, mac 34:1e:db:c4:01:0c for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.427871       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.437253       7 ipam.go:72] allocating static ip 10.144.39.86 from subnet net-aaa
I0107 12:30:24.437332       7 ipam.go:102] allocate v4 10.144.39.86, mac 08:24:09:10:de:33 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.449491       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.454234       7 ipam.go:72] allocating static ip 10.144.39.88 from subnet net-aaa
I0107 12:30:24.454304       7 ipam.go:102] allocate v4 10.144.39.88, mac 58:22:60:e7:36:22 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.462328       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.465144       7 ipam.go:72] allocating static ip 10.144.39.139 from subnet net-aaa
I0107 12:30:24.465415       7 ipam.go:102] allocate v4 10.144.39.139, mac 90:2a:54:5f:79:b4 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.473868       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.498956       7 ipam.go:72] allocating static ip 10.144.39.180 from subnet net-aaa
I0107 12:30:24.499290       7 ipam.go:102] allocate v4 10.144.39.180, mac 18:ff:c9:58:8d:3a for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.506481       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.511153       7 ipam.go:72] allocating static ip 10.144.39.98 from subnet net-aaa
I0107 12:30:24.527799       7 ipam.go:102] allocate v4 10.144.39.98, mac 58:a3:78:00:83:52 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.528060       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.551394       7 ipam.go:72] allocating static ip 10.144.39.131 from subnet net-aaa
I0107 12:30:24.551497       7 ipam.go:102] allocate v4 10.144.39.131, mac 08:09:ce:32:53:99 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.551590       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.586339       7 ipam.go:72] allocating static ip 10.144.39.191 from subnet net-aaa
I0107 12:30:24.586654       7 ipam.go:102] allocate v4 10.144.39.191, mac 2c:56:81:78:a3:bb for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.591792       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.601343       7 ipam.go:72] allocating static ip 10.144.39.127 from subnet net-aaa
I0107 12:30:24.601684       7 ipam.go:102] allocate v4 10.144.39.127, mac a0:e6:f6:23:51:be for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.608319       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.628312       7 ipam.go:72] allocating static ip 10.144.39.166 from subnet net-aaa
W0107 12:30:24.628678       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.629128       7 ipam.go:102] allocate v4 10.144.39.166, mac a4:f6:ad:28:0e:ab for vm/vm-aaa-0 from subnet net-aaa
I0107 12:30:24.651022       7 ipam.go:72] allocating static ip 10.144.39.72 from subnet net-aaa
I0107 12:30:24.652340       7 ipam.go:102] allocate v4 10.144.39.72, mac 5c:70:6a:a9:27:fd for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.658741       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.686271       7 ipam.go:72] allocating static ip 10.144.39.200 from subnet net-aaa
I0107 12:30:24.692507       7 ipam.go:102] allocate v4 10.144.39.200, mac 88:8f:e8:dd:fa:85 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.693387       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.709131       7 ipam.go:72] allocating static ip 10.144.39.189 from subnet net-aaa
I0107 12:30:24.709459       7 ipam.go:102] allocate v4 10.144.39.189, mac b0:dc:b3:e6:e5:7e for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.713392       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.728349       7 ipam.go:72] allocating static ip 10.144.39.199 from subnet net-aaa
I0107 12:30:24.728776       7 ipam.go:102] allocate v4 10.144.39.199, mac 08:16:61:d8:1f:61 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.733699       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.745718       7 ipam.go:72] allocating static ip 10.144.39.13 from subnet net-aaa
I0107 12:30:24.745814       7 ipam.go:102] allocate v4 10.144.39.13, mac f4:4a:b5:15:b3:01 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.753211       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.765847       7 ipam.go:72] allocating static ip 10.144.39.95 from subnet net-aaa
I0107 12:30:24.765935       7 ipam.go:102] allocate v4 10.144.39.95, mac 30:2b:95:6c:af:f4 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.793448       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.793648       7 ipam.go:72] allocating static ip 10.144.39.167 from subnet net-aaa
I0107 12:30:24.793731       7 ipam.go:102] allocate v4 10.144.39.167, mac 3c:ed:0b:af:a9:37 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.799320       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.813980       7 ipam.go:72] allocating static ip 10.144.39.92 from subnet net-aaa
I0107 12:30:24.814067       7 ipam.go:102] allocate v4 10.144.39.92, mac 5c:13:66:87:bb:d0 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.828001       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.839574       7 ipam.go:72] allocating static ip 10.144.39.96 from subnet net-aaa
I0107 12:30:24.839982       7 ipam.go:102] allocate v4 10.144.39.96, mac 04:39:c2:fb:bb:01 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:24.843827       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:24.940581       7 ipam.go:72] allocating static ip 10.144.39.168 from subnet net-aaa
I0107 12:30:24.940685       7 ipam.go:102] allocate v4 10.144.39.168, mac 44:dc:62:29:86:6c for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.028894       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.044534       7 ipam.go:72] allocating static ip 10.144.39.23 from subnet net-aaa
I0107 12:30:25.044821       7 ipam.go:102] allocate v4 10.144.39.23, mac 58:cd:cb:f9:67:15 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.085751       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.145399       7 ipam.go:72] allocating static ip 10.144.39.212 from subnet net-aaa
I0107 12:30:25.145525       7 ipam.go:102] allocate v4 10.144.39.212, mac 28:6f:b5:bc:d7:29 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.150042       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.167219       7 ipam.go:72] allocating static ip 10.144.39.223 from subnet net-aaa
I0107 12:30:25.167624       7 ipam.go:102] allocate v4 10.144.39.223, mac 14:1a:c0:f2:3b:d1 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.185975       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.195657       7 ipam.go:72] allocating static ip 10.144.39.60 from subnet net-aaa
I0107 12:30:25.195752       7 ipam.go:102] allocate v4 10.144.39.60, mac 08:75:61:ce:b8:e6 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.203014       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.208608       7 ipam.go:72] allocating static ip 10.144.39.135 from subnet net-aaa
I0107 12:30:25.208696       7 ipam.go:102] allocate v4 10.144.39.135, mac 34:96:e8:ad:94:60 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.228568       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.234620       7 ipam.go:72] allocating static ip 10.144.39.116 from subnet net-aaa
I0107 12:30:25.234728       7 ipam.go:102] allocate v4 10.144.39.116, mac dc:6e:ba:6c:62:7c for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.239623       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.247531       7 ipam.go:72] allocating static ip 10.144.39.238 from subnet net-aaa
I0107 12:30:25.247694       7 ipam.go:102] allocate v4 10.144.39.238, mac 50:56:d9:8a:11:c0 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.256179       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.264657       7 ipam.go:72] allocating static ip 10.144.39.156 from subnet net-aaa
I0107 12:30:25.264760       7 ipam.go:102] allocate v4 10.144.39.156, mac 9c:f1:23:25:f6:a7 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.269496       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.308753       7 ipam.go:72] allocating static ip 10.144.39.83 from subnet net-aaa
I0107 12:30:25.308858       7 ipam.go:102] allocate v4 10.144.39.83, mac b8:30:e2:b1:9c:50 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.310682       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.337706       7 ipam.go:72] allocating static ip 10.144.39.24 from subnet net-aaa
I0107 12:30:25.337802       7 ipam.go:102] allocate v4 10.144.39.24, mac 38:ba:f6:09:1f:da for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.343161       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.357678       7 ipam.go:72] allocating static ip 10.144.39.87 from subnet net-aaa
I0107 12:30:25.358027       7 ipam.go:102] allocate v4 10.144.39.87, mac 60:05:42:6d:51:ee for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.365009       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.398592       7 ipam.go:72] allocating static ip 10.144.39.181 from subnet net-aaa
I0107 12:30:25.398700       7 ipam.go:102] allocate v4 10.144.39.181, mac e4:2e:14:73:b8:b4 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.403450       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.428778       7 ipam.go:72] allocating static ip 10.144.39.15 from subnet net-aaa
I0107 12:30:25.428876       7 ipam.go:102] allocate v4 10.144.39.15, mac f8:97:ec:1a:fb:11 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.435155       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.441567       7 ipam.go:72] allocating static ip 10.144.39.239 from subnet net-aaa
I0107 12:30:25.441679       7 ipam.go:102] allocate v4 10.144.39.239, mac 78:f6:89:b3:18:04 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.447852       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.454911       7 ipam.go:72] allocating static ip 10.144.39.183 from subnet net-aaa
I0107 12:30:25.455399       7 ipam.go:102] allocate v4 10.144.39.183, mac 4c:35:95:48:f0:32 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.463008       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.490797       7 ipam.go:72] allocating static ip 10.144.39.227 from subnet net-aaa
I0107 12:30:25.490916       7 ipam.go:102] allocate v4 10.144.39.227, mac f0:0d:87:ea:47:17 for vm/vm-aaa-0 from subnet net-aaa
W0107 12:30:25.497005       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
W0107 12:30:25.512734       7 warnings.go:70] metadata.finalizers: "kube-ovn-controller": prefer a domain-qualified finalizer name to avoid accidental conflicts with other finalizer writers
I0107 12:30:25.576269       7 pod.go:757] sync pod vm/vm-aaa-0 routed
I0107 12:30:25.893296       7 pod.go:347] enqueue update pod vm/vm-aaa-0
I0107 12:30:30.201306       7 security_group.go:433] sync lsp for security group sg-net-aaa
I0107 12:30:31.497847       7 pod.go:401] take 8168 ms to handle sync pod vm/vm-aaa-0
I0107 12:30:31.497905       7 pod.go:519] handle add/update pod vm/vm-aaa-0
I0107 12:30:31.770336       7 security_group.go:420] handle delete security group sg-net-aaa
I0107 12:30:31.832391       7 pod.go:757] sync pod vm/vm-aaa-0 routed
I0107 12:30:36.806147       7 pod.go:401] take 5308 ms to handle sync pod vm/vm-aaa-0
I0107 12:30:36.900850       7 event.go:377] Event(v1.ObjectReference{Kind:"Subnet", Namespace:"", Name:"net-aaa", UID:"9c7461ee-587a-4210-a077-add4c28de6a8", APIVersion:"kubeovn.io/v1", ResourceVersion:"13009905", FieldPath:""}): type: 'Normal' reason: 'ResetLogicalSwitchAclSuccess' 
I0107 12:30:36.957239       7 vpc.go:147] handle status update for vpc ovn-cluster
I0107 12:30:36.958046       7 subnet.go:350] format subnet net-aaa, changed false
I0107 12:30:36.987285       7 subnet.go:999] delete u2o interconnection policy route for subnet net-aaa
I0107 12:30:36.990238       7 vpc.go:147] handle status update for vpc ovn-cluster
I0107 12:30:37.036146       7 subnet.go:1026] delete policy route for distributed subnet net-aaa
I0107 12:30:37.036177       7 subnet.go:2575] delete policy route for router: ovn-cluster, priority: 31000, match ip4.dst == 10.144.39.0/24
I0107 12:30:38.312480       7 ipam.go:319] delete subnet net-aaa
E0107 12:30:47.346057       7 pod.go:1391] failed to get subnet subnet.kubeovn.io "net-aaa" not found
E0107 12:30:47.346106       7 pod.go:487] subnet.kubeovn.io "net-aaa" not found
E0107 12:30:47.346114       7 pod.go:269] failed to get newPod nets subnet.kubeovn.io "net-aaa" not found
E0107 12:30:47.351348       7 pod.go:1391] failed to get subnet subnet.kubeovn.io "net-aaa" not found
E0107 12:30:47.351370       7 pod.go:487] subnet.kubeovn.io "net-aaa" not found
E0107 12:30:47.351377       7 pod.go:269] failed to get newPod nets subnet.kubeovn.io "net-aaa" not found
I0107 12:30:47.356888       7 pod.go:249] enqueue delete pod vm/vm-aaa-0
I0107 12:30:47.356961       7 pod.go:943] handle delete pod vm/vm-aaa-0
E0107 12:30:47.365316       7 pod.go:1391] failed to get subnet subnet.kubeovn.io "net-aaa" not found
E0107 12:30:47.365341       7 pod.go:487] subnet.kubeovn.io "net-aaa" not found
E0107 12:30:47.365353       7 pod.go:993] failed to get pod nets subnet.kubeovn.io "net-aaa" not found
I0107 12:30:47.443469       7 pod.go:439] take 86 ms to handle delete pod vm/vm-aaa-0

[zsxsoft]

[zsxsoft]

For case B, this subnet contains 2 pods. Pod 2 had multiple IPs but only a part of it deleted, and I can't delete any IP now. Pods are gone. I didn't find log, maybe flushed.

Then I patched all IPs, subnet still cannot be deleted. Logs follow:

[oilbeater]

Does the logical switch port in ovn-nb still exist?

[zsxsoft]

@oilbeater
For case A, no.

For case B, no after patched. I believe no before patched but I can't verify it now.

[oilbeater]

For case A, the subnet is deleted before the IP is created. As a result, when the pod is deleted, it cannot find the related subnet and fails to delete the IP CR. I believe case B behaves similarly.

To completely avoid this issue, a lock is needed for every subnet and related IP operation to maintain the correct order. However, this may slow down pod creation.

I will try to decouple pod deletion from the subnet and delete all related IP CRs based solely on pod annotations.

[zsxsoft]

Any progress?

[zsxsoft]

Any progress?