Calico loses LoadBalancer routes

[bootc]

I reported this issue here too: rancher/rke2#9484. Lots of calico-node log output is in that issue.

I have the same issue on 4 independent clusters. They all have in common:

• rke2 v1.34.3+rke2r1 or v1.35.0+rke2r1
• Calico (bundled chart and images)
• nftables dataplane in Calico and kube-proxy
• BGP for LoadBalancer route advertisements
• LoadBalancers with externalTrafficPolicy: Local

After a certain amount of time (I can't currently pin it on anything else), Calico loses the /32 (for IPv4) and /128 (for IPv6) routes for the LoadBalancers with externalTrafficPolicy: Local. Consequently the covering route takes effect and traffic is routed towards nodes which aren't running the service, so traffic is blackholed.

The problem appears to be that the static routes actually disappear from /etc/calico/confd/config/bird_aggr.cfg or /etc/calico/confd/config/bird6_aggr.cfg within the calico-node pods. The routes are then not advertised.

Restarting the calico-node pods seems to resolve the problem for a time, until it reoccurs.

Expected Behavior

The routes should persist.

Current Behavior

Calico loses the /32 or /128 routes for externalTrafficPolicy: Local LoadBalancer services.

Possible Solution

Steps to Reproduce (for bugs)

Not sure yet, currently mostly the passage of time.
1.
2.
3.
4.

Context

I need some of my LoadBalancer services to retain the source IP address, hence using externalTrafficPolicy: Local.

Your Environment

• Calico version: 3.31.2
• Calico dataplane: nftables
• Orchestrator version: rke2 v1.35.0+rke2r1, rke2 v1.34.3+rke2r1
• Operating System and version: Debian 13 and Ubuntu 24.04, kernels 6.8.0-1040-raspi, 6.12.57+deb13-amd64, 6.17.8+deb13-amd64
• Link to your project (optional): n/a

[caseydavenport]

Hmm, that definitely seems like a bug. Could you share as much of the following as you're able?

• kubectl get bgpconfiguration -o yaml
• Full logs from an affected calico/node pod
• kubectl get service -o yaml from one of the impacted Local services
• kubectl get endpointslices -o yaml

Hopefully this can tell me why Calico is deciding to withdraw the routes.

[bootc]

@caseydavenport there are logs in the linked ticket; did you see those? I can post more logs, but it's all very noisy indeed - the log I have is 3.5MB and contains a lot of bird: Netlink: File exists noise which is unrelated as it doesn't occur on the other clusters and doesn't seem to have any other effect.

I just had the problem happen again on my home cluster; I was rebooting nodes for an OS and kernel upgrade. In this case the haproxy-ingress pods that I notice this with immediately were not running on any of the nodes I was rebooting, though in all likelihood some other pods providing similar LoadBalancer services were moved around.

The BGPConfiguration:

apiVersion: v1
items:
- apiVersion: projectcalico.org/v3
  kind: BGPConfiguration
  metadata:
    creationTimestamp: "2024-03-02T13:17:03Z"
    generation: 1
    labels:
      kustomize.toolkit.fluxcd.io/name: k8s-hudson-flux
      kustomize.toolkit.fluxcd.io/namespace: flux-system
    name: default
    resourceVersion: "648205771"
    uid: 59f14d6c-1089-47b9-af27-8cfba4bf000b
  spec:
    asNumber: 64512
    serviceClusterIPs:
    - cidr: 10.45.0.0/16
    - cidr: 2001:67c:2e80:ff:1::/112
    serviceLoadBalancerAggregation: Enabled
    serviceLoadBalancerIPs:
    - cidr: 81.187.199.48/28
    - cidr: 2001:67c:2e80::/64
kind: List
metadata:
  resourceVersion: ""

The Service:

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: k8s-haproxy.boo.tc
    meta.helm.sh/release-name: haproxy-ingress
    meta.helm.sh/release-namespace: bootc-system
    metallb.io/ip-allocated-from-pool: default
    metallb.universe.tf/ip-allocated-from-pool: default
    metallb.universe.tf/loadBalancerIPs: 2001:67c:2e80::4,81.187.199.51
  creationTimestamp: "2022-02-22T00:40:40Z"
  labels:
    app.kubernetes.io/instance: haproxy-ingress
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: haproxy-ingress
    app.kubernetes.io/version: v0.15.1
    helm.sh/chart: haproxy-ingress-0.15.1
    helm.toolkit.fluxcd.io/name: haproxy-ingress
    helm.toolkit.fluxcd.io/namespace: bootc-system
  name: haproxy-ingress
  namespace: bootc-system
  resourceVersion: "1353661216"
  uid: b5df33e8-3a1f-4194-b973-dcf13bb0688f
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 10.45.74.168
  clusterIPs:
  - 10.45.74.168
  - 2001:67c:2e80:ff:1::b388
  externalTrafficPolicy: Local
  healthCheckNodePort: 30848
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: RequireDualStack
  ports:
  - name: http-80
    nodePort: 32748
    port: 80
    protocol: TCP
    targetPort: http
  - name: https-443
    nodePort: 30266
    port: 443
    protocol: TCP
    targetPort: https
  - name: 22-tcp
    nodePort: 32065
    port: 22
    protocol: TCP
    targetPort: 22-tcp
  selector:
    app.kubernetes.io/instance: haproxy-ingress
    app.kubernetes.io/name: haproxy-ingress
  sessionAffinity: None
  type: LoadBalancer
status:
  loadBalancer:
    ingress:
    - ip: 2001:67c:2e80::4
      ipMode: VIP
    - ip: 81.187.199.51
      ipMode: VIP

The related EndpointSlices:

apiVersion: v1
items:
- addressType: IPv6
  apiVersion: discovery.k8s.io/v1
  endpoints:
  - addresses:
    - 2001:67c:2e80:937:a30:f4fa:e091:f4e9
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: clemens
    targetRef:
      kind: Pod
      name: haproxy-ingress-64d6f4669c-l7xw4
      namespace: bootc-system
      uid: 096cda65-e329-4fe6-aaf9-162123822b19
  - addresses:
    - 2001:67c:2e80:923:955:59bc:f95d:2d5c
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: jude
    targetRef:
      kind: Pod
      name: haproxy-ingress-64d6f4669c-x5bwx
      namespace: bootc-system
      uid: d804d010-bd27-48f1-80a1-24ad033309aa
  - addresses:
    - 2001:67c:2e80:9d1:7f4e:5ff6:2112:673d
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: david
    targetRef:
      kind: Pod
      name: haproxy-ingress-64d6f4669c-9qx7x
      namespace: bootc-system
      uid: 7af51175-1eed-4ba1-a62f-45537c8de088
  kind: EndpointSlice
  metadata:
    creationTimestamp: "2024-09-30T16:35:05Z"
    generateName: haproxy-ingress-
    generation: 1372
    labels:
      app.kubernetes.io/instance: haproxy-ingress
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: haproxy-ingress
      app.kubernetes.io/version: v0.15.1
      endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
      helm.sh/chart: haproxy-ingress-0.15.1
      helm.toolkit.fluxcd.io/name: haproxy-ingress
      helm.toolkit.fluxcd.io/namespace: bootc-system
      kubernetes.io/service-name: haproxy-ingress
    name: haproxy-ingress-gjljq
    namespace: bootc-system
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: haproxy-ingress
      uid: b5df33e8-3a1f-4194-b973-dcf13bb0688f
    resourceVersion: "1359600889"
    uid: 3bf629ff-80cd-481e-a4a5-9181e868fd5f
  ports:
  - name: 22-tcp
    port: 22
    protocol: TCP
  - name: http-80
    port: 80
    protocol: TCP
  - name: https-443
    port: 443
    protocol: TCP
- addressType: IPv4
  apiVersion: discovery.k8s.io/v1
  endpoints:
  - addresses:
    - 10.44.196.60
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: clemens
    targetRef:
      kind: Pod
      name: haproxy-ingress-64d6f4669c-l7xw4
      namespace: bootc-system
      uid: 096cda65-e329-4fe6-aaf9-162123822b19
  - addresses:
    - 10.44.98.156
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: jude
    targetRef:
      kind: Pod
      name: haproxy-ingress-64d6f4669c-x5bwx
      namespace: bootc-system
      uid: d804d010-bd27-48f1-80a1-24ad033309aa
  - addresses:
    - 10.44.103.61
    conditions:
      ready: true
      serving: true
      terminating: false
    nodeName: david
    targetRef:
      kind: Pod
      name: haproxy-ingress-64d6f4669c-9qx7x
      namespace: bootc-system
      uid: 7af51175-1eed-4ba1-a62f-45537c8de088
  kind: EndpointSlice
  metadata:
    creationTimestamp: "2024-09-30T16:35:05Z"
    generateName: haproxy-ingress-
    generation: 1372
    labels:
      app.kubernetes.io/instance: haproxy-ingress
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: haproxy-ingress
      app.kubernetes.io/version: v0.15.1
      endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
      helm.sh/chart: haproxy-ingress-0.15.1
      helm.toolkit.fluxcd.io/name: haproxy-ingress
      helm.toolkit.fluxcd.io/namespace: bootc-system
      kubernetes.io/service-name: haproxy-ingress
    name: haproxy-ingress-rwr4h
    namespace: bootc-system
    ownerReferences:
    - apiVersion: v1
      blockOwnerDeletion: true
      controller: true
      kind: Service
      name: haproxy-ingress
      uid: b5df33e8-3a1f-4194-b973-dcf13bb0688f
    resourceVersion: "1359600885"
    uid: ac22492c-a7f0-402d-b1c5-55485666dc87
  ports:
  - name: 22-tcp
    port: 22
    protocol: TCP
  - name: http-80
    port: 80
    protocol: TCP
  - name: https-443
    port: 443
    protocol: TCP
kind: List
metadata:
  resourceVersion: ""

[bootc]

In fact draining a node, or it coming back after reboot, seems to be a pretty good trigger for this condition.