KIC 3.5 issue installation no endpoints for service: "ingress/ingress-kong-gateway-admin"

[cello86]

Hi All,
I tried to install via helm the ingress controller 3.5 version but the pods were always not ready.

pod/ingress-kong-controller-59676cf5bf-2q4c8   0/1     CrashLoopBackOff   6 (2m1s ago)   6m22s
pod/ingress-kong-gateway-d547d466b-f29gt       0/1     Running            0              6m22s

The logs on ingress-kong-controller reported:

2025-09-24T16:14:36Z    error   setup.initialize-kong-clients   Failed to create kong client(s), retrying...    {"delay": 1, "error": "no endpoints for service: \"ingress/ingress-kong-gateway-admin\""}
2025-09-24T16:14:37Z    info    Signal received, shutting down  {"v": 0, "graceful_period": "0s", "signal": "terminated"}
Error: failed to create manager: failed to create manager: unable to build kong api client(s): All attempts fail:
#1: no endpoints for service: "ingress/ingress-kong-gateway-admin"

I suspect that the problem is related to missing running pods behind service service/ingress-kong-gateway-admin and the ingress-kong-gateway probe receive an http 503 on /status/ready endpoint. These were the logs on ingress-kong-gateway pod:

2025/09/24 16:14:30 [notice] 1407#0: *1292 [lua] ready.lua:121: fn(): not ready for proxying: no configuration available (empty configuration present), client: 10.64.11.10, server: kong_status, request: "GET /status/ready HTTP/1.1", host: "10.244.4.117:8100"

Could someone help me to understand the issue on the chart?

Thanks,
Marcello

[cello86]

I'm trying to debug deeper the issue and I installed KIC 3.5 in sidecar mode and it worked fine. I also tried to install KIC in 2 different deployment and disable the gateway discovery on the controller and point to the admin service via service dns and also in this configuration worked fine. I cannot use the service DNS because the configuration update from the controller will be deployer on a single gateway pod.
I noticed that the serviceaccount used on the controller didn't have the permission to access only to the endpointslices and not endpoints and I tried to force the permissions with these manifests:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kong-controller-endpoints-access
rules:
- apiGroups: [""]
  resources: ["endpoints"]
  verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kong-controller-endpoints-access
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kong-controller-endpoints-access
subjects:
- kind: ServiceAccount
  name: ingress-kong-controller-kong
  namespace: ingress
I had no luck with this change.

Marcello

[cello86]

I also tried to apply the parameter publishNotReadyAddresses as reported on Kong/kubernetes-ingress-controller#5110 and change the CONTROLLER_GATEWAY_DISCOVERY_DNS_STRATEGY to"pod" without successo.

I tried the deployment with versions 3.1, 3.2, 3.3 and 3.4 without success.

I tested the serviceaccount from the kong proxy and the endpoints and endpointslices reported ready gateway/proxy pods:

# kubectl proxy --port=8001 &
# curl -s "http://localhost:8001/api/v1/namespaces/ingress/endpoints/ingress-kong-proxy-kong-admin" | jq '.subsets[].addresses'
[
  {
    "ip": "10.244.4.244",
    "nodeName": "aks-infra-24197528-vmss000001",
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-f9vr2",
      "uid": "c15f05a0-acf6-40b9-a80e-a7b2a6f14fb0"
    }
  },
  {
    "ip": "10.244.5.43",
    "nodeName": "aks-infra-24197528-vmss000000",
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-xzch6",
      "uid": "9724ab7c-d015-412a-8dc2-29c55d15286a"
    }
  }
# curl -s "http://localhost:8001/apis/discovery.k8s.io/v1/namespaces/ingress/endpointslices?labelSelector=kubernetes.io%2Fservice-name%3Dingress-kong-proxy-kong-admin" | jq '.items[].endpoints'
[
  {
    "addresses": [
      "10.244.5.43"
    ],
    "conditions": {
      "ready": true,
      "serving": true,
      "terminating": false
    },
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-xzch6",
      "uid": "9724ab7c-d015-412a-8dc2-29c55d15286a"
    },
    "nodeName": "aks-infra-24197528-vmss000000",
    "zone": "westeurope-2"
  },
  {
    "addresses": [
      "10.244.4.244"
    ],
    "conditions": {
      "ready": true,
      "serving": true,
      "terminating": false
    },
    "targetRef": {
      "kind": "Pod",
      "namespace": "ingress",
      "name": "ingress-kong-proxy-kong-f9vr2",
      "uid": "c15f05a0-acf6-40b9-a80e-a7b2a6f14fb0"
    },
    "nodeName": "aks-infra-24197528-vmss000001",
    "zone": "westeurope-3"
  }
]
# curl -s "http://localhost:8001/api/v1/namespaces/ingress/endpoints/ingress-kong-proxy-kong-admin" | jq '.subsets[].addresses | length'
2
# curl -s "http://localhost:8001/apis/discovery.k8s.io/v1/namespaces/ingress/endpointslices?labelSelector=kubernetes.io%2Fservice-name%3Dingress-kong-proxy-kong-admin" | jq '.items[].endpoints | length'
2

Marcello