bug: informer_factory.lua:310: watch failed, kind: Endpoints, reason: ReadBodyError

[zhangchl007]

Current Behavior

I install apisix on kubernetes cluster A , but I want to watch another kubernetes cluster B, as the official document, I specify the config as below, I got this issue:

discovery:
kubernetes:
- id: stg
service:
schema: https
host: 10.5.16.105
port: "6443"
client:
ssl_verify: false
token: xxxxxx

If I change the config of apisix to watch cluster A, everything is okay, but due to some reason, I have to separate apisix from in cluster, and I also ensure the token is valid for kubernetes cluster B below:

curl -k -H "Authorization: Bearer $TOKEN" https://10.5.16.105:6443/api/v1/namespaces/default
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name": "default",
"uid": "cb64fe72-1516-42c4-9b50-76566dc5e322",
"resourceVersion": "203",
"creationTimestamp": "2025-06-27T07:32:44Z",
"labels": {
"kubernetes.io/metadata.name": "default"
},
"managedFields": [
{
"manager": "kube-apiserver",
"operation": "Update",
"apiVersion": "v1",
"time": "2025-06-27T07:32:44Z",
"fieldsType": "FieldsV1",
"fieldsV1": {"f:metadata":{"f:labels":{".":{},"f:kubernetes.io/metadata.name":{}}}}
}
]
},
"spec": {
"finalizers": [
"kubernetes"
]
},
"status": {
"phase": "Active"
}
}

Expected Behavior

I expect we can install APISIX on kubernetes cluster A and watch the service endpoints for another kubernetes cluster B

Error Logs

the error log:
2025/07/20 01:32:50 [error] 55#55: *23179 [lua] informer_factory.lua:310: watch failed, kind: Endpoints, reason: ReadBodyError, message : closed, context: ngx.timer
2025/07/20 01:34:20 [error] 55#55: *24840 stream [lua] informer_factory.lua:310: watch failed, kind: Endpoints, reason: ReadBodyError, message : closed, context: ngx.timer
2025/07/20 01:34:20 [error] 55#55: *24842 [lua] informer_factory.lua:310: watch failed, kind: Endpoints, reason: ReadBodyError, message : closed, context: ngx.timer

Steps to Reproduce

1 run APISIX in kubernetes A
2. watch service discovery for kubernetes B

Environment

• APISIX version (run apisix version):
  apache/apisix:3.13.0-debian

• Operating system (run uname -a):
  azureuser@aks-nodepool1-27001917-vmss000008:$ lsb_release -a
  No LSB modules are available.
  Distributor ID: Ubuntu
  Description: Ubuntu 22.04.5 LTS
  Release: 22.04
  Codename: jammy
  azureuser@aks-nodepool1-27001917-vmss000008:$ uname -a
  Linux aks-nodepool1-27001917-vmss000008 5.15.0-1090-azure feature: checked the id of route. #99-Ubuntu SMP Thu May 22 21:15:50 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

• OpenResty / Nginx version (run openresty -V or nginx -V):
  image: apache/apisix:3.13.0-debian

• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
  image: docker.io/bitnami/etcd:3.5.10-debian-11-r2
  helm -n apisix list
  NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
  apisix apisix 1 2025-07-09 07:27:50.058773851 +0000 UTC deployed apisix-2.11.1 3.13.0

• APISIX Dashboard version, if relevant:

• Plugin runner version, for issues related to plugin runners:

• LuaRocks version, for installation issues (run luarocks --version):

[zhangchl007]

the token has permission to watch the service endpoints:

[zhangchl007]

Sorry, but kubernetes service discovery are working for both of cluster A and B， I am not sure why the issue still remain.

[Baoyuantop]

Can you manually access the endpoints api of cluster B from cluster A? Can you also manually send requests from the APISIX container and confirm if it works?

[Baoyuantop]

Hi @zhangchl007, any updates?

[zhangchl007]

@Baoyuantop As I mentioned apisix use kubernetes service discovery, it's working for both of cluster A and B, but I don't know why the error always occurs.

[Baoyuantop]

Can you manually access the endpoints api of cluster B from cluster A? Can you also manually send requests from the APISIX container and confirm if it works?

@zhangchl007, I mean, have you tried testing this?