Continuos loop in KubeServiceState when Kubernetes api server returns error

[ecejnj42]

Version

5.0.0

Context

KubeServiceState is using a WebSocket to connect to k8s api server to watch for change on "endpoints" from an specific version (resourceVersion)
This mechanism is working correctly until there is an error reported from k8s api server similar to this one:

{\"type\":\"ERROR\",\"object\":{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"too old resource version: 320409672 (320435551)\",\"reason\":\"Expired\",\"code\":410}}

No clear when this happens but probably when there are many changes and subscribed version gets too old.
In such situations the WebSocket is closed (probably by k8 api server) but current code in KubeServiceState just re-opens and subscribe with same old version.
This causes a continuos loop of closing because of error and re-opening with old version.

I guess that in such situation communications should be reinitiated again by the http client in KubeResolverImpl to force a fresh new version (resourceVersion)

Steps to reproduce

Best way to reproduce the error (although not always happens) is just to scalate-in the instances (increase PODs for related service) and later scalate-out (decrease PODs for related service).
Note during this whole process of scalate-in and scalate-out some "traffic" should be sent to the service (other way the WebSocket is disposed and the problem dissapears)

Do you have a reproducer?

No response

[vietj]

@ecejnj42 thank you for the proper description, could you tell me how / where the error is reported ?

[ecejnj42]

Hi

This error is reported from Kubernetes API Server during the WebSocket communication for updates in class KubeServiceState.
I just added some logs to class KubeServiceState, in method "handleUpdate" to see the error.
The exact conditions when this happens is unclear to me. I was just overloading my vertx application, so I assume that for some reason some notifications (from changes in k8s API server) could have been lost, therefore causing the resourceVersion to get too old.
Or perhaps there were a lot of changes in endpoints in K8s (as I was scaling a lot of pods), not allowing the k8s API to notify all changes (If this wete the case, perhaps from the vertx point of view would be simpler to subscribe only to changes to an specific service, instead to all services)

[vietj]

ok I am trying to find out the flow, in order to try to write a reproducer using using kindcontainer