Sometimes the api-pod does not respond any more

[FamilieRo]

From time to time the api pod stops to respond. So everything like attaching volumes, cli etc. gets a timeout.
e.g.
kubectl mayastor get volumes -n openebs
Failed to list volumes. Error error in request: request timed out

In the logs of the api pod you will find something like

.[2m2024-12-19T13:08:39.383948Z.[0m .[31mERROR.[0m .[1;31mgrpc::operations::volume::client.[0m.[31m: .[1;31merror.[0m.[31m: Unavailable: status: Unavailable, message: "error trying to connect: deadline has elapsed", details: [], metadata: MetadataMap { headers: {} }.[0m

I have attached a full log file for you to analyse.

Removing the api pod brings up a new one and everything works directly.

The error occurs regardless of whether workload happens on the cluster or not.

openebs-api-rest-676998984d-svxhr-api-rest.log

[tiagolobocastro]

Could you please try to increase the cpu limits for the rest just in case?
Example:
mayastor.apis.rest.resources.limits.cpu=500m

And could you also share logs from agents-core pod, agent-core container?

[FamilieRo]

I have increased the cpu limit. At the moment no more timeouts. I will observe the behaviour.

However, I'm on holiday for a few days and will upload more logs if necessary.

[tiagolobocastro]

Would you be able to try another work around?
Leaving the resource limits as is but modifying the rest deployment args to include --max--workers=2 ?

[FamilieRo]

Hi!
We had no more problems with the API so far.
Nevertheless I left the CPU limit as is and included the max-workers in the args. The pod is running fine, will observe it.
Thanks!

[tiagolobocastro]

Sorry I probably wrote my previous message incorrectly. By as is I had meant, as it is by default of 100m but with max workers of 2.
Thank you!

[FamilieRo]

OK, sorry!
Changed the limit back to 100m.

[Abhinandan-Purkait]

@tiagolobocastro I believe the fix for this is increasing the resources and modifying the workers. We also have added a liveness probe.

[FamilieRo]

@tiagolobocastro
I did an upgrade to version 2.7.2 last week. So the args and the limits are set to the defaults and everythings seems to run smoothly. No problems at all.

[tiagolobocastro]

That may be because a liveness probe has been added to the api-rest so it gets restarted if it becomes unresponsive. You might see this on the pod restarts.

Did you have any luck with max-workers as 2?

[FamilieRo]

Hi,

no restarts since 01/16:

With version 2.7.1 the arg for max-workers=2 worked well. But as far as I can see the problem is gone.