Stability under heavy load

[rvalle]

Description

https://polkawatch.app is now using the sidecar to calculate the staking rewards. We currently run a cluster of 10 nodes, 5 nodes per rely chain, to constantly recalculate the staking of every nominator on both rely chains.

Our processing is equivalent to 24x7 stress testing of the sidecar. In this context, we see frequent crashes of the sidecar with resource usage getting out of reasonable limits.

Expected vs. Actual Behavior

The sidecar should be stable and resource consumption be stable.

[rvalle]

We use autoheal, and restart sidecars servers when they become unresponsive for more than 10 seconds. We see this happening more or less once per hour under heavy stress.

[TarikGul]

Hey thanks for the issue! I am definitely aware of some of the burdens on memory and performance our staking endpoints have, so I appreciate putting pressure on this.

Do you have any metrics and or snapshots you can share with us, that would help with this?

In addition, which endpoints are you hitting directly?

[rvalle]

The bulk of the work is done by the following endpoint

• this.api.staking.getStakingPayoutsByAccountId

But you can check the code, there are other calls to compose the list of nominators in an era.

https://gitlab.com/polkawatch/polkawatch/-/blob/main/packages/indexer/src/sidecar.service.ts?ref_type=heads

As for resources, we have 8GB RAM 4 CPU SDD server for each rely chain. 2 in total Each running 5x sidecar nodes. There is another server that does the load balancing.

[TarikGul]

Thanks for letting me know. It's on the queue to tackle, there are some priorities around the asset hub migration that need to be finished on my end, but there is I believe on individual looking into the performance specifically that is helping with the effort.

I'll post any updates here!

[rvalle]

@TarikGul we are presenting our next proposal for polkawatch, we will include a task to help tackling this issue. at the end of the day we can easily reproduce it.

[TarikGul]

@rvalle Thanks for the patience I was OOO for the month of August. Feel free to attach that proposal here, would love to see some help on the issue :) Thanks so much!

[gituser]

Most likely #1485