`/blocks/:height` sometimes always return error message for a block

[Ljzn]

Description

Sometimes the /blocks/:height api always return this message:

{"code":500,"message":"No response received from RPC endpoint in 60s","stack":"Error: No response received from RPC endpoint in 60s\n    at WsProvider.value (/usr/src/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:406:32)\n    at Timeout._onTimeout (/usr/src/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:528:182)\n    at listOnTimeout (node:internal/timers:557:17)\n    at processTimers (node:internal/timers:500:7)","level":"error"}

Steps to Reproduce

Run substrate-api-sidecar connecting to a Kusama ws endpoint.

Expected vs. Actual Behavior

expected: Get block by height success.

actual: always return same error message for specific block.

[Ljzn]

It not affect other blocks, and the problem will gone after restarting the sidecar. So this maybe related with cache.

[jsdw]

Can you reliably reproduce this, or did the problem go away permanently for you?

It may have been lucky timing; perhaps the Kusama node being queried was being too slow to respond for a while and it happened to improve before you restarted Sidecar?

[Ljzn]

Can you reliably reproduce this, or did the problem go away permanently for you?

It may have been lucky timing; perhaps the Kusama node being queried was being too slow to respond for a while and it happened to improve before you restarted Sidecar?

Unfortunate, I don't know how to reproduce it. It happens about once a week.

In that situation, the rest api will respond that error immediately. Just like no new query was be made.

[Ljzn]

happened again, here is more log:

2022-10-20 05:09:53 error: RPC-CORE: getRuntimeVersion(at?: BlockHash): RuntimeVersion:: No response received from RPC endpoint in 60s
2022-10-20 05:09:53 error: RPC-CORE: chain(): Text:: No response received from RPC endpoint in 60s
2022-10-20 05:09:53 error: RPC-CORE: version(): Text:: No response received from RPC endpoint in 60s
2022-10-20 05:09:53 error: No response received from RPC endpoint in 60s 
 Error: No response received from RPC endpoint in 60s
    at WsProvider.value (/usr/src/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:406:32)
    at Timeout._onTimeout (/usr/src/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:528:182)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)

[Ljzn]

hi, @jsdw . I found this error can not only happened on /blocks/:height/, but also on other routes such as /node/version.

2022-10-20 08:31:51 error: RPC-CORE: getRuntimeVersion(at?: BlockHash): RuntimeVersion:: No response received from RPC endpoint in 60s
2022-10-20 08:31:51 error: No response received from RPC endpoint in 60s 
 Error: No response received from RPC endpoint in 60s
    at WsProvider.value (/usr/src/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:406:32)
    at Timeout._onTimeout (/usr/src/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:528:182)
    at listOnTimeout (node:internal/timers:557:17)
    at processTimers (node:internal/timers:500:7)
2022-10-20 08:31:51 error: GET /node/version 500 15001ms

[Ljzn]

Not only Kusama, many other chains has this error today. We are using the WS api of Onfinality. So this maybe related with their rate limit policy. Feel free to close this issue if it is WS's problem.

[jsdw]

Mmm, the logs do very much suggest that the node simply isn't responding in time. I have found in related sorts of queries that the RPC nodes can often get very slow and fail to respond within 60s. Perhaps have a go with pointing at a different RPC node, or in the worst case, run and sycn your own node so that the reponsiveness is within your control :)

@TarikGul is the request timeout configurable per chance?

[TarikGul]

@Ljzn Okay so first, I highly suggest running your own archive node. That way everything is under your control and easily monitor-able. Sidecar was never intended to be used heavily against public nodes.

is the request timeout configurable per chance?

Yes, this is configurable. I can add it as an ENV var. It gets attached to WsProvider as an option called timeout. This would be specific to the Websocket provider though, and is not an option for the http provider.

[Ljzn]

Thanks!

[cdiarra-ledger]

Hello,
I'm facing the same issue, and I'm running my own full archive nodes (in fact 8 of them).
From time to time, I'm seeing this same error No response received from RPC endpoint in 60s on a specific node, when querying a specific block (example : blocks/13325820).
I receive this message instantly, so I find the "timeout" log quite misleading.

• When querying the same block on a different node, it works well {…,”success”:true,…}
• When querying an other block on the same node, it works well {…,”success”:true,…}

So it doesn't seem that there is a timeout issue or a connectivity on our components.

Like @Ljzn, restarting the substrate-api make the query work again.

I'm running sidecar v13.1.0 and node v0.9.3.0

Any ideas ?

[TarikGul]

Hmm, yea this is an interesting issue becuase it's hard to reproduce. That being said few things to always potentially mitigate these issue:

1. Use the latest version of SAS which in this case would be 14.2.1

2. Always use the latest version of the node you are querying.

Questions:
What chain are you referencing specifically, polkadot?

I receive this message instantly, so I find the "timeout" log quite misleading.

Also this is very odd as well, do you have any logs you can attach for these errors so we can check some patterns and try to load test on our end>

[TarikGul]

Also what version of node.js are you running?

[cdiarra-ledger]

Hello,
Thanks for answering !
I'm not at liberty to update to the latest versions right now, but I made sure to make this on top our priority list, I hope in the next 2 weeks !
In the meantime I had to implement some sort of self-healing script that will reboot the SAS service whenever we detect a HTTP 500 error. I only kept 2 non production servers without this script, so it will be a little bit more difficult to reproduce the error.

But I kept some logs from the last time where we encoutered this issue :

curl localhost:30002/blocks/13325820
{"code":500,"message":"No response received from RPC endpoint in 60s","stack":"Error: No response received from RPC endpoint in 60s\n    at WsProvider.value (/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:406:32)\n    at Timeout._onTimeout (/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:528:182)\n    at listOnTimeout (node:internal/timers:559:17)\n    at processTimers (node:internal/timers:502:7)","level":"error"}

At the same time, here's the output on the SAS service :

2022-12-21 14:01:26 �[31merror�[39m: RPC-CORE: getBlock(hash?: BlockHash): SignedBlock:: No response received from RPC endpoint in 60s
2022-12-21 14:01:26 �[31merror�[39m: RPC-CORE: getRuntimeVersion(at?: BlockHash): RuntimeVersion:: No response received from RPC endpoint in 60s
2022-12-21 14:01:26 �[31merror�[39m: RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: No response received from RPC endpoint in 60s
2022-12-21 14:01:26 �[31merror�[39m: RPC-CORE: getStorage(key: StorageKey, at?: BlockHash): StorageData:: No response received from RPC endpoint in 60s
2022-12-21 14:01:26 �[31merror�[39m: No response received from RPC endpoint in 60s
Error: No response received from RPC endpoint in 60s
at WsProvider.value (/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:406:32)
at Timeout._onTimeout (/app/node_modules/@polkadot/rpc-provider/cjs/ws/index.js:528:182)
at listOnTimeout (node:internal/timers:559:17)
at processTimers (node:internal/timers:502:7)

I can confirm again that it took less than 1second to have this error message.

All of our nodes are connected to polkadot mainnet.

node.js version used is : v16.18.1 (LTS)

[TarikGul]

Hey @cdiarra-ledger Sorry this got lost in the queue. Are you still experiencing this issue?