"Invalid string length" error when querying states

[jshufro]

Describe the bug

curl -Ssv "http://consensus:5052/eth/v2/debug/beacon/states/finalized"
*   Trying 172.21.0.12:5052...
* Connected to consensus (172.21.0.12) port 5052
> GET /eth/v2/debug/beacon/states/finalized HTTP/1.1
> Host: consensus:5052
> User-Agent: curl/8.4.0
> Accept: */*
> 
< HTTP/1.1 500 Internal Server Error
< vary: Origin
< access-control-allow-origin: *
< content-type: application/json; charset=utf-8
< content-length: 84
< Date: Thu, 23 Nov 2023 04:16:45 GMT
< Connection: keep-alive
< Keep-Alive: timeout=72
< 
{ [84 bytes data]
* Connection #0 to host consensus left intact
{"statusCode":500,"error":"Internal Server Error","message":"Invalid string length"}

Expected behavior

State json or an error that is more descriptive

Steps to reproduce

(see above curl)

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.12.0

[jshufro]

NB: This was on holesky, which I suppose has quite large states

[nflaig]

We are hitting several limits now using json, there might be possible workarounds but those are potentially really slow and still require a lot of memory.

The error is thrown during serialization but the message it definitely not great, should be more descriptive but proper solution is to support ssz for all apis (#6080) and follow spec (e.g. set all metadata headers).

[wemeetagain]

just tried again, this is still an issue

[nflaig]

It's kinda good that it errors before converting the whole state to a stringified json as it would crash the node with default memory limit.

We support ssz on almost all apis now, just need to get alignment on the state validators api ethereum/beacon-APIs#449 to be able to return a ssz response and this should be resolved.

I don't think it's worth to try and optimize the json response, if that's even technically possible.