getStateV3 with selection on fields

[arkh-consensys]

Currently getStateV2 /eth/v2/debug/beacon/states/{state_id} returns the full beacon state which is heavy in response size +800MB ~ 1GB.
There are scenarios that the consumer is only interested in partial parts of the state, but still had to fetch the full beacon state which puts load on the beacon node to stream this response and also on the consumer itself to fetch and parse it.

I would like to propose a getStateV3 where we can pass a filter param to only retrieve a selected set of higher level fields from the state /eth/v2/debug/beacon/states/{state_id}?filter=nextWithdrawalValidatorIndex, totalActiveBalanceIncrements, pendingDepositBalanceToConsume` ...

With this approach we lower the barrier on accessing to the beacon state.

[rolfyone]

what data is likely to be needed thats not already available via non debug endpoints?

To clarify, a filter as suggested is hard WRT the debug endpoint, because we're looking often to return a full SSZ object.

If there's endpoints that are missing, they're better mapped to /eth/v1/beacon/states/{state_id}/field like many other have potentially...

[mcdee]

Not commenting directly on the proposal, but a quick check of the sizes of this endpoint's response (in bytes) over the wire:

• JSON 755_681_269
• compressed JSON 119_710_818
• SSZ 257_384_660
• compressed SSZ 104_992_421

so there are definitely existing ways to reduce the size of the response. I'm aware that this does not fully address some in-memory concerns, however the ~1GB response size suggested by the OP already has mitigations.

[arkh-consensys]

what data is likely to be needed thats not already available via non debug endpoints?

To clarify, a filter as suggested is hard WRT the debug endpoint, because we're looking often to return a full SSZ object.

If there's endpoints that are missing, they're better mapped to /eth/v1/beacon/states/{state_id}/field like many other have potentially...

• Would it be still hard if the returned response is limited to JSON?

• Adding an endpoint for every single field is cumbersome. For some objects like partial_withdrawals, pending_deposits, it makes sense to have a separate endpoint, but for other state variables like nextWithdrawalValidatorIndex , totalActiveBalanceIncrements , etc, would you still recommend a separate endpoint?

[arkh-consensys]

Not commenting directly on the proposal, but a quick check of the sizes of this endpoint's response (in bytes) over the wire:

• JSON 755_681_269
• compressed JSON 119_710_818
• SSZ 257_384_660
• compressed SSZ 104_992_421

so there are definitely existing ways to reduce the size of the response. I'm aware that this does not fully address some in-memory concerns, however the ~1GB response size suggested by the OP already has mitigations.

Good point about reducing the size of the response. Although my observation on ssz was different than 100MB. Actually was around 700MB. The response sizes that I mentioned above were for holesky (before the incident).

Still if the response size is lower, then the consumer should fetch and decompress and unnecessary large object to only sue a few field values.

[nflaig]

Although my observation on ssz was different than 100MB. Actually was around 700MB. The response sizes that I mentioned above were for holesky (before the incident).

The state on holesky is ~258MB, this sounds like a bug or misuse to me but irrespective of that I am not opposed to this feature however I think this is something that needs to be proven to be feasible by doing a PoC on a client implementation and show that it's a) doable at all to work with ssz and b) is reasonable in terms of complexity before we add this to the spec and "force" everyone to implement it.