Flashblocks: issue when handling chainstate updates on flashblocks consensus sync vs default sync

[sieniven]

Describe the bug

Issue

Referencing to the really nice PR added by @0x00101010 #19786, it seems like on further testing, there are 3 potentially problematic issues that could surface from this design.

1. Currently FCU sent on the flashblocks consensus sets unsafe, safe and finalized as the flashblocks sequence hash. This is incorrect and violates optimism finality states, and is resolved in the WIP PR: fix(flashblocks): add chainstate check on consensus update and add flashblock websocket connection retry delay #20490
2. The current flashblocks consensus layer doesnt take into account the possibility that the default CL sync could be ahead of flashblocks. This can be resolved pretty easily by adding a chainstate check on the flashblocks consensus client to check with the chain provider the current latest tip, and skip if the FCU call is behind the curren tip. A WIP PR fix(flashblocks): add chainstate check on consensus update and add flashblock websocket connection retry delay #20490 should resolve this.
3. The other issue is flashblocks consensus layer is ahead of the CL sync, then this situation may happen which could be undesirable.
   • flashblocks advances chainstate to x -> sends FCU to x
   • CL sync is still at x-1 -> sends FCU to x-1
   • chainstate reorgs to x-1
   • Next flashblock sequence advances chainstate to x+1

I am thinking of what are the possible solutions to resolve this - perhaps allowing flashblocks RPC to advance the EL's chainstate might not be such a good idea afterall? However the performance improvements, if the chainstate is able to sync with flashblocks, is massive. Also, resolving issue 2 is not trivial as if the resolution is on the EL layer, it might touch core protocol which is dangerous if an actual re-org needs to happen.

Solution 1

The easiest and least complicated solution to resolve this is for flashblocks consensus layer to not be able to advance chainstate, but pre-warm the payload and insert into the locally built blocks cache. Subsequently on CL sync, engine_newPayload will skip the payload validation logic all together which will help greatly in RPC syncs.

However, this solution could limit the flashblocks RPC potential since driving the chainstate and reducing chain latencies is dependent on the default CL sync.

Solution 2

Perhaps an alternative solution will be to design a separate accumulated flashblock sequences chainstate that is used in the exposed eth APIs. We maintain this flashblocks state layer separate from the default sync (which updates the canonical chainstate). This flashblocks state layer is flushed once the default sync's block is received, and we similarly we implement solution 1 by default to speed up default RPC sync.

Steps to reproduce

Node logs

In high stress test env, we can notice that flashblocks consensus and the default sync (FCU call from CL)

Platform(s)

No response

Container Type

Docker

What version/commit are you on?

main branch, v1.9.3

What database version are you on?

Which chain / network are you on?

What type of node are you running?

Archive (default)

What prune config do you use, if any?

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[sieniven]

@0x00101010 @mattsse Would love to hear your thoughts about this :) Thanks for taking the time 🙏

[sieniven]

@mattsse @0x00101010 #20490 For now, I intend to work on changing the flashblocks consensus with solution 1 as a working solution for us to use for now if that sounds okay? However I believe we can discuss further the design and work towards achieving solution 2, by allowing flashblocks consensus to also advance the chainstate in a safer manner.

[0x00101010]

Thanks @sieniven for working on this.

The point of flashblocks consensus layer is to avoid needing a CL to sync the chain.

If you want to have a CL to drive the chain forward, then you can choose not to enable --flashblock-consensus flag, and in that case CL could drive the chain forward while EL listens to flashblocks stream and answers flashblock related rpc requests.

[mattsse]

The current flashblocks consensus layer doesnt take into account the possibility that the default CL sync could be ahead

it isnt intended to be run in addition to op-node, if you already have op-node + flashblocks then you only need it for the pending state.

this service is really just convenience to basically follow flashblcoks entirely and trust that

[sieniven]

@0x00101010 @mattsse I see this makes sense now, thanks for clarifying. If thats the case perhaps maybe we can look into an alternative to support flashblocks RPC to be able to use both CL sync and flashblocks consensus?

The reason for this is because flashblocks (as of now) are still unreliable state changes, and are fallible (for example, if rollup-boost chooses default EL payload on failures, leader sequencer on optimism arch changes to non-flashblock sequencing, etc), which could potentially result in the flashblocks consensus to be unable to sync.

Since flashblocks consensus flag is meant to support syncing only from the flashblock builder source, perhaps if it sounds good to you guys, I will add an alternative solution to supporting a hybrid sync approach? We could look to achieve solution 1 first (use flashblock sequences to pre-warm local execution block cache) to optimize syncing with 2 sources and FCU to be driven by CL sync.

Subsequently, in the long term, allowing support for eth apis to access the flashblocks sequence manager cache layer, allowing flashblock RPCs to be optimized with flashblocks low latencies while canonical chainstate is still driven by the CL.

[sieniven]

The current flashblocks consensus layer doesnt take into account the possibility that the default CL sync could be ahead

it isnt intended to be run in addition to op-node, if you already have op-node + flashblocks then you only need it for the pending state.

this service is really just convenience to basically follow flashblcoks entirely and trust that

I understand that by turning off flashblocks-consensus, we could by default achieve the hybrid sync (flashblocks as pending state). However, perhaps we could optimize this - the current logic (with flashblocks-consensus disabled) still relies on the default CL sync height (which determines the next pending height). Solution 1 described above doesnt resolve this, but would help with the CL sync to be faster (and not do double validation compute on the same payload). However, I believe if we could expose the local sequence manager cache (difference between canonical chainstate vs flashblock chainstate) to the eth api handler, we could enjoy the best of both worlds?