Mismatched block state root #17500
Description
Describe the bug
At some point today my node went down.
Trying to restart it got this error.
backfill sync failed err=stage encountered an error in block #22945786: validation error: mismatched block state root: got 0x0127d90e6965d277f94f2d3a088bf44f6b46ca643279168250e458dc919d47ac, expected 0xb579aab3ceca5a7d906eb48a32a766b017da92a50dbd474697cb0740de78305c
Steps to reproduce
Not clear
Node logs
2025-07-18T20:06:47.275809Z DEBUG reth::cli: Event: BackfillSyncStarted
2025-07-18T20:06:47.275811Z DEBUG engine::tree: received new engine message msg=Event(BackfillSyncStarted)
2025-07-18T20:06:47.275816Z DEBUG engine::tree: received backfill sync started event
2025-07-18T20:06:47.275887Z DEBUG ChainOrchestrator::poll: static_file: StaticFileProducer started targets=StaticFileTargets { headers: None, receipts: None, transactions: None, block_meta: Some(0..=22948275) }
2025-07-18T20:06:47.275902Z DEBUG ChainOrchestrator::poll: static_file: StaticFileProducer finished targets=StaticFileTargets { headers: None, receipts: None, transactions: None, block_meta: Some(0..=22948275) } elapsed=1.822µs
2025-07-18T20:06:47.275903Z DEBUG reth_node_events::node: Static File Producer started targets=StaticFileTargets { headers: None, receipts: None, transactions: None, block_meta: Some(0..=22948275) }
2025-07-18T20:06:47.275908Z DEBUG reth_node_events::node: Static File Producer finished targets=StaticFileTargets { headers: None, receipts: None, transactions: None, block_meta: Some(0..=22948275) } elapsed=1.822µs
2025-07-18T20:06:47.275962Z DEBUG ChainOrchestrator::poll: pruner: Pruner started tip_block_number=22948275
2025-07-18T20:06:47.275967Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning started segment=Headers purpose=StaticFile to_block=22948275 prune_mode=Before(22948276)
2025-07-18T20:06:47.275983Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning finished segment=Headers purpose=StaticFile to_block=22948275 prune_mode=Before(22948276) segment_output.pruned=0
2025-07-18T20:06:47.275986Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=22948275 prune_mode=Before(22948276)
2025-07-18T20:06:47.275992Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=22948275 prune_mode=Before(22948276) segment_output.pruned=0
2025-07-18T20:06:47.275994Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning started segment=Receipts purpose=StaticFile to_block=22499999 prune_mode=Before(22500000)
2025-07-18T20:06:47.275998Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning finished segment=Receipts purpose=StaticFile to_block=22499999 prune_mode=Before(22500000) segment_output.pruned=0
2025-07-18T20:06:47.275999Z DEBUG ChainOrchestrator::poll: pruner: Pruner finished tip_block_number=22948275 elapsed=34.419µs deleted_entries=0 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 18446744073709551615, deleted: 0 }), time_limit: None } output=PrunerOutput { progress: Finished, segments: [(Headers, SegmentOutput { progress: HasMoreData(Unknown), pruned: 0, checkpoint: Some(SegmentOutputCheckpoint { block_number: None, tx_number: None }) }), (Transactions, SegmentOutput { progress: Finished, pruned: 0, checkpoint: None }), (Receipts, SegmentOutput { progress: Finished, pruned: 0, checkpoint: None })] } stats=[]
2025-07-18T20:06:47.276019Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=8.766µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:47.276052Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=6.313µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:47.276058Z INFO reth_node_events::node: Preparing stage pipeline_stages=1/15 stage=Era checkpoint=0 target=None
2025-07-18T20:06:47.276066Z INFO reth_node_events::node: Executing stage pipeline_stages=1/15 stage=Era checkpoint=0 target=None
2025-07-18T20:06:47.276069Z DEBUG ChainOrchestrator::poll: sync::stages::headers: Commencing sync tip=Hash(0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe) head=0xf09724e5624a9e301f290f6b08a411814330769cb9f98f668e8438f7c26b814c
2025-07-18T20:06:47.276073Z INFO reth_node_events::node: Finished stage pipeline_stages=1/15 stage=Era checkpoint=0 target=None
2025-07-18T20:06:47.276078Z INFO reth_node_events::node: Preparing stage pipeline_stages=2/15 stage=Headers checkpoint=22948275 target=0
2025-07-18T20:06:48.024612Z DEBUG ChainOrchestrator::poll: sync::stages::headers: Commencing sync tip=Hash(0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe) head=0xf09724e5624a9e301f290f6b08a411814330769cb9f98f668e8438f7c26b814c
2025-07-18T20:06:48.024658Z INFO ChainOrchestrator::poll: sync::stages::headers: Received headers total=32 from_block=22948307 to_block=22948276
2025-07-18T20:06:48.024769Z INFO ChainOrchestrator::poll: sync::stages::headers: Writing headers total=32
2025-07-18T20:06:48.024810Z INFO reth_node_events::node: Executing stage pipeline_stages=2/15 stage=Headers checkpoint=22948275 target=0
2025-07-18T20:06:48.025466Z INFO ChainOrchestrator::poll: sync::stages::headers: Writing headers hash index total=32
2025-07-18T20:06:48.029283Z INFO reth_node_events::node: Finished stage pipeline_stages=2/15 stage=Headers checkpoint=22948307 target=0 stage_progress=100.00%
2025-07-18T20:06:48.063968Z DEBUG ChainOrchestrator::poll: provider::static_file: Commit segment=Headers path="/mnt/extra/reth_data/static_files/static_file_headers_22500000_22999999" duration=34.787111ms
2025-07-18T20:06:48.071075Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=6.798389ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 9, audit: 0, write: 0, sync: 426, ending: 0, whole: 436, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:48.071195Z INFO downloaders::bodies: Downloading bodies count=32 range=22948276..=22948307
2025-07-18T20:06:48.071205Z INFO reth_node_events::node: Preparing stage pipeline_stages=3/15 stage=Bodies checkpoint=22948275 target=22948307
2025-07-18T20:06:48.453786Z DEBUG ChainOrchestrator::poll: sync::stages::bodies: Commencing sync stage_progress=22948276 target=22948307
2025-07-18T20:06:48.453791Z INFO reth_node_events::node: Executing stage pipeline_stages=3/15 stage=Bodies checkpoint=22948275 target=22948307
2025-07-18T20:06:48.476689Z INFO reth_node_events::node: Finished stage pipeline_stages=3/15 stage=Bodies checkpoint=22948307 target=22948307 stage_progress=100.00%
2025-07-18T20:06:48.513216Z DEBUG ChainOrchestrator::poll: provider::static_file: Commit segment=Transactions path="/mnt/extra/reth_data/static_files/static_file_transactions_22500000_22999999" duration=36.625945ms
2025-07-18T20:06:48.520144Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=6.613187ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 7, audit: 0, write: 0, sync: 417, ending: 0, whole: 426, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:48.520292Z INFO ChainOrchestrator::poll: sync::stages::sender_recovery: Recovering senders tx_range=2896624002..2896630409
2025-07-18T20:06:48.520326Z INFO reth_node_events::node: Preparing stage pipeline_stages=4/15 stage=SenderRecovery checkpoint=22948275 target=22948307
2025-07-18T20:06:48.520360Z INFO reth_node_events::node: Executing stage pipeline_stages=4/15 stage=SenderRecovery checkpoint=22948275 target=22948307
2025-07-18T20:06:48.520360Z DEBUG ChainOrchestrator::poll: sync::stages::sender_recovery: Sending batch for processing tx_range=2896624002..2896630409
2025-07-18T20:06:48.520426Z DEBUG ChainOrchestrator::poll: sync::stages::sender_recovery: Appending recovered senders to the database tx_range=2896624002..2896630409
2025-07-18T20:06:48.595677Z DEBUG ChainOrchestrator::poll: sync::stages::sender_recovery: Finished recovering senders batch tx_range=2896624002..2896630409
2025-07-18T20:06:48.595763Z INFO reth_node_events::node: Finished stage pipeline_stages=4/15 stage=SenderRecovery checkpoint=22948307 target=22948307 stage_progress=99.99%
2025-07-18T20:06:48.599775Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=4.049975ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 4, audit: 0, write: 0, sync: 259, ending: 0, whole: 264, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:48.599833Z INFO reth_node_events::node: Preparing stage pipeline_stages=5/15 stage=Execution checkpoint=22948275 target=22948307
2025-07-18T20:06:48.599915Z DEBUG ChainOrchestrator::poll: sync::stages::execution: Calculating gas used from headers range=22948276..=22948307
2025-07-18T20:06:48.599931Z INFO reth_node_events::node: Executing stage pipeline_stages=5/15 stage=Execution checkpoint=22948275 target=22948307
2025-07-18T20:06:48.599936Z DEBUG ChainOrchestrator::poll: sync::stages::execution: Finished calculating gas used from headers range=22948276..=22948307 duration=16.409µs
2025-07-18T20:06:48.599938Z DEBUG ChainOrchestrator::poll: sync::stages::execution: Executing range start=22948276 end=22948307
2025-07-18T20:06:49.920008Z DEBUG ChainOrchestrator::poll: sync::stages::execution: Finished executing block range start=22948276 end=22948307 throughput="454.87 Mgas/second"
2025-07-18T20:06:50.151456Z DEBUG ChainOrchestrator::poll: sync::stages::execution: Execution time block_fetch=4.864922ms execution=1.314426758s write_preparation=546.166µs write=231.431813ms
2025-07-18T20:06:50.153088Z INFO reth_node_events::node: Finished stage pipeline_stages=5/15 stage=Execution checkpoint=22948307 target=22948307 stage_progress=100.00%
2025-07-18T20:06:50.240670Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=87.634175ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 116, audit: 0, write: 0, sync: 5617, ending: 7, whole: 5741, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:50.240763Z INFO reth_node_events::node: Preparing stage pipeline_stages=6/15 stage=PruneSenderRecovery checkpoint=22948275 target=22948307
2025-07-18T20:06:50.240769Z INFO reth_node_events::node: Executing stage pipeline_stages=6/15 stage=PruneSenderRecovery checkpoint=22948275 target=22948307
2025-07-18T20:06:50.240781Z DEBUG ChainOrchestrator::poll: pruner: Pruner started tip_block_number=22948307
2025-07-18T20:06:50.240786Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning started segment=Headers purpose=StaticFile to_block=22948307 prune_mode=Before(22948308)
2025-07-18T20:06:50.240826Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning finished segment=Headers purpose=StaticFile to_block=22948307 prune_mode=Before(22948308) segment_output.pruned=0
2025-07-18T20:06:50.240830Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning started segment=Transactions purpose=StaticFile to_block=22948307 prune_mode=Before(22948308)
2025-07-18T20:06:50.240842Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning finished segment=Transactions purpose=StaticFile to_block=22948307 prune_mode=Before(22948308) segment_output.pruned=0
2025-07-18T20:06:50.240844Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning started segment=Receipts purpose=StaticFile to_block=22499999 prune_mode=Before(22500000)
2025-07-18T20:06:50.240848Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning finished segment=Receipts purpose=StaticFile to_block=22499999 prune_mode=Before(22500000) segment_output.pruned=0
2025-07-18T20:06:50.240850Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning started segment=SenderRecovery purpose=User to_block=22948307 prune_mode=Full
2025-07-18T20:06:50.242211Z DEBUG ChainOrchestrator::poll: pruner: Segment pruning finished segment=SenderRecovery purpose=User to_block=22948307 prune_mode=Full segment_output.pruned=6407
2025-07-18T20:06:50.242214Z DEBUG ChainOrchestrator::poll: pruner: Pruner finished tip_block_number=22948307 elapsed=1.4294ms deleted_entries=6407 limiter=PruneLimiter { deleted_entries_limit: Some(PruneDeletedEntriesLimit { limit: 1000000, deleted: 6407 }), time_limit: None } output=PrunerOutput { progress: Finished, segments: [(Headers, SegmentOutput { progress: HasMoreData(Unknown), pruned: 0, checkpoint: Some(SegmentOutputCheckpoint { block_number: None, tx_number: None }) }), (Transactions, SegmentOutput { progress: Finished, pruned: 0, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(22948307), tx_number: Some(2896630408) }) }), (Receipts, SegmentOutput { progress: Finished, pruned: 0, checkpoint: None }), (SenderRecovery, SegmentOutput { progress: Finished, pruned: 6407, checkpoint: Some(SegmentOutputCheckpoint { block_number: Some(22948307), tx_number: Some(2896630408) }) })] } stats=[PrunedSegmentInfo { segment: SenderRecovery, pruned: 6407, progress: Finished }]
2025-07-18T20:06:50.242352Z INFO reth_node_events::node: Finished stage pipeline_stages=6/15 stage=PruneSenderRecovery checkpoint=22948307 target=22948307
2025-07-18T20:06:50.247536Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=5.284194ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 1, audit: 0, write: 0, sync: 336, ending: 0, whole: 338, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:50.247573Z INFO ChainOrchestrator::poll: sync::stages::merkle::unwind: Stage is always skipped
2025-07-18T20:06:50.247670Z INFO reth_node_events::node: Preparing stage pipeline_stages=7/15 stage=MerkleUnwind checkpoint=22948275 target=22948307
2025-07-18T20:06:50.247678Z INFO reth_node_events::node: Executing stage pipeline_stages=7/15 stage=MerkleUnwind checkpoint=22948275 target=22948307
2025-07-18T20:06:50.247697Z INFO reth_node_events::node: Finished stage pipeline_stages=7/15 stage=MerkleUnwind checkpoint=22948307 target=22948307
2025-07-18T20:06:50.250508Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=2.89944ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 1, audit: 0, write: 0, sync: 181, ending: 0, whole: 183, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:50.250548Z INFO reth_node_events::node: Preparing stage pipeline_stages=8/15 stage=AccountHashing checkpoint=22945786 target=22948307
2025-07-18T20:06:50.250557Z INFO reth_node_events::node: Executing stage pipeline_stages=8/15 stage=AccountHashing checkpoint=22945786 target=22948307
2025-07-18T20:06:50.700366Z DEBUG engine::tree: received new engine message msg=Request(NewPayload(parent: 0x01099f0784c365d5d0123b3f75830f7806f68370ccbea69dba86ae184fe0c38d, number: 22948371, hash: 0x54af1298400a694a01ab17a44be33e80fbc489783b3b784a33214a6ca8b1cca6))
2025-07-18T20:06:50.703939Z DEBUG reth::cli: Event: Handler(BlockReceived(NumHash { number: 22948371, hash: 0x54af1298400a694a01ab17a44be33e80fbc489783b3b784a33214a6ca8b1cca6 }))
2025-07-18T20:06:50.703966Z INFO reth_node_events::node: Received block from consensus engine number=22948371 hash=0x54af1298400a694a01ab17a44be33e80fbc489783b3b784a33214a6ca8b1cca6
2025-07-18T20:06:52.405655Z DEBUG engine::tree: received new engine message msg=Request(ForkchoiceUpdated { state: ForkchoiceState { head_block_hash: 0x54af1298400a694a01ab17a44be33e80fbc489783b3b784a33214a6ca8b1cca6, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, has_payload_attributes: false })
2025-07-18T20:06:52.405734Z DEBUG reth::cli: Event: Handler(ForkchoiceUpdated(ForkchoiceState { head_block_hash: 0x54af1298400a694a01ab17a44be33e80fbc489783b3b784a33214a6ca8b1cca6, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, Syncing))
2025-07-18T20:06:53.503655Z INFO reth_node_events::node: Finished stage pipeline_stages=8/15 stage=AccountHashing checkpoint=22948307 target=22948307 stage_progress=100.00%
2025-07-18T20:06:54.103473Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=599.844779ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 298, audit: 0, write: 0, sync: 39004, ending: 6, whole: 39309, gc_cputime: 68, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:54.103594Z INFO reth_node_events::node: Preparing stage pipeline_stages=9/15 stage=StorageHashing checkpoint=22945786 target=22948307
2025-07-18T20:06:54.103614Z INFO reth_node_events::node: Executing stage pipeline_stages=9/15 stage=StorageHashing checkpoint=22945786 target=22948307
2025-07-18T20:06:56.391378Z INFO reth::cli: Status connected_peers=2 stage=StorageHashing checkpoint=22945786 target=22948307
2025-07-18T20:06:58.907190Z INFO reth_node_events::node: Finished stage pipeline_stages=9/15 stage=StorageHashing checkpoint=22948307 target=22948307 stage_progress=100.00%
2025-07-18T20:06:59.650576Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=743.48284ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 519, audit: 0, write: 0, sync: 48177, ending: 15, whole: 48713, gc_cputime: 236, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:06:59.650664Z DEBUG ChainOrchestrator::poll: sync::stages::merkle::exec: Updating trie in chunks current=22945786 target=22948307
2025-07-18T20:06:59.650668Z DEBUG ChainOrchestrator::poll: sync::stages::merkle::exec: Processing chunk current=22945786 target=22948307 incremental_threshold=7000 chunk_range=22945787..=22948307
2025-07-18T20:06:59.650671Z DEBUG ChainOrchestrator::poll: trie::loader: incremental state root range=22945787..=22948307
2025-07-18T20:06:59.650711Z INFO reth_node_events::node: Preparing stage pipeline_stages=10/15 stage=MerkleExecute checkpoint=22945786 target=22948307
2025-07-18T20:06:59.650719Z INFO reth_node_events::node: Executing stage pipeline_stages=10/15 stage=MerkleExecute checkpoint=22945786 target=22948307
2025-07-18T20:07:00.934007Z DEBUG engine::tree: received new engine message msg=Request(NewPayload(parent: 0x54af1298400a694a01ab17a44be33e80fbc489783b3b784a33214a6ca8b1cca6, number: 22948372, hash: 0xe49fd943a22bb35e17e1dd8a10c6503396a2ba07430978e2b1cc26c5a46cf104))
2025-07-18T20:07:00.934761Z DEBUG reth::cli: Event: Handler(BlockReceived(NumHash { number: 22948372, hash: 0xe49fd943a22bb35e17e1dd8a10c6503396a2ba07430978e2b1cc26c5a46cf104 }))
2025-07-18T20:07:00.934787Z INFO reth_node_events::node: Received block from consensus engine number=22948372 hash=0xe49fd943a22bb35e17e1dd8a10c6503396a2ba07430978e2b1cc26c5a46cf104
2025-07-18T20:07:01.017105Z DEBUG engine::tree: received new engine message msg=Request(ForkchoiceUpdated { state: ForkchoiceState { head_block_hash: 0xe49fd943a22bb35e17e1dd8a10c6503396a2ba07430978e2b1cc26c5a46cf104, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, has_payload_attributes: false })
2025-07-18T20:07:01.017133Z DEBUG reth::cli: Event: Handler(ForkchoiceUpdated(ForkchoiceState { head_block_hash: 0xe49fd943a22bb35e17e1dd8a10c6503396a2ba07430978e2b1cc26c5a46cf104, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, Syncing))
2025-07-18T20:07:08.595645Z DEBUG net: fork id mismatch, removing peer peer_id=0x9ea69b55dd5d35cf6689268f6d58493a5ddcfd51705653e4c62abafb120c139cf31bbd3605e9c526b9b6a65552cafc507e3cd89f121bcd046d88201e5e389c5f remote_fork_id=ForkId { hash: ForkHash("be46d57c"), next: 0 } our_fork_id=ForkId { hash: ForkHash("c376cf8b"), next: 0 }
2025-07-18T20:07:08.678395Z DEBUG net: fork id mismatch, removing peer peer_id=0xcc42211164c2f41a0fb60ca9180b62a23bdf713018e0d8dc42aaef774f5a674928ee42abc2c8b6b359ae727e6186aca5b39c855a0abbdcad8b184539d6848900 remote_fork_id=ForkId { hash: ForkHash("2f095d4a"), next: 0 } our_fork_id=ForkId { hash: ForkHash("c376cf8b"), next: 0 }
2025-07-18T20:07:11.819924Z DEBUG net: fork id mismatch, removing peer peer_id=0xe0b3b6cf83fec50cc0a80d0d79e43e1444f262e365312d20f1d0a11b1b3b9c9031f54a930da38e051899b65c20720db3042240447a663eecdbca55659f811057 remote_fork_id=ForkId { hash: ForkHash("879d6e30"), next: 0 } our_fork_id=ForkId { hash: ForkHash("c376cf8b"), next: 0 }
2025-07-18T20:07:12.490505Z DEBUG net: Session established remote_addr=40.160.13.217:30303 client_version=r peer_id=0x3db469f727c9cdca42d0b63922caabd21c683735629d053f0b16cd4d9ae4d6887e7ea2cb4b18b6238479c13d21998ae0e86487eed5a7ce2e30648c6e6aa03547 total_active=3 kind=outgoing peer_enode=enode://3db469f727c9cdca42d0b63922caabd21c683735629d053f0b16cd4d9ae4d6887e7ea2cb4b18b6238479c13d21998ae0e86487eed5a7ce2e30648c6e6aa03547@40.160.13.217:30303
2025-07-18T20:07:12.490552Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x3db469f727c9cdca42d0b63922caabd21c683735629d053f0b16cd4d9ae4d6887e7ea2cb4b18b6238479c13d21998ae0e86487eed5a7ce2e30648c6e6aa03547
2025-07-18T20:07:14.470834Z DEBUG engine::tree: received new engine message msg=Request(NewPayload(parent: 0xe49fd943a22bb35e17e1dd8a10c6503396a2ba07430978e2b1cc26c5a46cf104, number: 22948373, hash: 0x31dca443fbe3bbd055ff7e5e78b38fc4eb9764c7cdccbcfc85065aacacc6be06))
2025-07-18T20:07:14.471384Z DEBUG reth::cli: Event: Handler(BlockReceived(NumHash { number: 22948373, hash: 0x31dca443fbe3bbd055ff7e5e78b38fc4eb9764c7cdccbcfc85065aacacc6be06 }))
2025-07-18T20:07:14.471417Z INFO reth_node_events::node: Received block from consensus engine number=22948373 hash=0x31dca443fbe3bbd055ff7e5e78b38fc4eb9764c7cdccbcfc85065aacacc6be06
2025-07-18T20:07:14.557531Z DEBUG engine::tree: received new engine message msg=Request(ForkchoiceUpdated { state: ForkchoiceState { head_block_hash: 0x31dca443fbe3bbd055ff7e5e78b38fc4eb9764c7cdccbcfc85065aacacc6be06, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, has_payload_attributes: false })
2025-07-18T20:07:14.557560Z DEBUG reth::cli: Event: Handler(ForkchoiceUpdated(ForkchoiceState { head_block_hash: 0x31dca443fbe3bbd055ff7e5e78b38fc4eb9764c7cdccbcfc85065aacacc6be06, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, Syncing))
2025-07-18T20:07:18.091050Z DEBUG net: Session established remote_addr=40.160.16.181:30303 client_version=r peer_id=0xb6083ecc457da689ece677a3d66380f6db378c22429a7bcb231519ba388f2fb3b90244c39783ce0a358cb6286db1a8093428313540b3d20d928f67d4bfb25c05 total_active=3 kind=outgoing peer_enode=enode://b6083ecc457da689ece677a3d66380f6db378c22429a7bcb231519ba388f2fb3b90244c39783ce0a358cb6286db1a8093428313540b3d20d928f67d4bfb25c05@40.160.16.181:30303
2025-07-18T20:07:18.097455Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0xb6083ecc457da689ece677a3d66380f6db378c22429a7bcb231519ba388f2fb3b90244c39783ce0a358cb6286db1a8093428313540b3d20d928f67d4bfb25c05
2025-07-18T20:07:18.709679Z DEBUG net: Session established remote_addr=57.129.36.159:30303 client_version=reth/v1.3.12-VERGEN_/x86_64-unknown-linux-gnu peer_id=0x7b312df67bf742364de2756f583f9834497a5ae9afdc8bd4351e99fbbdc569ebd0953a02f54e3c69635419c0da9f9feb26e9fff5436c9733e0e7879a60e1f7d2 total_active=3 kind=outgoing peer_enode=enode://7b312df67bf742364de2756f583f9834497a5ae9afdc8bd4351e99fbbdc569ebd0953a02f54e3c69635419c0da9f9feb26e9fff5436c9733e0e7879a60e1f7d2@57.129.36.159:30303
2025-07-18T20:07:18.709717Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x7b312df67bf742364de2756f583f9834497a5ae9afdc8bd4351e99fbbdc569ebd0953a02f54e3c69635419c0da9f9feb26e9fff5436c9733e0e7879a60e1f7d2
2025-07-18T20:07:21.390720Z INFO reth::cli: Status connected_peers=2 stage=MerkleExecute checkpoint=22945786 target=22948307
2025-07-18T20:07:23.775602Z DEBUG net: Session established remote_addr=57.128.92.156:30303 client_version=r peer_id=0x8bff820cc77ac38b35ac1076d35f86b80aba255f0bdf4b4174464ec1382235b57f7d792f409d184028df37dd824e4ee9892367f44b52a706990056d1a8c4e332 total_active=3 kind=outgoing peer_enode=enode://8bff820cc77ac38b35ac1076d35f86b80aba255f0bdf4b4174464ec1382235b57f7d792f409d184028df37dd824e4ee9892367f44b52a706990056d1a8c4e332@57.128.92.156:30303
2025-07-18T20:07:23.778027Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x8bff820cc77ac38b35ac1076d35f86b80aba255f0bdf4b4174464ec1382235b57f7d792f409d184028df37dd824e4ee9892367f44b52a706990056d1a8c4e332
2025-07-18T20:07:24.134463Z DEBUG net: Session established remote_addr=94.72.165.178:30304 client_version=Geth/v1.15.10-stable-2bf8a789/windows-amd64/go1.24.2 peer_id=0x72d6a302a92edccd641781621e5853c0523ec31c76ac0532f92039f603a1c3dd7cd4874e6f19b3ed32f8292f57bcfb070ff4121ba804552ec281e5a171612e49 total_active=3 kind=outgoing peer_enode=enode://72d6a302a92edccd641781621e5853c0523ec31c76ac0532f92039f603a1c3dd7cd4874e6f19b3ed32f8292f57bcfb070ff4121ba804552ec281e5a171612e49@94.72.165.178:30304
2025-07-18T20:07:24.929971Z DEBUG net: Session established remote_addr=172.99.190.92:30303 client_version=Geth/v1.15.11-stable-36b2371c/linux-amd64/go1.24.2 peer_id=0x509f4265ab31f6ba99ecdf687ab3be9ed6121379fe81a2a325ad094dc4cf39e008ffc6b56d26cf18c65a9dd64aec18c086359fe5b4e986bb7033ab25d43a7d0f total_active=4 kind=outgoing peer_enode=enode://509f4265ab31f6ba99ecdf687ab3be9ed6121379fe81a2a325ad094dc4cf39e008ffc6b56d26cf18c65a9dd64aec18c086359fe5b4e986bb7033ab25d43a7d0f@172.99.190.92:30303
2025-07-18T20:07:25.273253Z DEBUG engine::tree: received new engine message msg=Request(NewPayload(parent: 0x31dca443fbe3bbd055ff7e5e78b38fc4eb9764c7cdccbcfc85065aacacc6be06, number: 22948374, hash: 0x9d45186e1588e649d09728d6b73d17443be8a32bbfbe2c1a226eb92070337953))
2025-07-18T20:07:25.275261Z DEBUG reth::cli: Event: Handler(BlockReceived(NumHash { number: 22948374, hash: 0x9d45186e1588e649d09728d6b73d17443be8a32bbfbe2c1a226eb92070337953 }))
2025-07-18T20:07:25.275293Z INFO reth_node_events::node: Received block from consensus engine number=22948374 hash=0x9d45186e1588e649d09728d6b73d17443be8a32bbfbe2c1a226eb92070337953
2025-07-18T20:07:25.362142Z DEBUG engine::tree: received new engine message msg=Request(ForkchoiceUpdated { state: ForkchoiceState { head_block_hash: 0x9d45186e1588e649d09728d6b73d17443be8a32bbfbe2c1a226eb92070337953, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, has_payload_attributes: false })
2025-07-18T20:07:25.362171Z DEBUG reth::cli: Event: Handler(ForkchoiceUpdated(ForkchoiceState { head_block_hash: 0x9d45186e1588e649d09728d6b73d17443be8a32bbfbe2c1a226eb92070337953, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, Syncing))
2025-07-18T20:07:25.737015Z DEBUG net: Session established remote_addr=91.134.56.2:30303 client_version=r peer_id=0x4233381e93f3d28bbba880b86321786063d1760047f23c50c9e061508d16540e687e5033d661fe4ea59086b237e5873bbdb512b8e7af3ef56df0e1ae079d792b total_active=5 kind=outgoing peer_enode=enode://4233381e93f3d28bbba880b86321786063d1760047f23c50c9e061508d16540e687e5033d661fe4ea59086b237e5873bbdb512b8e7af3ef56df0e1ae079d792b@91.134.56.2:30303
2025-07-18T20:07:25.737066Z DEBUG net::session: failed to receive message err=disconnected remote_peer_id=0x4233381e93f3d28bbba880b86321786063d1760047f23c50c9e061508d16540e687e5033d661fe4ea59086b237e5873bbdb512b8e7af3ef56df0e1ae079d792b
2025-07-18T20:07:29.474975Z ERROR ChainOrchestrator::poll: sync::stages::merkle: Failed to verify block state root!
Invalid state root error on stage verification!
This is an error that likely requires a report to the reth team with additional information.
Please include the following information in your report:
* This error message
* The state root of the block that was rejected
* The output of `reth db stats --checksum` from the database that was being used. This will take a long time to run!
* 50-100 lines of logs before and after the first occurrence of the log message with the state root of the block that was rejected.
* The debug logs from __the same time period__. To find the default location for these logs, run:
`reth --help | grep -A 4 'log.file.directory'`
Once you have this information, please submit a github issue at https://github.com/paradigmxyz/reth/issues/new
target_block=22948307 got=0xfc3f6ddd4f05b1d0988cdc393c615c83be73e2c41c09b8c50225b7679f2f0fd1 expected=SealedHeader { hash: OnceLock(0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe), header: Header { parent_hash: 0x61ef6b171121c8c798b5a462c30b638614f6dd69a2df92aa1e9fcecfd2702b23, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4838b106fce9647bdf1e7877bf73ce8b0bad5f97, state_root: 0x8f6f48e6d6abb8d3da67ddd96229f1d59a4271640d133ff68721b4c37bfee08a, transactions_root: 0x71eb4421db39baf7890646b2186e0d7f22965c4cb0fbff01bb9e350b48553c88, receipts_root: 0x8657f8287c16226bc889554dae8acbee9d78cf3ee29493d2c0c767721c49deda, logs_bloom: 0x11e0e86325603a36b01644fdb2095605930359f35d8d4a94d6c1005a26269cb008063d5cfb0d1b8e0bd35ec041c3053c273da2691c1ae9295791665b013464add41af10d08c5ba385c8237eab8c0f8f5102698b02bde002950033f52896c4e12140d478d227d0676a2a69b007d8e6c1bda2938d03dd54c66a01e6db6611e02460831031812b3e19dc9c5b441dbb41e47b5419d61b55017286ca42ae2ecfd928a0bca28203a1828821a7a53c21b84279b7a166cee0730240d766ba531d40fa971ef86ec6abcd22b3325799b98217bb83409a5b24a44c01cfa6dd1854ea0336a7f23bcacd58fa571200445b8b428315cd465ebc039b15324c8384f91b362397184, difficulty: 0, number: 22948307, gas_limit: 36238245, gas_used: 12343773, timestamp: 1752868439, extra_data: 0x546974616e2028746974616e6275696c6465722e78797a29, mix_hash: 0x90a8645ea5ab520f478a2138307ca31cffc6ec006c86f170aa8bdb1545f729fc, nonce: 0x0000000000000000, base_fee_per_gas: Some(3026936074), withdrawals_root: Some(0x7debcad385fee3c515c5ae6aa69f2c435ce83d52681f564811b2b11ba9499779), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x5e09e30a28b3bd1c4e5011b9949edd142aa11f6299612843565b4404a5d39373), requests_hash: Some(0x13d5fac6f914c719561417abc83ec8cc926186ca19a1eade8b6f999f10f82849) } }
2025-07-18T20:07:29.475252Z ERROR ChainOrchestrator::poll: sync::pipeline: Stage encountered a validation error: mismatched block state root: got 0xfc3f6ddd4f05b1d0988cdc393c615c83be73e2c41c09b8c50225b7679f2f0fd1, expected 0x8f6f48e6d6abb8d3da67ddd96229f1d59a4271640d133ff68721b4c37bfee08a stage=MerkleExecute bad_block=22948307
2025-07-18T20:07:29.475306Z DEBUG ChainOrchestrator::poll: storage::db::mdbx: Commit total_duration=7.914µs commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 0, audit: 0, write: 0, sync: 0, ending: 0, whole: 1, gc_cputime: 0, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:07:29.475347Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mFinish}: sync::pipeline: Starting unwind from=22945786 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:29.475357Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mPrune}: sync::pipeline: Starting unwind from=22945786 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:29.475363Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mIndexAccountHistory}: sync::pipeline: Starting unwind from=22945786 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:29.475368Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mIndexStorageHistory}: sync::pipeline: Starting unwind from=22945786 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:29.475375Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mTransactionLookup}: sync::pipeline: Starting unwind from=22945786 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:29.475380Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mMerkleExecute}: sync::pipeline: Starting unwind from=22945786 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:29.475386Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mStorageHashing}: sync::pipeline: Starting unwind from=22948307 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:33.223388Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mStorageHashing}: sync::pipeline: Stage unwound stage=StorageHashing unwind_to=22945786 progress=22945786 done=true
2025-07-18T20:07:35.282639Z DEBUG ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mStorageHashing}: storage::db::mdbx: Commit total_duration=2.059082508s commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 386, audit: 0, write: 0, sync: 134534, ending: 14, whole: 134935, gc_cputime: 248, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:07:35.282837Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mAccountHashing}: sync::pipeline: Starting unwind from=22948307 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:37.053160Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mAccountHashing}: sync::pipeline: Stage unwound stage=AccountHashing unwind_to=22945786 progress=22945786 done=true
2025-07-18T20:07:37.301760Z DEBUG engine::tree: received new engine message msg=Request(NewPayload(parent: 0x9d45186e1588e649d09728d6b73d17443be8a32bbfbe2c1a226eb92070337953, number: 22948375, hash: 0xe5f497c6de7b19cc4806a38adfff8d827363aa816894682e57127cb44d81e7c5))
2025-07-18T20:07:37.308941Z DEBUG reth::cli: Event: Handler(BlockReceived(NumHash { number: 22948375, hash: 0xe5f497c6de7b19cc4806a38adfff8d827363aa816894682e57127cb44d81e7c5 }))
2025-07-18T20:07:37.308980Z INFO reth_node_events::node: Received block from consensus engine number=22948375 hash=0xe5f497c6de7b19cc4806a38adfff8d827363aa816894682e57127cb44d81e7c5
2025-07-18T20:07:37.404409Z DEBUG engine::tree: received new engine message msg=Request(ForkchoiceUpdated { state: ForkchoiceState { head_block_hash: 0xe5f497c6de7b19cc4806a38adfff8d827363aa816894682e57127cb44d81e7c5, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, has_payload_attributes: false })
2025-07-18T20:07:37.404447Z DEBUG reth::cli: Event: Handler(ForkchoiceUpdated(ForkchoiceState { head_block_hash: 0xe5f497c6de7b19cc4806a38adfff8d827363aa816894682e57127cb44d81e7c5, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, Syncing))
2025-07-18T20:07:37.790865Z DEBUG ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mAccountHashing}: storage::db::mdbx: Commit total_duration=737.653017ms commit_latency=Some(CommitLatency(MDBX_commit_latency { preparation: 0, gc_wallclock: 403, audit: 0, write: 0, sync: 47930, ending: 6, whole: 48340, gc_cputime: 190, gc_prof: MDBX_commit_latency__bindgen_ty_1 { wloops: 0, coalescences: 0, wipes: 0, flushes: 0, kicks: 0, work_counter: 0, work_rtime_monotonic: 0, work_xtime_cpu: 0, work_rsteps: 0, work_xpages: 0, work_majflt: 0, self_counter: 0, self_rtime_monotonic: 0, self_xtime_cpu: 0, self_rsteps: 0, self_xpages: 0, self_majflt: 0, pnl_merge_work: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 }, pnl_merge_self: MDBX_commit_latency__bindgen_ty_1__bindgen_ty_1 { time: 0, volume: 0, calls: 0 } } })) is_read_only=false
2025-07-18T20:07:37.790931Z INFO ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mMerkleUnwind}: sync::pipeline: Starting unwind from=22948307 to=22945786 bad_block=Some(22948307)
2025-07-18T20:07:37.790940Z DEBUG ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mMerkleUnwind}: trie::loader: incremental state root range=22945787..=22948307
2025-07-18T20:07:40.026676Z DEBUG net: Session established remote_addr=192.69.209.178:30303 client_version=Geth/v1.15.10-stable-2bf8a789/linux-amd64/go1.24.2 peer_id=0xd25f3134339832d79472230a735fe2e6a973b8755211063e1d0eb3cecc113d5bbcfda1b028a35807d81902aec4981740c68e14796b6b73ee700baaf68454a418 total_active=5 kind=outgoing peer_enode=enode://d25f3134339832d79472230a735fe2e6a973b8755211063e1d0eb3cecc113d5bbcfda1b028a35807d81902aec4981740c68e14796b6b73ee700baaf68454a418@192.69.209.178:30303
2025-07-18T20:07:42.882295Z DEBUG net: fork id mismatch, removing peer peer_id=0x05aa8071a617dc5702dada134fb85f2b0d19063b2f065630e88a27c2756b301f103e6c2546e9d649030a33e84ba861f2139cdf65cf3178c35e975a3b7cfc2df5 remote_fork_id=ForkId { hash: ForkHash("9f3d2254"), next: 0 } our_fork_id=ForkId { hash: ForkHash("c376cf8b"), next: 0 }
2025-07-18T20:07:46.391298Z INFO reth::cli: Status connected_peers=5 stage=MerkleUnwind checkpoint=22948307 target=22945786
2025-07-18T20:07:48.760141Z DEBUG engine::tree: received new engine message msg=Request(NewPayload(parent: 0xe5f497c6de7b19cc4806a38adfff8d827363aa816894682e57127cb44d81e7c5, number: 22948376, hash: 0x7ab212ca00547c75286038a24291e0a74bbe9ecb2ab7e63dbb126c2b9a510819))
2025-07-18T20:07:48.764981Z DEBUG reth::cli: Event: Handler(BlockReceived(NumHash { number: 22948376, hash: 0x7ab212ca00547c75286038a24291e0a74bbe9ecb2ab7e63dbb126c2b9a510819 }))
2025-07-18T20:07:48.765011Z INFO reth_node_events::node: Received block from consensus engine number=22948376 hash=0x7ab212ca00547c75286038a24291e0a74bbe9ecb2ab7e63dbb126c2b9a510819
2025-07-18T20:07:48.859692Z DEBUG engine::tree: received new engine message msg=Request(ForkchoiceUpdated { state: ForkchoiceState { head_block_hash: 0x7ab212ca00547c75286038a24291e0a74bbe9ecb2ab7e63dbb126c2b9a510819, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, has_payload_attributes: false })
2025-07-18T20:07:48.859783Z DEBUG reth::cli: Event: Handler(ForkchoiceUpdated(ForkchoiceState { head_block_hash: 0x7ab212ca00547c75286038a24291e0a74bbe9ecb2ab7e63dbb126c2b9a510819, safe_block_hash: 0xe587ca9ee15800237b4157c6f70327dc86a8e4c10665c6e7852d668c25668bbe, finalized_block_hash: 0x4b5c269f82218fd0c5e4345bf0b092bcf94b5d0bde6c6460dad5921bbb1891fe }, Syncing))
2025-07-18T20:07:50.564652Z DEBUG net: Session established remote_addr=103.94.169.230:30311 client_version=Geth/v1.15.11-stable-36b2371c/linux-amd64/go1.24.2 peer_id=0x44abf0b0d26ccda100818494ae061cf666be17322dcbc5223b7ccece04ad34c9600001bf237f3d361ec0195e01fcb8ffed161d35e29e43b9ce20fa50f27701ac total_active=6 kind=outgoing peer_enode=enode://44abf0b0d26ccda100818494ae061cf666be17322dcbc5223b7ccece04ad34c9600001bf237f3d361ec0195e01fcb8ffed161d35e29e43b9ce20fa50f27701ac@103.94.169.230:30311
2025-07-18T20:07:54.646616Z DEBUG net: Session established remote_addr=199.231.166.242:30307 client_version=Geth/v1.15.10-stable-2bf8a789/windows-amd64/go1.24.2 peer_id=0xf0cd6ecbba7680396fc5b42de181f25bdcbdc39f375162d574c0702835305aec96bf0b3e34aea96d033f062aab90174a3d07eff78805e87c789d699066666dce total_active=7 kind=outgoing peer_enode=enode://f0cd6ecbba7680396fc5b42de181f25bdcbdc39f375162d574c0702835305aec96bf0b3e34aea96d033f062aab90174a3d07eff78805e87c789d699066666dce@199.231.166.242:30307
2025-07-18T20:07:56.693714Z ERROR ChainOrchestrator::poll:Unwinding{�[3mstage�[0m�[2m=�[0mMerkleUnwind}: sync::stages::merkle: Failed to verify block state root!
Invalid state root error on stage verification!
This is an error that likely requires a report to the reth team with additional information.
Please include the following information in your report:
* This error message
* The state root of the block that was rejected
* The output of `reth db stats --checksum` from the database that was being used. This will take a long time to run!
* 50-100 lines of logs before and after the first occurrence of the log message with the state root of the block that was rejected.
* The debug logs from __the same time period__. To find the default location for these logs, run:
`reth --help | grep -A 4 'log.file.directory'`
Once you have this information, please submit a github issue at https://github.com/paradigmxyz/reth/issues/new
target_block=22945786 got=0x0127d90e6965d277f94f2d3a088bf44f6b46ca643279168250e458dc919d47ac expected=SealedHeader { hash: OnceLock(0xfaff73d40e8e7fed14921c5e01733d14af7fb389cb075d453fee0fbb08419e46), header: Header { parent_hash: 0xcd77d02f0b22829995cda9fae0ee5cc5e784fa3e890c932d2ea973040184d048, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0xdadb0d80178819f2319190d340ce9a924f783711, state_root: 0xb579aab3ceca5a7d906eb48a32a766b017da92a50dbd474697cb0740de78305c, transactions_root: 0xdf660ca6c3ae02a58f33f9977343cf378cc6b7ef1f8d8a50608b8b3f79fa5de6, receipts_root: 0x1f4ca579314127bf071367586b73cf7369db23701e5c31e8069fa26f05f8db47, logs_bloom: 0x91e9d6e6c0079d4776280e95c3b690a17d4ee52b04ac631b9baf930ec65cc6221c54611ab04a1aa60f14fad40c3fcb3f33afc8e0cc137ba32e11a4c225bc6086c9e4f068f29ba93b6866733f984077a466cdb06bc06c4e29985b1dcc8f018a629fdf30016e5f1262b5f4f22d099cac09da3f1d36c3b606dc5e08d9d410f9989dc8bed87f254fc0cd405d38fa47281b621e55d9c9e9ddf24aacbf6b42adb6578b2fc3636ab535ffd80f5974849ad8f517742c364d36e6d88d632f667877650f66f3675baba60a0b01419913f2213becfc2935f646ac0895303e0d256e3730267e957fbcc2c34494aa90443bf5ab9108e7ef65d071431874d088c9f5bb3e97fd18, difficulty: 0, number: 22945786, gas_limit: 35999965, gas_used: 15672764, timestamp: 1752838103, extra_data: 0x4275696c6465724e65742028466c617368626f747329, mix_hash: 0x44cd33968341a871a6bfb6d36735c91fd085cebb0f01dd574ddbaf8386d64324, nonce: 0x0000000000000000, base_fee_per_gas: Some(4292883331), withdrawals_root: Some(0x3f2159935e71cf5368929bfc34c1c8a3fe34aa6e899985350f1b956461fb14b4), blob_gas_used: Some(655360), excess_blob_gas: Some(131072), parent_beacon_block_root: Some(0x4d886e83332b5c5fdb0fa691e5b23769371890feca606394b19076d029595779), requests_hash: Some(0xe3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855) } }
2025-07-18T20:07:56.824291Z ERROR ChainOrchestrator::poll: reth_engine_tree::chain: backfill sync failed err=stage encountered an error in block #22945786: validation error: mismatched block state root: got 0x0127d90e6965d277f94f2d3a088bf44f6b46ca643279168250e458dc919d47ac, expected 0xb579aab3ceca5a7d906eb48a32a766b017da92a50dbd474697cb0740de78305c
2025-07-18T20:07:56.824300Z DEBUG reth::cli: Event: FatalError
2025-07-18T20:07:56.824303Z ERROR reth::cli: Fatal error in consensus engine
2025-07-18T20:07:56.824359Z ERROR reth::cli: shutting down due to error
Platform(s)
Linux (x86)
Container Type
Not running in a container
What version/commit are you on?
v1.5.1, prior was on 1.3.12
What database version are you on?
Current database version: 2
Local database version: 2
Which chain / network are you on?
mainnet
What type of node are you running?
Full via --full flag
What prune config do you use, if any?
No response
If you've built Reth from source, provide the full command you used
No response
Code of Conduct
- I agree to follow the Code of Conduct