Attempted to process a blockchain with missing state root

[totoCZ]

1.30.1

Dec 19 07:13:05 systemd-nethermind[2527546]: 19 Dec 07:13:05 | Processing loop threw an exception. Block: 21434999 (0x79d895...3bbf3a), Exception: System.InvalidOperationException: Attempted to process a blockchain with missing state root 0x90ef04e2f7e3f1b623e524ce6598fce71ee2f1736b3457ef7080929f01b2c010
Dec 19 07:13:05 systemd-nethermind[2527546]:    at Nethermind.Consensus.Processing.BlockchainProcessor.RunProcessingLoop() in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 319
Dec 19 07:13:05 systemd-nethermind[2527546]:    at Nethermind.Consensus.Processing.BlockchainProcessor.Process(Block suggestedBlock, ProcessingOptions options, IBlockTracer tracer, String& error) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 393
Dec 19 07:13:05 systemd-nethermind[2527546]:    at Nethermind.Consensus.Processing.BlockchainProcessor.PrepareBlocksToProcess(Block suggestedBlock, ProcessingOptions options, ProcessingBranch processingBranch) in /src/Nethermind/Nethermind.Consensus/Processing/BlockchainProcessor.cs:line 588
Dec 19 07:13:05 systemd-nethermind[2527546]: 19 Dec 07:13:05 | Processing loop threw an exception. Block: 21434998 (0x10eae8...164985), Exception: System.InvalidOperationException: Attempted to process a blockchain with missing state root 0x90ef04e2f7e3f1b623e524ce6598fce71ee2f1736b3457ef7080929f01b2c010
Dec 19 07:13:05 systemd-nethermind[2527546]: 19 Dec 07:13:05 |  Block throughput       14.82 MGas/s  |      149.9 tps |          0.72 Blk/s | recover     0 | process     6
Dec 19 07:13:05 systemd-nethermind[2527546]: 19 Dec 07:13:05 |  Block mev 0.0459 ETH   20.56 MGas    |      208   txs |  calls  5,030 (107) | sload  16,678 | sstore  6,478 | create   1(-512)
Dec 19 07:13:05 systemd-nethermind[2527546]: 19 Dec 07:13:05 | Processed            21434995         |    1,387.5 ms  |  slot    100,395 ms |<E2> Gas gwei: 10.03 .. 10.91 (12.52) .. 138.79
Dec 19 07:13:03 systemd-nethermind[2527546]: 19 Dec 07:13:03 | Non consecutive block commit. This is likely a reorg. Last block commit: 21434994. New block commit: 21434991.
Dec 19 07:13:03 systemd-nethermind[2527546]: 19 Dec 07:13:03 | Rerunning block after reorg or pruning: 21434994 (0x6a6265...173894)
Dec 19 07:13:03 systemd-nethermind[2527546]: 19 Dec 07:13:03 | Rerunning block after reorg or pruning: 21434993 (0xcfe019...dadb63)
Dec 19 07:13:03 systemd-nethermind[2527546]: 19 Dec 07:13:03 | Rerunning block after reorg or pruning: 21434992 (0xc726c0...fe6ae2)
Dec 19 07:13:03 systemd-nethermind[2527546]: 19 Dec 07:13:03 | Rerunning block after reorg or pruning: 21434991 (0x7f9514...b66733)
Dec 19 07:13:02 systemd-nethermind[2527546]: 19 Dec 07:13:02 | Received ForkChoice: 21435001 (0xd232ac...967aa2), Safe: 21434967 (0x7e42ff...c8a39a), Finalized: 21434935 (0xd72350...d86cbf)

[MarekM25]

@totoCZ, did you get it right after updating and restarting with the new version, or had you been running the node for some time?

[MarekM25]

Unfortunately, the solution is resync. However, it would be great if you share more details

[totoCZ]

It was a fresh sync on that version with grandine as the CL. Restarting ended up replaying the blocks to the same crash.

And it ran for about ~24 hr with no issues.

[MarekM25]

Thanks a lot for the context; this will help us solve the problem. Sorry for the inconvenience.

[totoCZ]

full logs
https://filetransfer.io/data-package/sG8yXrez#link

[MarekM25]

Thx a lot!