Persistent State Storage for Improved State Syncing #4
Description
Currently, during the state syncing process from the network, our node retains its internal state, including downloaded portions of the blockchain state and the target block, in memory. This poses a significant challenge: if the node restarts mid-sync, all the data must be re-downloaded.
To improve this process, I propose persistently storing the internal state outside of memory. This could be in a database or a plain file, allowing us to preserve the internal state across node restarts. This would be especially beneficial when handling a large blockchain state, which doesn't need to remain in memory.
However, one potential issue to consider is if the node is offline for an extended period, the state of the block may no longer be available on full nodes within the network, but only on archive nodes. To mitigate this, we could consider the implementation suggested in Issue paritytech/polkadot-sdk#523.