universe: re-try sync if root mismatches #810
Description
Background
Related to #807.
If we're syncing the leaves of a universe root, it's possible that the root changes (for example someone else pushing new proofs) while we're still fetching leaves, which then leads to a root mismatch.
Currently we just log an error and abort the sync. On the next sync iteration we then get the new root, so it fixes itself eventually.
To avoid this issue, we should just re-fetch the root if we get a mismatch and re-try syncing leaves for that updated root.
Example from the sending_multi_asset_groups_hashmail_courier integration test (with debug log statements added which aren't in the current code):
On the universe when fetching a leaf:
2024-02-23 14:28:18.589 [INF] TADB: Universe root for key (outpoint=566f0f0c7084ccdd39cc9167092bf89e4e80161a80094024c5584f0e74e7e0c2:0, script_key=020743e20d741b8f4ac6c6d858c33b476200ae28bce661a3f3a3145f230b6c47ee) is: root_hash=4aa28f4c4403b5dc648ccc099421de7099569a25742eb348c5a3ea4f26cdd08d, root_sum=50
But the newly booted node fetched the root while there only were 49 of the 50 asset minting proofs pushed, so it has an old root it is comparing the received root to:
2024-02-23 14:28:18.593 [ERR] UNIV: Proof for universe root key (outpoint=566f0f0c7084ccdd39cc9167092bf89e4e80161a80094024c5584f0e74e7e0c2:0, script_key=020743e20d741b8f4ac6c6d858c33b476200ae28bce661a3f3a3145f230b6c47ee) is: root_hash=4aa28f4c4403b5dc648ccc099421de7099569a25742eb348c5a3ea4f26cdd08d, root_sum=50 but remote root is at root_hash=5fb1345a5dfa4f285a4b107c0028627281cba384468c83029cea619f6cad139c, root_sum=49
Which then leads to this error:
2024-02-23 14:18:16.386 [WRN] UNIV: encountered an error whilst syncing with server=(universe.ServerAddr) {
ID: (int64) 0,
addrStr: (string) (len=15) "127.0.0.1:19656",
addr: (net.Addr) <nil>
}
: proof for key=(universe.LeafKey) {
OutPoint: (wire.OutPoint) 82eda06e8e4b0e35149e833498395eed20ec632291ceaaa0ac40bd224e988c77:0,
ScriptKey: (*asset.ScriptKey)(0xc000df8370)({
PubKey: (*secp256k1.PublicKey)(0xc001c849b0)({
x: (secp256k1.FieldVal) 1b2fdb4d0e3b0a2a2638500fcdc658e3b6e359ec7795d969de60ba674051d9fe,
y: (secp256k1.FieldVal) 3112cf3c5168d3de691fc3be572df1484c7411ba65bc59ebf1c122008bc7e778
}),
TweakedScriptKey: (*asset.TweakedScriptKey)(<nil>)
})
}
is invalid