Transaction duplication across operator bundles

[jfrank-summit]

Summary

There is potential for duplication of domain transactions across multiple operator bundles within the same consensus block and across consecutive blocks. While the domain client deduplicates prior to execution, the consensus chain still includes full bundle bodies, resulting in redundant transactions being gossiped, stored, validated, and archived. This increases block size, bandwidth, and storage footprint.

Current behavior

• Consensus blocks include full bundle bodies for all accepted bundles.
• Multiple operators can submit bundles for the same domain and height with overlapping extrinsics.
• The consensus runtime validates bundle headers and associated execution receipts (ERs), and records accepted bundles in ExecutionInbox via digests (header hash, extrinsics root, size), while the full bundle bodies live in the consensus block body.
• Domain nodes later extract accepted bundles and run deduplicate_and_shuffle_extrinsics() before building and executing the domain block, so execution occurs at most once even when the same transaction appears multiple times.
• Validation of bundle bodies (decoding, tx-range, XDM MMR proof checks, etc.) is performed statelessly in the domain preprocessor, not by the consensus runtime.

Impact

• Larger consensus blocks due to repeated transactions for overlapping bundles.
• Higher bandwidth usage for nodes (both network propagation and block import).
• Increased storage growth

Notes

• What is the current duplication profile across (distribution, tail behavior)?
  • Probably low given there are not many transactions.
  • We should measure this during load testing on test networks

[teor2345]

See also this forum post for background and potential solutions:
https://forum.autonomys.xyz/t/multiple-bundles-per-block-vs-one-bundle-per-block/2276

[teor2345]

Some blocks with large numbers of domain bundles seem to contain lots of duplicates of the same bundle info:
https://autonomys.subscan.io/extrinsic/4009315-189
https://autonomys.subscan.io/extrinsic/4009315-188

Is that related to this issue, or something separate?

[jfrank-summit]

That is more related to the production of empty bundles. In order to force the clearing of the challenge period empty bundles are created when there are no new extrinsics. This is another opportunity for optimization.