PubSub Mechanism - Follow up on #9994

[metricaez]

This PR evolves from #9994

The redesign addresses prior feedback by eliminating the forced inclusion of all data and publisher roots, moving subscription logic from the relay chain to the parachain (enabling selective data access), and introducing reusable, generic APIs (KeyToIncludeInRelayProofApi) that allow runtimes to request additional proofs to be included in the relay chain state beyond pub-sub use cases. Data is then extracted from the verified proof within the inherent execution flow.

The resulting pattern consists of published data passing via verified proof delivery followed by parachain-side extraction.

Therefore this PR implements a publish mechanism for parachains to efficiently share data through relay chain storage, addressing issue #606 and following up on feedback obtained from the initial design proposal in #9994.

Context:

The original issue identified expensive and complex inter-parachain information exchange. Current methods (XCM messages, off-chain protocols) are inefficient when data needs to be disseminated across many parachains. 

• Parachains publish (key, value) data to relay chain via XCM instruction
• Data stored in separate child tries per publisher
• Subscribing parachains access data through collator-fetched relay chain state proofs
• Efficient change detection using child trie root comparisons

Implementation Flow based on #9994 feedback

1. Publishing: Parachains publish data via XCM Publish instruction to relay chain broadcaster pallet
2. Proof Request: Collator retrieves storage keys via KeyToIncludeInRelayProofApi for subscribed data
3. Validation: parachain-system validates relay chain state proof and passes to processor via ProcessRelayProofKeys
4. Extraction: Processor pallet extracts data from verified proof using child trie root change detection
5. Consumption: Runtime pallets receive updates through SubscriptionHandler::on_data_updated()

The design provides generic APIs (relay proof requests, proof processing) reusable beyond pub-sub, with handler-based patterns for flexible runtime integration.

Architecture Overview

Parachain A (Publisher)
    ↓ pallet-xcm send → Publish XCM instruction
Relay Chain XCM Executor
    ↓ BroadcastHandler::handle_publish()
Broadcaster Pallet (Relay Chain)
    ↓ Stores in child trie per publisher
    ↓ Updates child trie root
Relay Chain Storage

Parachain B (Subscriber)
    ↓ Implements SubscriptionHandler trait
    ↓ Returns subscriptions via keys_to_prove() API
Collator
    ↓ Requests child trie proofs for subscriptions
    ↓ Includes proofs in relay chain state proof
Parachain System (cumulus-pallet-parachain-system)
    ↓ Verifies relay chain state proof in set_validation_data
    ↓ Calls ProcessRelayProofKeys::process_relay_proof_keys()
Subscriber Pallet (Parachain)
    ↓ Reads child trie roots from verified proof
    ↓ Compares roots for change detection
    ↓ Extracts updated key-value pairs from child tries
    ↓ Calls SubscriptionHandler::on_data_updated()

Components Implemented

1. XCM v5 Publish Instruction

Location: polkadot/xcm/src/v5/mod.rs



Publish { data: PublishData } 


Allows parachains to publish bounded key-value data to the relay chain.

Type: PublishData = BoundedVec<(PublishKey, BoundedVec<u8, MaxPublishValueLength>), MaxPublishItems>

• PublishKey: Fixed 32-byte hash ([u8; 32])
• Current Limits: Max 16 items, 32-byte keys, 1024-byte values per operation

XCM Error Addition: Added PublishFailed error to xcm::v5::Error and pallet_xcm::errors::ExecutionError. 


XCM v4 Compatibility: The instruction is not supported in XCM v4. Downgrades from v5 to v4 return an error for Publish instructions. 



Note: The instruction is being back ported to xcm v5. 



The instruction is intended to be called via pallet-xcm send with proper fee payment instructions.

2. Broadcaster Pallet (Relay Chain)


Location: polkadot/runtime/parachains/src/broadcaster/

V1 of the core relay chain pallet managing published data with registration-based access control. 



Registration System:

• System parachains (ID < 2000): Use force_register_publisher(manager, deposit, para_id) (Root origin) with custom deposit amounts (expected zero)
• Public parachains (ID ≥ 2000): Use register_publisher(para_id) requiring PublisherDeposit from caller

Deposits held via HoldReason::PublisherDeposit using fungible traits

Storage Architecture:

• Child trie per publisher: ChildInfo::new_default((b"pubsub", para_id).encode())
• Keys: Fixed 32-byte hashes
• Values: Bounded by MaxValueLength (1024 bytes)
• Total storage limit: MaxTotalStorageSize per publisher (sum of all 32-byte keys + value lengths)

Storage Maps:

• RegisteredPublishers: Publisher info (manager account, deposit amount)
• PublisherExists: Tracks active publishers with child tries
• PublishedKeys: Enumerates all keys per publisher (bounded by MaxStoredKeys)
• TotalStorageSize: Current storage usage per publisher in bytes

Extrinsics:

• register_publisher(para_id): Register with standard deposit
• force_register_publisher(manager, deposit, para_id): Root registration with custom deposit
• cleanup_published_data(para_id): Remove all key-value pairs (manager only, required before deregister)
• deregister_publisher(para_id): Release deposit after cleanup (manager only)
• force_deregister_publisher(para_id): Root-only immediate cleanup and deposit release

Integrity Checks: Runtime integrity tests ensure MaxPublishItems and MaxValueLength don't exceed XCM v5 bounds. 

Trait: 

• Publish

pub trait Publish {
    fn publish_data(publisher: ParaId, data: Vec<([u8; 32], Vec<u8>)>) -> DispatchResult;
}


Integration:

• Session change integration via new OnNewSessionOutgoing trait in initializer
• If integrated, automatic cleanup of published data when parachains are offboarded.

3. Initializer Pallet Extension (Session Change Hook)

Location: polkadot/runtime/parachains/src/initializer.rs

OnNewSessionOutgoing trait:
pub trait OnNewSessionOutgoing<N> {
    fn on_new_session_outgoing(
        notification: &SessionChangeNotification<N>,
        outgoing_paras: &[polkadot_primitives::Id],
    );
}

Added new trait for handling session changes when parachains are being offboarded. This enables operations or automatic cleanup of parachain-specific state during session transitions. 

Config Extension:
type OnNewSessionOutgoing: OnNewSessionOutgoing<BlockNumberFor<Self>>;

The initializer pallet now calls OnNewSessionOutgoing::on_new_session_outgoing(&notification, &outgoing_paras) during session changes, passing the list of offboarded parachains. 

Intended Broadcaster Integration: The broadcaster pallet implements OnNewSessionOutgoing to automatically cleanup published data when parachains are offboarded, preventing orphaned data on the relay chain.

4. BroadcastHandler Trait & Adapter

Location: polkadot/xcm/xcm-executor/src/traits/broadcast_handler.rs, polkadot/xcm/xcm-builder/src/broadcast_adapter.rs 

BroadcastHandler trait:
pub trait BroadcastHandler {
    fn handle_publish(origin: &Location, data: PublishData) -> XcmResult;
}

ParachainBroadcastAdapter:

• Validates XCM origin against configurable filter
• Extracts ParaId from XCM Location
• Bridges XCM executor to broadcaster pallet
• Provides OnlyParachains filter for direct parachain origins
  XCM Executor Integration:
• Added type BroadcastHandler: BroadcastHandler to xcm_executor::Config
• Default () implementation (no-op)
• Executor processes Publish instruction by calling configured handler
• Weight refunding for unused execution
  XCM Benchmarking: Added Publish instruction benchmarking to pallet-xcm-benchmarks::genericas weight consumption for each runtime will depend on handler implementation.

5. Relay Proof Request System

Location: cumulus/primitives/core/src/lib.rs

RelayProofRequest:
pub struct RelayProofRequest {
    pub keys: Vec<RelayStorageKey>,
}

pub enum RelayStorageKey {
    Top(Vec<u8>),
    Child { storage_key: Vec<u8>, key: Vec<u8> },
}

KeyToIncludeInRelayProofApi:
fn keys_to_prove() -> RelayProofRequest;

Allows parachains to request both top-level and child trie storage proofs. Collators generate proofs for requested keys and include them in ParachainInherentData::relay_chain_state. 

Dedicated PR: #10678

6. Collator Integration

Location: cumulus/client/parachain-inherent/src/lib.rs

Collators call keys_to_prove() runtime API to retrieve RelayProofRequest from the parachain runtime.

The collect_additional_storage_proofs function processes both RelayStorageKey::Top and RelayStorageKey::Childentries, generating proofs via RelayChainInterface. 

All proofs are included in the relay chain state proof passed to the parachain runtime through ParachainInherentData. 

Relay Chain Interface Extension:

• RelayChainInterface extended with child trie proof support
• Implemented in RelayChainInProcessInterface
• Implemented in RelayChainRpcInterface

Dedicated PR: #10678

7. Parachain System Integration

Location: cumulus/pallets/parachain-system/src/ 

Config Extension:

type RelayProofKeysProcessor: ProcessRelayProofKeys;
ProcessRelayProofKeys trait (relay_state_snapshot.rs):
pub trait ProcessRelayProofKeys {
    fn process_relay_proof_keys(verified_proof: &RelayChainStateProof) -> Weight;
}

Integration Point: During set_validation_data, after verifying the relay chain state proof, the pallet calls RelayProofKeysProcessor::process_relay_proof_keys(&relay_state_proof).
The returned weight is accumulated into the inherent's total weight. 
This is the critical hook that enables pallet-subscriber and other pallets to securely access relay chain data within the verified context.

Dedicated PR: #10678

8. Subscriber Pallet (Parachain)

Location: cumulus/pallets/subscriber/ 

Generic pallet for processing child trie data from relay chain proofs.

Implements ProcessRelayProofKeys to integrate with parachain-system. 

Defines SubscriptionHandler trait:

pub trait SubscriptionHandler {
    fn subscriptions() -> (Vec<(ParaId, Vec<Vec<u8>>)>, Weight);
    fn on_data_updated(publisher: ParaId, key: Vec<u8>, value: Vec<u8>) -> Weight;
}

Runtime implements this trait to define subscription logic and data processing. 

Integration Flow:

• get_relay_proof_requests() transforms subscriptions into child trie proof requests for keys_to_prove() API
• ProcessRelayProofKeys::process_relay_proof_keys() reads child trie roots from verified proof
• Compares roots with PreviousPublishedDataRoots storage (change detection)
• Extracts updated key-value pairs from child tries when roots differ
• Calls SubscriptionHandler::on_data_updated() for each changed key

Storage:

• PreviousPublishedDataRoots: Maps ParaId to child trie root hash (32 bytes), bounded by MaxPublishers

Extrinsics:

• clear_stored_roots(publisher): Root-only call to force reprocessing in next block (recovery scenarios)

Change Detection: Only publishers with changed child trie roots trigger data extraction and handler calls, significantly reducing storage writes and computation.

Provides no_std bench_proof_builder for benchmarking.

Testing

Rococo Integration (Testing Purposes)
Relay - Location: polkadot/runtime/rococo/src/
Parachain - Location: cumulus/parachains/runtimes/testing/rococo-parachain/ (RIP)

Handlers and API's have been setup plus a simple pubsubConsumer pallet has been provided to demo a potential integration with Subscriber

A full demo integration of config can be found at:
https://github.com/blockdeep/polkadot-sdk/tree/feat/pubsub-rev1225-dev

Rationale: Enables reviewers to test the complete flow using Zombienet (config provided in pubsub-dev/zombienet.toml).

It is recommended to run this test on the testing branch as it has both tooling and setups and it can be used as reference on any other integration. It also showcases benchmarking setup.

Local Testing with Zombienet

A Zombienet configuration is provided in pubsub-dev/ for local testing:
cd pubsub-dev
./build.sh # Build polkadot and polkadot-parachain
zombienet spawn zombienet.toml

This spins up:

• Rococo relay chain (4 validators)
• Rococo parachain (2 collators)

Extrinsics:

• [Relay] Fund Parachain's Sovereign Account: 0x04030070617261e80300000000000000000000000000000000000000000000000000000b00407a10f35a

• [Relay] Force register Parachain 1000: 0xff004101d43593c715fdd31c61141abd04a99fd6822c8558854ccde39a5684e7a56da27d00000000000000000000000000000000e8030000

• [Parachain] Publish some Data via pallet-xcm send call: 0x02003300050100050c000400000002286bee1300000002286bee00340800000000000000000000000000000000000000000000000000000000000000010812340000000000000000000000000000000000000000000000000000000000000002084321

Closure

Please share any concerns, suggestions, or alternative approaches.

Related original issue