Asset Hub Migration: Remaining Tasks for Single Instance Multi Chain Support

[TarikGul]

Asset Hub Migration: Remaining Tasks for Single Instance Multi Chain Support

Overview

Complete the remaining work for Asset Hub Migration support in substrate-api-sidecar. The goal is to ensure everything is reachable from a single instance of sidecar. Currently, everything works with the exception of staking progress when using 2 instances. These tasks address historical data access and cross-chain query capabilities needed for a seamless transition to single-instance operation.

Tasks

1. Add Historical Support for Staking Progress Endpoint (#1695)

Endpoint: /pallets/staking/progress?at={height}

Problem: Currently lacks historical support for querying staking progress at specific block heights.

Solution:

• Hardcode BABE information for epochs and sessions
• Eliminate the need for multi-chain queries by using static configuration
• Enable historical lookups without cross-chain dependencies

Acceptance Criteria:

• Endpoint supports at={height} parameter for historical queries
• BABE epoch and session data is hardcoded and accessible
• No multi-chain queries required for historical staking progress (with the exception of skippedEpochs)

2. Implement useRcBlock Query Parameter Support

Goal: Enable cross-chain state queries by allowing Asset Hub endpoints to query state at specific relay chain block heights using the useRcBlock parameter in conjunction with the existing at parameter.

Implementation:

• Add useRcBlock parameter support across all relevant endpoints
• When useRcBlock=true, treat the at parameter as a relay chain block identifier
• Ensure responses include additional metadata (rcBlockNumber, ahTimestamp)
• Maintain backward compatibility with existing functionality

Acceptance Criteria:

• useRcBlock parameter implemented across all applicable endpoints
• Responses include relay chain block number and Asset Hub timestamp metadata
• Smart block mapping handles 2:1 RC-to-AH block ratio
• Multi-block scenarios handled when Asset Hub produces multiple blocks per RC block
• Documentation updated with parameter usage and examples

3. Implement useRcBlock Query Parameter Support for Block Endpoints

Goal: Enable block endpoints to accept relay chain block identifiers instead of Asset Hub block identifiers.

Implementation:

• Add useRcBlock parameter to all block-related endpoints
• Implement relay chain to Asset Hub block mapping
• Include enhanced response metadata when parameter is used

Acceptance Criteria:

• useRcBlock parameter implemented on all block endpoints
• Block identifier mapping from relay chain to Asset Hub blocks
• Range endpoints skip RC blocks without corresponding AH blocks
• Enhanced responses include rcBlockNumber and ahTimestamp fields
• Backward compatibility maintained (defaults to false)
• OpenAPI specifications updated with new parameter

4. Cross-Chain Historical Data Handling

Problem: When endpoints require era fields or need to retrieve historical data that spans the migration boundary (Asset Hub → Relay Chain), sidecar needs intelligent handling.

Requirements:

• Determine migration start and end points
• Map relay chain blocks to corresponding parachain block numbers
• Handle historical queries that cross the migration boundary

Implementation Strategy:

• Identify the exact migration start/end blocks
• Create a mapping system between relay chain and parachain block numbers
• Implement logic to route historical queries to the appropriate chain

Acceptance Criteria:

• Migration start/end blocks are identified and stored
• Block number mapping between relay chain and parachain is implemented
• Historical queries automatically route to correct chain based on block height
• Era-based queries work seamlessly across migration boundary

5. Create Mirrored Relay Chain Endpoints

Goal: Enable a single Asset Hub sidecar instance to access both Asset Hub and Relay Chain historical data.

Implementation:

• Create mirrored endpoints that proxy relay chain requests
• Allow Asset Hub sidecar to serve historical relay chain data
• Maintain API consistency while supporting dual-chain access

Acceptance Criteria:

• Asset Hub sidecar instance can serve relay chain historical data
• Mirrored endpoints maintain consistent API structure
• Single sidecar deployment supports both chains
• Documentation updated with new endpoint capabilities

Substrate API Sidecar Endpoints

Accounts Endpoints (#1701)

• /accounts/{accountId}/asset-balances
  • useRcBlock parameter
• /accounts/{accountId}/asset-approvals
  • useRcBlock parameter
• /accounts/{accountId}/balance-info
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /accounts/{accountId}/pool-asset-balances
  • useRcBlock parameter
• /accounts/{accountId}/pool-asset-approvals
  • useRcBlock parameter
• /accounts/{accountId}/proxy-info
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /accounts/{accountId}/staking-info
  • useRcBlock parameter
• /accounts/{accountId}/staking-payouts
  • useRcBlock parameter
• /accounts/{accountId}/vesting-info
  • useRcBlock parameter

Blocks Endpoints (#1702)

• /blocks
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /blocks/{blockId}
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /blocks/{blockId}/header
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /blocks/{blockId}/extrinsics/{extrinsicIndex}
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /blocks/head
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /blocks/head/header
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /blocks/{blockId}/extrinsics-raw
  • Mirrored /rc endpoint
  • useRcBlock parameter

Node Endpoints

• /node/network
  • Mirrored /rc endpoint
• /node/transaction-pool
  • Mirrored /rc endpoint
• /node/version
  • Mirrored /rc endpoint

Runtime Endpoints

• /runtime/metadata
  • Mirrored /rc endpoint
• /runtime/metadata/{metadataVersion}
  • Mirrored /rc endpoint
• /runtime/metadata/versions
  • Mirrored /rc endpoint
• /runtime/spec
  • Mirrored /rc endpoint
• /runtime/code
  • Mirrored /rc endpoint

Pallets Endpoints

• /pallets/assets/{assetId}/asset-info
  • useRcBlock parameter
• /pallets/asset-conversion/liquidity-pools
  • useRcBlock parameter
• /pallets/asset-conversion/next-available-id
  • useRcBlock parameter
• /pallets/foreign-assets
  • useRcBlock parameter
• /pallets/nomination-pools/info
  • useRcBlock parameter
• /pallets/nomination-pools/{poolId}
  • useRcBlock parameter
• /pallets/on-going-referenda
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/consts
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/consts/{constantItemId}
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/dispatchables
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/dispatchables/{dispatchableItemId}
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/errors
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/errors/{errorItemId}
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/events
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/events/{eventItemId}
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/storage
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/{palletId}/storage/{storageItemId}
  • Mirrored /rc endpoint
  • useRcBlock parameter
• /pallets/pool-assets/{assetId}/asset-info
  • useRcBlock parameter
• /pallets/staking/progress
  • useRcBlock parameter (#1700)
• /pallets/staking/validators
  • useRcBlock parameter

[filvecchiato]

I think 1. is fair to me.

2. is a bit more complex:

• For the migration start/ends we can solve it
• the mapping is a bit harder. If we are connected to RC postAHM it can be fairly easy to check which block is the corresponding one on AH, but the other way around is hard, we don't really have (AFAIK) a way to retrieve which block in a parachain is corresponding to a block in RC. This makes it harder/expensive to find the right corresponding BN. Basically we need a binary search per query, which can be optimised with a KV cache to use as checkpoints, but still expensive IMO.

I can see two scenarios where:
a. the user indexes using RC blocknumbers (and get timestamp data there)
b. the user indexes using RC numbers only

in the first scenario, we are likely fine just using the AH blocknumbers which then contain the timestamp and data is stored using the previously used timestamp without issues to the user.

The second scenario might be an issue if they index by RC block number, since we are then asking them to start indexing by AH block number.

To make sure that a user with scenario b. is not affected, we could make we could implement a setup config that allows the user to specify which block number convention to follow (ie. connected to AH, but querying using RC block numbers - which we can convert to AH bn to make the actual query). This would probably make the life of current users affected by this easier (maybe) and make our life easier.

3. this is also a sound proposal IMO

[TarikGul]

2. is a bit more complex:

For the migration start/ends we can solve it
the mapping is a bit harder. If we are connected to RC postAHM it can be fairly easy to check which block is the corresponding one on AH, but the other way around is hard, we don't really have (AFAIK) a way to retrieve which block in a parachain is corresponding to a block in RC. This makes it harder/expensive to find the right corresponding BN. Basically we need a binary search per query, which can be optimised with a KV cache to use as checkpoints, but still expensive IMO.

So for this, to mention there is really only one small specific problem we are solving atm. That is what happens when a team is migrating sidecar from RC to AH, and they are in the phase where they are only 2 eras in on AH but they query the full history depth of 84 eras.

They need the remaining 82 eras. Once the history depth is fully on AH we technically wont be concerned with this anymore.

[filvecchiato]

So for this, to mention there is really only one small specific problem we are solving atm. That is what happens when a team is migrating sidecar from RC to AH, and they are in the phase where they are only 2 eras in on AH but they query the full history depth of 84 eras.

They need the remaining 82 eras. Once the history depth is fully on AH we technically wont be concerned with this anymore.

I see, ok so it makes it lighter, since we know the era size we can technically calculate the distance from the first era and then get the max and min blocks in RC (more or less), we still need to find where to start calculating from if its back in history, but if they are split between RC and AH we can assume it will be around migration time so we can narrow the search