feat(msp): MSP retries responding storage requests

[snowmead]

MSPs retry storage requests in the event when they fail due to proof invalidations i.e. ForestProofVerificationFailed, KeyProofVerificationFailed, FailedToApplyDelta. Under these circumstances, the MSP will re-queue the storage requests to be responded to again.

Design

BatchProcessStorageRequests is a new event handler which queries the pending storage requests which are waiting for the MSP's response. It will cause; a NewStorageRequest event to be emitted via a new command (PreprocessStorageRequest) to initiate the natural storage request process which receives the file from the user and adds the file to storage and queue the storage request for response.

The file keys that are processed and responded to are tracked via a shared HashMap in a single MspUploadFileTask instance. This single instance is critical for the retry mechanism to work correctly, as all event handlers (BatchProcessStorageRequests, NewStorageRequest, RemoteUploadRequest, ProcessMspRespondStoringRequest) must share the same file_key_statuses HashMap to see status updates from each other.

A new FileKeyStatus enum tracks each file key's processing state across concurrent event handlers:

• Processing - File key is in the pipeline
• Accepted - Successfully accepted on-chain
• Rejected - Explicitly rejected on-chain
• Abandoned - Failed with non-proof dispatch errors (permanent failures, won't be retried)

Retry Mechanism: When proof errors occur (ForestProofVerificationFailed, FailedToApplyDelta), file keys are removed from the file_key_statuses HashMap rather than being marked with a Failed status. This signals to the next BatchProcessStorageRequests cycle that the file key should be re-processed. The file key will be re-inserted with Processing status, triggering a new NewStorageRequest event and regenerating proofs with the updated forest root.

The batch processing cycle is executed atomically controlled via a semaphore (size 1). The blockchain service will emit this event at every block if the semaphore isn't held. This allows more linear control over the process of when the MSP will preprocess storage requests and queue them up for responding.

The MspHandler tracks storage requests which are queued in an in-memory FIFO queue (pending_respond_storage_requests) with O(1) deduplication via a HashSet (pending_respond_storage_request_file_keys) to prevent duplicate queueing of the same file key.

Storage Deprecation

Deprecates Column Families from RocksDb storage:

• pending_msp_respond_storage_request
• pending_msp_respond_storage_request_left_index
• pending_msp_respond_storage_request_right_index

Will be removed once we implement proper migration automations for RockDb.

Notable changes

• Adds a new ModuleError associated type to the StorageEnableRuntime to enable type safety when reading decoding extrinsic errors into specific pallet errors
• New integration test which verifies the MSP can retry failed storage request acceptance based on invalid proof errors explicitly
• Use batchStorageRequests helper api for integration tests that create multiple storage requests
• Fixes a bug with conditional checks in batchStorageRequests helper api and adds logging for observability sake
• Add .mcp.json to .gitignore

Simplified Dataflow

flowchart TD
    A[BatchProcessStorageRequests<br/><i>periodic, from BlockchainService</i>] --> B[NewStorageRequest<br/><i>file 1</i>]
    A --> C[NewStorageRequest<br/><i>file 2</i>]
    A --> D[NewStorageRequest<br/><i>file N</i>]
    B --> E{File in storage?}
    C --> E
    D --> E
    E -->|No| F[RemoteUploadRequest<br/><i>chunk uploads from user</i>]
    E -->|Yes| G[on_file_complete]
    F -->|file complete| G
    G --> H[queue_msp_respond_storage_request<br/><i>in-memory FIFO queue</i>]
    H --> I[ProcessMspRespondStoringRequest<br/><i>batched extrinsic submission</i>]
    I -->|Proof Error| J[Remove from file_key_statuses<br/><i>triggers retry on next cycle</i>]
    I -->|Success| K[Accepted/Rejected status]
    I -->|Non-proof Error| L[Abandoned status<br/><i>permanent skip</i>]

Loading