StorageHub v0.2.0

Summary

StorageHub v0.2.0 focuses on scalability and resilience of MSP/BSP operations, better observability and indexing of storage requests, and SDK and backend improvements around authentication, file upload, and client UX. This includes support for multiple MSP/BSP instances (leader role), more robust handling of storage request lifecycles, a new Blockchain Service Postgres store for pending extrinsics, and SDK performance improvements for large files.

Components

• Client code: v0.2.0
• Pallets code: v0.2.0
• Runtime code: v0.2.0 (spec_name/spec_version: parachain 1, solochain-evm 1)
• SH Backend Docker image: v0.2.0 (image: ghcr.io/<org>/storage-hub-msp-backend:v0.2.0)
• SH SDK (npm): v0.3.3 (@storagehub-sdk/core, @storagehub-sdk/msp-client)
• types-bundle/api-augment (npm): @storagehub/types-bundle v0.2.7, @storagehub/api-augment v0.2.10

Changes since last tag

Base: release/v0.1 (b7e6082c0bf88579c3979a54542628ba1cc1e214)

• Highlights:
  • Indexing & observability: indexer now stores transaction hash and block hash for each new storage request, improving traceability from the chain to the database (#580).
  • Indexer API: getFiles responses now include an uploaded_at timestamp, making it easier to reason about file lifecycles (#576).
  • Multiple MSP/BSP instances (leader role): initial implementation of support for running multiple MSP/BSP instances, including leader election logic for storage request handling (#577). Follower logic is not yet implemented.
  • MSP file handling: when the MSP Backend sends a file to the MSP Node, it now streams chunks from disk instead of loading whole files into memory, significantly reducing peak memory usage on large uploads (#572).
  • MSP robustness: rejected storage requests now trigger file clean‑up where needed, MSP uses a retry strategy for storage request responses, and the MSP upload code has been refactored to avoid holding the file storage write lock, risking deadlocks (#547, #558, #582).
  • Fisherman: fisherman node can batch file deletions, improving deletion throughput (#531).
  • Blockchain Service DB: SH Client's Blockchain Service now persists pending extrinsics in a dedicated Postgres DB, improving resilience across restarts and better tracking of in‑flight transactions (#563). Has to be enabled through the pending_db_url configuration option, either via the CLI or configuration file.
  • Transactions “implicit” construction: implicit data in transactions is now built dynamically via a runtime API, allowing a client to work with a runtime build different from the one it was compiled against (#557).
  • SDK: improved ergonomics (for example optional call_scale) and faster fingerprinting for large files, alongside general SDK clean‑ups and version bumps (#551, #554, #569, #574).
  • Backend auth: backend now supports optional authentication, refactored auth expiry handling, and SIWE‑related fixes (URL/domain config and retry of verification), plus various clean‑ups (#544, #550, #552, #579).
  • Backend correctness: validate uploads against the bucket owner (not the file owner), avoid concurrent uploads for the same file key, and fill in missing backend_config.toml fields; improve logging and remove unnecessary peer‑id lookups in upload paths (#517, #568, #570, #573, #555, #586).
  • Tooling & examples: new demo app, improved local scripts (including adding fisherman node), and benchmarks for file deletion extrinsics to guide performance tuning (#516, #556, #560).
• Full diff: https://github.com/Moonsong-Labs/storage-hub/compare/b7e6082c0bf88579c3979a54542628ba1cc1e214...d9a283293a2612a1a567ab5b6848e84e4ea0a858
• PRs included:
  • #588 docs: 📝 Add missing parameters to config files
  • #587 fix: 🐛 update column order of migration in indexer DB to match schema and code
  • #586 fix(backend): 🐛 avoid concurrent uploads for the same file key
  • #582 fix: 🚑 avoid deadlocks during the MSP upload process
  • #580 feat: ✨ add tx hash for each new storage request in the indexer
  • #579 fix: SIWE config URL and domain
  • #576 feat: Add uploaded_at field to getFiles response
  • #577 feat: 🚧 Add multiple MSP/BSP instances support - Leader role implementation only
  • #575 feat(test): Add flag to initialise MSP with arbitrary size files
  • #572 feat(msp): stream chunks instead of full load in memory in save_file_to_disk
  • #574 feat: ✨ Make call_scale optional
  • #563 feat(client): ✨ Persist pending extrinsics in new Blockchain Service Postgres DB
  • #573 refactor(backend): do not get peer id in upload file
  • #570 fix(backend): 🐛 validate file upload against bucket owner instead of file owner
  • #560 benchmark: file deletion extrinsics
  • #547 feat(msp): rejected storage request triggers file clean up if needed
  • #531 feat(fisherman): 🌎 batch file deletions
  • #569 chore: 🔖 bump SDK packages version
  • #568 fix: 📝 add missing fields to the backend_config.toml file
  • #566 chore: 🔖 bump backend version after major changes
  • #562 fix: 🐛 remove either leading or trailing comma, not both
  • #557 feat: ✨ Build implicit in transactions dynamically using a runtime API
  • #558 feat: 🔊 make the MSP use a retry strategy for its storage request responses
  • #516 Demo app
  • #556 feat: ✨ Add fisherman node to script
  • #555 chore: 🔊 update logs for MSP distributing
  • #553 fix: 🐛 BSP volunteering issues
  • #552 Retry verify on SIWE
  • #544 feat(backend): optional auth
  • #550 Refactor/auth expiry

Migrations

RocksDB (File Storage)

• Changes:
  • No new schema changes.
• Action required:
  • None for fresh deployments or chains already migrated to the v0.1.x RocksDB schema.

RocksDB (Forest Storage)

• Changes:
  • No new schema changes.
• Action required:
  • None.

RocksDB (State store)

• Changes:
  • None that require migration.
• Action required:
  • None.

Indexer DB (Postgres)

• Migrations:
  • Includes 2025-11-27-182418_add_tx_hash_and_block_hash_to_file; its column order was updated in PR #587 before any release shipped, so this release only contains the corrected version.
• How to apply: The indexer service runs migrations automatically on startup. Alternatively run diesel migration run.

Pending extrinsics DB (Postgres)

• Migrations:
  • Freshly introduced. No migrations required as it didn't exist before this release.
• How to apply: The Blockchain Service runs migrations automatically on startup. Alternatively run diesel migration run.

⚠️ Breaking Changes ⚠️

• PR #517: backend configuration and HTTP API changes (new auth/expiry/callback/retry config fields; authenticated upload/download; alloy Address for addresses; file list response now rooted at a single tree field; /distribute endpoint removed; bucket table extended with value-proposition and size/count columns).
• PR #557: transaction implicit data is now built dynamically via a runtime API; clients that manually constructed implicit payloads must align with the new API.
• PR #531: fisherman now batches file deletions; any tooling assuming one-by-one deletion semantics should be reviewed.
• PR #547: MSP performs clean-up of on-disk file data when storage requests are rejected, altering behaviour for incomplete or failed storage flows.
• PR #563: new Postgres database for pending extrinsics in the Blockchain Service, controlled via pending_db_url, changes how in-flight extrinsics are persisted across restarts.
• PR #554: @storagehub-sdk/core and @storagehub-sdk/msp-client v0.3.3 introduce SDK API/typing adjustments that may require client code changes.
• PR #572: MSP file handling now streams chunks from disk instead of loading full files into memory, which can change performance characteristics and any extensions hooking into the old path.
• PR #577: groundwork for multiple MSP/BSP instances with a leader role; multi-instance deployments must follow the new leader configuration model.
• PR #576: getFiles response and related indexer models now include an uploaded_at timestamp; clients parsing file metadata must handle the new field.
• PR #579: SIWE configuration URL and domain have been corrected; deployments must update their SIWE settings to match.
• PR #580: indexer now records a tx_hash (and associated block_hash) per new storage request; any direct consumers of the indexer DB must account for these columns.
• PR #586: backend config TOML gains a new file_transfer section with max_upload_sessions and max_download_sessions, which operators must add to their config.
• PR #587: column order in migration 2025-11-27-182418_add_tx_hash_and_block_hash_to_file was updated to match schema and code; no extra operator action is required beyond running migrations for this release.

Runtime

• Upgrades (spec_version): parachain and solochain-evm remain at spec_version 1.
• Migrations: No runtime storage migrations detected.
• Constants changed: None requiring operator action.
• Scripts to run: None.

Client

• Behaviour changes:
  • Pending extrinsics durability: Blockchain Service now persists pending extrinsics in a dedicated Postgres database, improving restart resilience and introspection of transaction state. This also paves the way for multiple MSP/BSP instances running in a leader-follower configuration (#563).
  • Multi‑instance support: groundwork for multiple MSP/BSP instances with a leader role for coordinating responsibilities across instances. Follower logic is not yet implemented yet. (#577).
  • File lifecycle semantics: getFiles now reports uploaded_at and indexer file rows track whether a file is currently in a bucket and their associated tx/block hashes, improving API expressiveness and debuggability (#576, 2025-11-12, 2025-11-27 migrations).
  • MSP robustness: MSP cleans up disk state for rejected storage requests and retries storage request responses on failures, reducing orphaned data and transient errors (#547, #558).
  • Fisherman behaviour: fisherman node can batch file deletions, reducing load and latency when deleting many files at once (#531).
  • Developer tooling: improvements to logs for MSP distributing (#555).
• Initialisation changes:
  • When starting the client services, ensure the Blockchain Service Postgres DB is configured and reachable, as pending extrinsics persistence now depends on it. If no DB URL is provided, the Blockchain Service will run in standalone mode and pending extrinsics will not be persisted.

Backend

• Behaviour changes:
  • Optional authentication: backend can now be run with optional auth, easing local development and certain trusted deployments while still supporting SIWE‑style authentication in production (#544).
  • Authentication quality: SIWE verification and session expiry logic have been refactored, with retries on verification and clearer expiry handling; configuration now correctly uses the SIWE URL and domain settings (#550, #552, #579).
  • Correctness fixes:
    • File uploads are now validated against the bucket owner, not the file owner, aligning with expected authorisation semantics (#570).
    • Miscellaneous clean‑ups in upload paths (no unnecessary peer‑id fetches), avoidance of concurrent uploads for the same file key, and other backend code paths (#517, #573, #586).
  • Configuration & logging: missing fields in backend_config.toml have been filled in, and logs around MSP distributing and backend behaviour have been improved for easier debugging (#555, #568).
• Initialisation changes:
  • If you rely on backend auth, review the updated auth configuration (SIWE settings, optional auth toggles) and ensure new config fields are set in your backend_config.toml.

SDK

• Behaviour changes:

  • Performance: significantly faster fingerprinting of large files in @storagehub-sdk/core, reducing upload preparation time for big datasets (#551).
  • Ergonomics: call_scale is now optional, simplifying common call flows in the SDK (#574).
  • General improvements: additional tweaks and internal clean‑ups, plus version bumps to @storagehub-sdk/core and @storagehub-sdk/msp-client (now at v0.3.3) (#554, #569).
  • SDK auth: SDK now allows external configuration from domain and url fields from SIWE method (#579).

• Initialisation changes:

  • Ensure your applications depend on SDK >= v0.3.3 to benefit from the new features and performance characteristics.

Versions

• Polkadot SDK: polkadot-stable2412-6
• Rust: 1.87 (from rust-toolchain.toml)

Compatibility

• SH Backend v0.2.0 → compatible with pallets/runtime v0.2.0 and client v0.2.0 (all built from this release).
• SDK v0.3.3 → compatible with backend v0.2.0, client v0.2.0, and pallets/runtime v0.2.0.

Upgrade Guide

• PR #517 – backend configuration, HTTP API and indexer DB:

  • Breaking change:
    • Configuration:
      • Add configuration items for auth expirations.
      • Add configuration item for node callback URL.
      • Configurable number of retries for upload.
    • API:
      • Use alloy Address for addresses instead of strings.
      • Authenticate users for upload/download operations.
      • FileList response now has a single tree field at the root and folder items do not have a children field.
      • Remove /distribute endpoint as the backend is not in charge of this any more.
    • DB migrations:
      • The bucket table now has 3 more columns: value_prop_id, total_size, file_count.
    • API changes:
      • Added pagination query parameters to /files and /buckets endpoints (limit and page).
  • Suggested changes: extend your backend configuration with the new auth expiry, node callback URL and upload‑retry items; update all backend callers to use alloy Address, authenticate upload/download, consume the new file list tree shape, and stop calling /distribute; run the bucket migration and adjust any custom SQL for bucket; optionally adopt the new pagination parameters on /files and /buckets.

• PR #557 – runtime transaction implicits:

  • Breaking change:
    • Removed the previous implicit computation ExtensionOperations::implicit(genesis_block_hash, current_block_hash) from runtime/*/configs/storage_hub.rs.
    • Added a new TxImplicitsApi runtime API.
  • Suggested changes: remove any usage/implementation of ExtensionOperations::implicit(...) in your runtimes, and implement the TxImplicitsApi runtime API as shown in the PR.

• PR #531 – indexer/MSP DB wiring and fisherman CLI:

  • Breaking change:
    • configure_and_spawn_indexer no longer returns the DB pool:
      • Result<Option<DbPool>, sc_service::Error> → Result<(), sc_service::Error>.
    • MSP now creates its own DB pool instead of reusing the one created for the indexer service.
    • New required CLI option for running MSP nodes: --msp-database-url.
    • MSP no longer needs --indexer and --indexer-database-url flags to connect to the index database.
    • Removed fisherman CLI options:
      • --fisherman-incomplete-sync-max
      • --fisherman-incomplete-sync-page-size
      • --fisherman-sync-mode-min-blocks-behind
  • Suggested changes: update any code expecting a DB pool from configure_and_spawn_indexer; ensure MSP nodes are started with --msp-database-url and no longer pass the indexer flags; remove usage of the deprecated fisherman CLI options from your run scripts and tooling.

• PR #547 – StorageRequestRejected event:

  • Breaking change: StorageRequestRejected now includes msp_id and bucket_id:
    • StorageRequestRejected { file_key, msp_id, bucket_id, reason }.
  • Suggested changes: regenerate any event type bindings and update off‑chain consumers that decode or pattern‑match this event so they handle the new fields.

• PR #563 – client pending‑extrinsics DB:

  • Breaking change:
    • Blockchain Service now has an optional pending_db_url configuration variable to persistently track the lifecycle of transactions sent.
  • Suggested changes: add the --pending-db-url CLI flag to nodes using the SH Client, wiring it into bs_options.pending_db_url as shown in the PR, and provision the backing Postgres database if you want durable tracking of pending extrinsics.

• PR #554 – SDK type changes:

  • Breaking change:
    • Removed incorrect name field from StorageFileInfo.
      • Impact: code accessing fileInfo.name will break.
      • Migration: use fileInfo.location instead, or import FileInfo from @storagehub-sdk/core.
    • Hex string format:
      • All hex fields now consistently return 0x${string} format.
      • Impact: code expecting hex without 0x prefix may break (but this is a fix to be compatible with other tooling).
  • Suggested changes: replace uses of fileInfo.name with fileInfo.location or the correct FileInfo type, and ensure any code that strips or assumes hex prefixes is compatible with the 0x‑prefixed format.

• PR #572 – MSP internal buffer size:

  • Breaking change: new configuration field internal_buffer_size was added to the node CLI to configure the internal buffer channel and the size of the batch while reading file chunks from the DB.
  • Suggested changes: when running MSP nodes, set internal_buffer_size appropriately for your workload, and update any deployment tooling or config templates to include this new CLI option.

• PR #577 – client leader/follower groundwork:

  • Breaking change: builds on top of PR #563, adding behaviour to the breaking changes there (leader/follower behaviour for multi‑instance setups).
  • Suggested changes: when using multiple MSP/BSP instances with the client, follow the leader/follower model described in the PR and ensure your configuration matches the expectations introduced in #563 and #577.

• PR #576 – file upload dates:

  • Breaking change:
    • Backend: file lists now include upload date, returning uploadedAt (ISO string) on each file.
    • SDK: exposes uploadedAt on file entries as a Date.
  • Suggested changes: update any backend or SDK consumers that parse file list responses to handle the new uploadedAt field and adjust typings or UI as needed.

• PR #579 – SIWE URL and domain:

  • Breaking change:
    • Backend: /auth/nonce endpoint now expects url and domain params.
    • SDK: must set url and domain params for SIWE.
  • Suggested changes: make sure your front‑end/SDK SIWE flows pass url and domain to /auth/nonce as required, and that these values match your deployment’s domain and URL.

• PR #580 – transaction hash integration:

  • Breaking change:
    • Runtimes that want to integrate StorageHub must implement the TransactionHashProvider trait.
    • There is a new migration to run in the indexer DB.
    • The response from the backend and the SDK for file info now has two new fields (including transaction hash information).
  • Suggested changes: implement TransactionHashProvider in your runtimes as shown in the PR, apply the new indexer migration, and update any code reading file info from the backend/SDK to handle and, where useful, surface the new transaction‑hash‑related fields.

• PR #586 – backend file transfer limits:

  • Breaking change:
    • The backend config TOML file now has an additional file_transfer category with two new fields: max_upload_sessions and max_download_sessions.
  • Suggested changes: update your backend config file to add the file_transfer section and set appropriate values for max_upload_sessions and max_download_sessions, as shown in the PR, so the backend can limit concurrent upload/download sessions correctly.

• PR #587 – indexer migration column order:

  • Breaking change:
    • Changes the column order of migration 2025-11-27-182418_add_tx_hash_and_block_hash_to_file to match schema and code.
    • Since no currently running node had the previous version of this migration (it was unreleased), this change and the original migration ship together in this release.
  • Suggested changes: no special action is required beyond running the indexer migrations for this release; this note exists for completeness if you inspect the migration history.