feat: add telemetry metrics instrumentation

[snowmead]

Client Telemetry Infrastructure

Adds comprehensive Prometheus metrics instrumentation to the StorageHub client, enabling observability into storage provider operations, event handler lifecycle, file transfer throughput, and system resource utilization.

Design

The telemetry system is built around three core components:

1. StorageHubMetrics - Central metrics registry containing all Prometheus counters, histograms, and gauges
2. MetricsLink - Optional wrapper that enables zero-cost no-op when metrics are disabled
3. LifecycleMetricRecorder - Trait for tracking event handler states (pending → success/failure)

Event Handler Lifecycle Metrics

The LifecycleMetricRecorder trait is injected into event bus listeners via the subscribe_actor_event_map! macro. This enables automatic tracking of:

Metric	Description
event_handler_pending_total	Events received, handler starting
event_handler_success_total	Handler completed successfully
event_handler_failure_total	Handler failed with error
event_handler_duration_seconds	Handler execution time histogram

Usage:

subscribe_actor_event_map!(
    service: &self.blockchain,
    spawner: &self.task_spawner,
    context: self.clone(),
    metrics: self.metrics.clone(),  // Enables automatic lifecycle metrics
    critical: true,
    [
        NewStorageRequest<Runtime> => BspUploadFileTask,
        ProcessConfirmStoringRequest => BspUploadFileTask,
        // ...
    ]
);

When metrics is provided, the macro auto-generates an EventMetricRecorder for each mapping with the event label derived from the event type name (converted to snake_case).

Task-Level Metrics

Tasks record domain-specific metrics using helper macros:

inc_counter!(
    metrics: self.storage_hub_handler.metrics(),
    storage_requests_total,
    "bsp",      // provider_type
    "accept"    // status
);

observe_histogram!(
    metrics: self.storage_hub_handler.metrics(),
    file_download_seconds,
    "success",
    download_duration.as_secs_f64()
);

System Resource Metrics

A background task collects system and process resource metrics every 5 seconds using the sysinfo crate:

Metric	Description
storagehub_system_cpu_usage_percent	System-wide CPU usage (0-100%, averaged across cores)
storagehub_process_cpu_usage_percent	Process CPU usage (can exceed 100% on multi-core)
storagehub_system_memory_total_bytes	Total system memory
storagehub_system_memory_used_bytes	Used system memory
storagehub_system_memory_available_bytes	Available system memory
storagehub_process_memory_rss_bytes	Process resident set size (RSS)

The collector is spawned automatically when MetricsLink::new() is called with a registry, enabling zero-configuration resource monitoring.

Metrics Catalog

Storage Request Metrics

Metric	Labels	Description
storage_requests_total	provider_type, status	Storage request outcomes (accept/reject/fail)
storage_request_retries_total	provider_type	Proof-related retry attempts

Proof Metrics

Metric	Labels	Description
proofs_submitted_total	status	Proof submission outcomes
proof_submission_duration_seconds	status	Time to submit proofs

File Transfer Metrics

Metric	Labels	Description
bytes_downloaded_total	status	Download throughput
bytes_uploaded_total	status	Upload throughput
chunks_downloaded_total	status	Chunks downloaded
chunks_uploaded_total	status	Chunks uploaded
file_download_seconds	status	File download duration histogram

Provider Metrics

Metric	Labels	Description
slashing_events_total	role	Slashing events by provider role
bucket_move_total	status	Bucket move operations
user_evictions_total	status	Insolvent user evictions
forest_verifications_total	status	Forest verification outcomes
file_deletions_total	provider_type, status	File deletion operations

System Resource Metrics

Metric	Description
storagehub_system_cpu_usage_percent	System-wide CPU usage percentage
storagehub_process_cpu_usage_percent	Process CPU usage percentage
storagehub_system_memory_total_bytes	Total system memory in bytes
storagehub_system_memory_used_bytes	Used system memory in bytes
storagehub_system_memory_available_bytes	Available system memory in bytes
storagehub_process_memory_rss_bytes	Process RSS memory in bytes

Docker Infrastructure

Prometheus Integration

Each network type has a dedicated Prometheus configuration:

• docker/bspnet-prometheus.yml - BSPNet scrape configs
• docker/fullnet-prometheus.yml - FullNet scrape configs (BSPs, MSPs, Fisherman)

Prometheus scrapes metrics from provider containers on port 9615 (Substrate's default Prometheus port). The role label is automatically attached by Prometheus based on the scrape job configuration, enabling per-role filtering in dashboards.

Grafana Dashboards

Pre-built dashboards for each provider role:

Dashboard	Panels
bsp-dashboard.json	Storage requests, proofs, downloads, event handlers, resource usage
msp-dashboard.json	Storage requests, uploads, bucket ops, event handlers, resource usage
fisherman-dashboard.json	Challenge verifications, slashing events, resource usage

Each dashboard includes a Resource Usage section with:

• CPU Usage timeseries (process and system)
• Memory Usage timeseries (process RSS, system used/available/total)
• Process CPU gauge
• System CPU gauge
• Process Memory stat
• System Memory Usage gauge

Dashboards are automatically provisioned via docker/grafana/provisioning/.