[v2.0] Consolidate module state management patterns

[thomhurst]

Summary

Module state is tracked through multiple overlapping mechanisms, making it unclear which source of truth to use and potentially allowing inconsistent states.

Current State Management

Location: src/ModularPipelines/Engine/Execution/

Multiple state tracking paths:

1. ModuleExecutionContext - Tracks execution metadata
2. ModuleState - Holds individual module status
3. ModuleResult - Final result wrapper
4. SubModuleTracker - Tracks sub-operations

Problems

1. Multiple sources of truth: State can be checked in different places
2. Unclear ownership: Which component owns state transitions?
3. Mutation patterns: State changes happen through multiple channels
4. Race condition risk: Concurrent access during error scenarios
5. Testing difficulty: Hard to mock/verify state transitions

Current Flow (Simplified)

Module Starts → ModuleExecutionContext updated
                ↓
            ModuleState set to Running
                ↓
            Execution happens
                ↓
            ModuleResult created
                ↓
            ModuleState set to Success/Failed
                ↓
            SubModuleTracker updated (if sub-modules exist)

Proposed Solutions

Option A: Immutable State Machine

public abstract record ModuleExecutionState
{
    public sealed record Pending : ModuleExecutionState;
    public sealed record Running(DateTime StartedAt) : ModuleExecutionState;
    public sealed record Completed(DateTime StartedAt, DateTime EndedAt, object? Result) : ModuleExecutionState;
    public sealed record Failed(DateTime StartedAt, DateTime EndedAt, Exception Error) : ModuleExecutionState;
    public sealed record Skipped(string Reason) : ModuleExecutionState;
}

Benefits:

• Clear state transitions
• Immutable = thread-safe
• Pattern matching for handling
• Compile-time exhaustiveness checks

Option B: Single State Owner

Consolidate all state into ModuleExecutionContext:

• Remove ModuleState as separate concept
• ModuleResult becomes read-only projection
• Clear single point of mutation

Option C: Event Sourcing (Lightweight)

public interface IModuleStateEvents
{
    void Emit(ModuleStateEvent @event);
    IReadOnlyList<ModuleStateEvent> GetHistory(IModule module);
}

public abstract record ModuleStateEvent(DateTime Timestamp);
public record ModuleStarted(DateTime Timestamp) : ModuleStateEvent(Timestamp);
public record ModuleCompleted(DateTime Timestamp, object? Result) : ModuleStateEvent(Timestamp);

Recommended Approach

Option A (Immutable State Machine) provides the best balance of:

• Type safety
• Thread safety
• Explicit transitions
• Easy testing

Impact

• Breaking change: Internal refactoring, may affect extensions
• Improves: Reliability, testability, debugging

Labels

• v2.0
• architecture
• enhancement