[Workflows] Parallel.ForEachAsync not compatible

[olitomlinson]

Expected Behavior

You could use Parallel.ForEachAsync to successfully execute 1 or many tasks. Tasks which represent calls to CallActivityAsync and WaitForExternalEventAsync for example.

Actual Behavior

The tasks (Activities) are scheduled and completed, but the results of those Tasks are never handled gracefully in the Workflow.

Steps to Reproduce the Problem

using Microsoft.Extensions.Logging;
using System.Threading.Tasks;
using Dapr.Workflow;
using DurableTask.Core.Exceptions;
using Activities;
using Models;

internal sealed partial class MyWorkflow : Workflow<string, string>
{
    public override async Task<string> RunAsync(WorkflowContext context, string input)
    {
        var activity1 = context.CallActivityAsync<string>(nameof(Act), "activity 1");
        var activity2 = context.CallActivityAsync<string>(nameof(Act), "activity 2");
        var activity3 = context.CallActivityAsync<string>(nameof(Act), "activity 3");
        var tasks = new List<Task<string>>() { activity1, activity2, activity3 };

        ParallelOptions options = new()
        {
            MaxDegreeOfParallelism = 1
        };

        var output = "start";

        await Parallel.ForEachAsync(tasks, options, async (task, token) =>
        {
            var result = await task;
            output += result;
        });

        return output;
    }
}


internal sealed partial class Act : WorkflowActivity<string, string>
{
    public override Task<string> RunAsync(WorkflowActivityContext context, string input)
    {
        return Task.FromResult(input);
    }
}

• This should produce an output which looks like "startactivity 1activity 2activity 3" however it only produces start

• logs from the workflow show that the tasks are completed

Validating config and starting app "order-processor"
ℹ️  Started Dapr with app id "order-processor". HTTP Port: 3500. gRPC Port: 51636
ℹ️  Writing log files to directory : /Users/olivertomlinson/Desktop/dev/quickstarts/workflows/csharp/sdk/order-processor/.dapr/logs
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       WorkflowLoggingService started
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       List of registered workflows
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       OrderProcessingWorkflow
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       MyWorkflow
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       List of registered activities:
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       NotifyActivity
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       VerifyInventoryActivity
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       RequestApprovalActivity
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       ProcessPaymentActivity
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       UpdateInventoryActivity
== APP - order-processor == info: Dapr.Workflow.WorkflowLoggingService[0]
== APP - order-processor ==       Act
== APP - order-processor == info: Dapr.DurableTask[1]
== APP - order-processor ==       Durable Task gRPC worker starting and connecting to localhost:51636.
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
== APP - order-processor ==       Start processing HTTP request POST http://localhost:51636/TaskHubSidecarService/GetWorkItems
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[100]
== APP - order-processor ==       Sending HTTP request POST http://localhost:51636/TaskHubSidecarService/GetWorkItems
== APP - order-processor == info: Dapr.DurableTask[4]
== APP - order-processor ==       Sidecar work-item streaming connection established.
== APP - order-processor == info: Microsoft.Hosting.Lifetime[0]
== APP - order-processor ==       Application started. Press Ctrl+C to shut down.
== APP - order-processor == info: Microsoft.Hosting.Lifetime[0]
== APP - order-processor ==       Hosting environment: Production
== APP - order-processor == info: Microsoft.Hosting.Lifetime[0]
== APP - order-processor ==       Content root path: /Users/olivertomlinson/Desktop/dev/quickstarts/workflows/csharp/sdk/order-processor
== APP - order-processor == info: Dapr.DurableTask.Client.Grpc.GrpcDurableTaskClient[40]
== APP - order-processor ==       Scheduling new MyWorkflow orchestration with instance ID 'f18aa0e0' and 26 bytes of input data.
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
== APP - order-processor ==       Start processing HTTP request POST http://localhost:51636/TaskHubSidecarService/StartInstance
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[100]
== APP - order-processor ==       Sending HTTP request POST http://localhost:51636/TaskHubSidecarService/StartInstance
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[101]
== APP - order-processor ==       Received HTTP response headers after 3104.4159ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
== APP - order-processor ==       End processing HTTP request after 3116.4297ms - 200
== APP - order-processor == info: Dapr.DurableTask.Worker.Orchestrations[600]
== APP - order-processor ==       'MyWorkflow' orchestration with ID 'f18aa0e0' started.
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
== APP - order-processor ==       Start processing HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteOrchestratorTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[100]
== APP - order-processor ==       Sending HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteOrchestratorTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[101]
== APP - order-processor ==       Received HTTP response headers after 3.2045ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
== APP - order-processor ==       End processing HTTP request after 3.2716ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[101]
== APP - order-processor ==       Received HTTP response headers after 3133.3687ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
== APP - order-processor ==       End processing HTTP request after 3133.4222ms - 200
== APP - order-processor == Workflow : f18aa0e0
== APP - order-processor == info: Dapr.DurableTask.Client.Grpc.GrpcDurableTaskClient[43]
== APP - order-processor ==       Waiting for instance 'f18aa0e0' to complete, fail, or terminate.
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
== APP - order-processor ==       Start processing HTTP request POST http://localhost:51636/TaskHubSidecarService/WaitForInstanceCompletion
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[100]
== APP - order-processor ==       Sending HTTP request POST http://localhost:51636/TaskHubSidecarService/WaitForInstanceCompletion
== APP - order-processor == info: Dapr.DurableTask.Worker.Activities[603]
== APP - order-processor ==       'Act' activity of orchestration ID 'f18aa0e0' started.
== APP - order-processor == info: Dapr.DurableTask.Worker.Activities[603]
== APP - order-processor ==       'Act' activity of orchestration ID 'f18aa0e0' started.
== APP - order-processor == info: Dapr.DurableTask.Worker.Activities[603]
== APP - order-processor ==       'Act' activity of orchestration ID 'f18aa0e0' started.
== APP - order-processor == info: Dapr.DurableTask.Worker.Activities[604]
== APP - order-processor ==       'Act' activity of orchestration ID 'f18aa0e0' completed.
== APP - order-processor == info: Dapr.DurableTask.Worker.Activities[604]
== APP - order-processor ==       'Act' activity of orchestration ID 'f18aa0e0' completed.
== APP - order-processor == info: Dapr.DurableTask.Worker.Activities[604]
== APP - order-processor ==       'Act' activity of orchestration ID 'f18aa0e0' completed.
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
== APP - order-processor ==       Start processing HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteActivityTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
== APP - order-processor ==       Start processing HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteActivityTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
== APP - order-processor ==       Start processing HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteActivityTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[100]
== APP - order-processor ==       Sending HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteActivityTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[100]
== APP - order-processor ==       Sending HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteActivityTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[100]
== APP - order-processor ==       Sending HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteActivityTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[101]
== APP - order-processor ==       Received HTTP response headers after 0.9395ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[101]
== APP - order-processor ==       Received HTTP response headers after 0.9417ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
== APP - order-processor ==       End processing HTTP request after 0.9824ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
== APP - order-processor ==       End processing HTTP request after 0.9852ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[101]
== APP - order-processor ==       Received HTTP response headers after 0.97ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
== APP - order-processor ==       End processing HTTP request after 1.0369ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[100]
== APP - order-processor ==       Start processing HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteOrchestratorTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[100]
== APP - order-processor ==       Sending HTTP request POST http://localhost:51636/TaskHubSidecarService/CompleteOrchestratorTask
== APP - order-processor == info: System.Net.Http.HttpClient.Default.ClientHandler[101]
== APP - order-processor ==       Received HTTP response headers after 0.6279ms - 200
== APP - order-processor == info: System.Net.Http.HttpClient.Default.LogicalHandler[101]
== APP - order-processor ==       End processing HTTP request after 0.7722ms - 200

• The workflow becomes stuck in the RUNNING status, and can't progress.

{
    "instanceID": "f18aa0e0",
    "workflowName": "MyWorkflow",
    "createdAt": "2025-07-06T19:33:22.859479Z",
    "lastUpdatedAt": "2025-07-06T19:33:25.991837Z",
    "runtimeStatus": "RUNNING",
    "properties": {
        "dapr.workflow.input":"\"7/6/2025 7:33:22\ PM\""
    }
}

Release Note

RELEASE NOTE:

[WhitWaldo]

@olitomlinson I've dug into this. At a high level, Parallel.ForEachAsync is not deterministic and as such is not eligible to use within a Workflow type.

Getting into more detail, when Task.WhenAll runs, taking your example, it's scheduling all the activities in the order provided and waits for each to complete, assigning them to subsequent history events:

var activity1 = context.CallActivityAsync<string>(nameof(Act), "activity 1"); // Event #1
var activity2 = context.CallActivityAsync<string>(nameof(Act), "activity 2"); // Event #2
var activity3 = context.CallActivityAsync<string>(nameof(Act), "activity 3"); // Event #3

var results = await Task.WhenAll(task1, task2, task3);

During the initial execution of the workflow then:

1. Three activities are scheduled and their events created (important that they're created here instead of when the tasks complete)
2. Activities execute in parallel
3. Three activities are completed and their completion events recorded
4. Task.WhenAll completes with results

During replay then:

1. The same three activities are scheduled
2. Each matches up with the existing history events
3. Each task completes immediately with stored results
4. Task.WhenAll completes with the same results as before.

Put another way, then using Task.WhenAll:
Orchestrator:

• CallActivityAsync() called -> record history event
• CallActivityAsync() called -> record history event
• CallActivityAsync() called -> record history event
• Task.WhenAll() called -> wait for recorded operations to complete

So turning to Parallel.ForEachAsync:

var activity1 = context.CallActivityAsync<string>(nameof(Act), "activity 1");
var activity2 = context.CallActivityAsync<string>(nameof(Act), "activity 2");
var activity3 = context.CallActivityAsync<string>(nameof(Act), "activity 3");
var tasks = new List<Task<string>>() { activity1, activity2, activity3 };

ParallelOptions options = new()
{
    MaxDegreeOfParallelism = 1
};

var output = "start";

await Parallel.ForEachAsync(tasks, options, async (task, token) =>
{
    var result = await task;
    output += result;
});

return output;

Here, the execution flow instead executes each of the async callbacks within a separate execution context. Setting the maximum degree of parallelism to 1 doesn't help here because it doesn't negate the fact that unlike Task.WhenAll each task is completing within that separate context. This ultimately leads to the orchestrator failing to track operations across these context boundaries because the individual activities are no longer deterministically assigned to each of the events in the log.

From the orchestration perspective then:
Orchestrator:

• Parallel.ForEachAsync() called -> No history events recorded
• Orchestration completes -> No operations to replay

The simple guidance here then is that Parallel.ForEachAsync is not supported within a workflow and I've updated the list of proposed analyzers at #1426 to reflect this.

In the longer term, it might be useful to provide a high-level primitive available as an extension method that allows something similar to a Parallel.ForEachAsync that this analyzer can point to as an alternative/code fix that ensures all the tasks are created in the same context and executed in parallel via Task.WhenAll with a built-in concurrency limiter (instead of leaving it to developers).

[WhitWaldo]

Following typing this up, I've introduced a high-level primitive as an extension method on WorkflowContext in #1580 that allows a similar signature as Parallel.ForEachAsync that should be used in place of Parallel.ForEachAsync.

I'm going to go ahead and close this in the meantime in hopes we can also get that analyzer put together, but given the non-deterministic nature of Parallel.ForEachAsync, I don't think it makes sense to change the framework to support this at this time.