Worker shutdown leads to a terminated Durable Orchestration and possible corrupted Durable Entity state

[jukkamasikainen]

Description

While running a durable orchestration, worker shutting down led to a complete Durable Orchestration termination with only framework level errors in the stacktrace (below).

This is problematic as Durable Orchestrations are supposed to be rehydrated. In this particular case Durable Entity was also left in a corrupted state.

"message": "Function 'ExecuteWorkload' failed with an unhandled exception.",
"details": "Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.ExecuteWorkload
 ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
Exception: Cannot access a disposed object.
Object name: 'IServiceProvider'.
Stack:    at Microsoft.Extensions.DependencyInjection.ServiceLookup.ThrowHelper.ThrowObjectDisposedException()
   at Microsoft.Extensions.DependencyInjection.ServiceProvider.CreateScope()
   at Microsoft.Azure.Functions.Worker.DefaultFunctionContext.get_InstanceServices() in /_/src/DotNetWorker.Core/Context/DefaultFunctionContext.cs:line 46
   at Microsoft.Extensions.Hosting.MiddlewareWorkerApplicationBuilderExtensions.<>c__DisplayClass3_0`1.<UseMiddleware>b__1(FunctionContext context) in /_/src/DotNetWorker.Core/Hosting/WorkerMiddlewareWorkerApplicationBuilderExtensions.cs:line 105
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in /_/src/DotNetWorker.Core/FunctionsApplication.cs:line 96
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in /_/src/DotNetWorker.Grpc/Handlers/InvocationHandler.cs:line 89
   at Microsoft.Azure.WebJobs.Script.Description.WorkerFunctionInvoker.InvokeCore(Object[] parameters, FunctionInvocationContext context) in /_/src/WebJobs.Script/Description/Workers/WorkerFunctionInvoker.cs:line 101
   at Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase.Invoke(Object[] parameters) in /_/src/WebJobs.Script/Description/FunctionInvokerBase.cs:line 82
   at Microsoft.Azure.WebJobs.Script.Description.FunctionGenerator.Coerce[T](Task`1 src) in /_/src/WebJobs.Script/Description/FunctionGenerator.cs:line 225
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2.InvokeAsync(Object instance, Object[] arguments) in D:\\a\\_work\\1\\s\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs:line 53
   at Microsoft.Azure.WebJobs.Extensions.DurableTask.OutOfProcMiddleware.<>c__DisplayClass10_0.<<CallOrchestratorAsync>b__0>d.MoveNext() in /_/src/WebJobs.Extensions.DurableTask/OutOfProcMiddleware.cs:line 135
--- End of stack trace from previous location ---
   at Microsoft.Azure.WebJobs.Host.Executors.TriggeredFunctionExecutor`1.<>c__DisplayClass7_0.<<TryExecuteAsync>b__0>d.MoveNext() in D:\\a\\_work\\1\\s\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\TriggeredFunctionExecutor.cs:line 51
--- End of stack trace from previous location ---
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.InvokeWithTimeoutAsync(IFunctionInvoker invoker, ParameterHelper parameterHelper, CancellationTokenSource timeoutTokenSource, CancellationTokenSource functionCancellationTokenSource, Boolean throwOnTimeout, TimeSpan timerInterval, IFunctionInstance instance) in D:\\a\\_work\\1\\s\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 581
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithWatchersAsync(IFunctionInstanceEx instance, ParameterHelper parameterHelper, ILogger logger, CancellationTokenSource functionCancellationTokenSource) in D:\\a\\_work\\1\\s\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 527
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\\a\\_work\\1\\s\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 306
   --- End of inner exception stack trace ---
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.ExecuteWithLoggingAsync(IFunctionInstanceEx instance, FunctionStartedMessage message, FunctionInstanceLogEntry instanceLogEntry, ParameterHelper parameterHelper, ILogger logger, CancellationToken cancellationToken) in D:\\a\\_work\\1\\s\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 352
   at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor.TryExecuteAsync(IFunctionInstance functionInstance, CancellationToken cancellationToken) in D:\\a\\_work\\1\\s\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs:line 108"

Expected behavior

Durable orchestrations are rehydrated through worker shutdowns / restarts.

Actual behavior

Worker died (due to SIGTERM?), and durable orchestration was never rehydrated.

Known workarounds

No workarounds.

App Details

• Durable Functions extension version (e.g. v1.8.3): 1.8.0
• Azure Functions runtime version (1.0 or 2.0): 2.0
• Programming language used: C#

Screenshots

If deployed to Azure

Deployed to Azure but cannot divulge information.

[Ilia-Kosenkov]

Just to add a bit of a context, we have an orchestration that looks like this:

try
{
   await SomeOrchestrations+Activities();
}
finally
{
   await CallEntity();
}

The exception seems to have surfaced either during entity construction or during transition between suborchestration -> orchestration. The final result was that finally block never got executed (or it was executed but due to the abovementioned error entity did not receive the signal), which left the system in the corrupt state.

[jukkamasikainen]

This specific instance leading to an error was actually done with try-catch only:

try
{
   await SomeOrchestrations+Activities();

   await FinalizeOrchestration();
}
catch (Exception e)
{
   await FinalizeOrchestration();

   LogError(e, "...");

   throw;
}

In our error case, the FinalizeOrchestration was never run, as the above exception was thrown inside SomeOrchestrations+Activities part.