Fail in-flight invocations when worker channel shuts down

[satvu]

Issue describing the changes in this PR

resolves #10936

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

• Backporting to the in-proc branch is not required
  • Otherwise: Link to backporting PR
• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• My changes do not require diagnostic events changes
  • Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
• I have added all required tests (Unit tests, E2E tests)

Additional information

Additional PR information

[satvu]

Prior to this change, calling TryFailExecutions (replaced by Shutdown) would not fail executions without an exception being passed in. This test makes sure that in-flight invocations are failed whether or not an exception is passed in.

[Copilot]

Pull Request Overview

This PR modifies the worker channel shutdown behavior to better handle in-flight invocations when a worker channel encounters a fatal error. The key change is replacing the TryFailExecutions method with a more descriptive Shutdown method that properly fails in-flight invocations with a specific exception type.

Key changes:

• Replace TryFailExecutions method with Shutdown method across all worker channel implementations
• Introduce WorkerShutdownException to provide more specific error information for failed invocations
• Update HTTP proxy service to properly handle worker shutdown scenarios during request retries

Reviewed Changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/WebJobs.Script/Workers/Rpc/IRpcWorkerChannel.cs	Updates interface to replace TryFailExecutions with Shutdown method
src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs	Implements new Shutdown method with WorkerShutdownException
src/WebJobs.Script/Exceptions/WorkerShutdownException.cs	Adds new exception type for worker shutdown scenarios
src/WebJobs.Script/Http/RetryProxyHandler.cs	Updates HTTP retry logic to handle worker shutdown exceptions
src/WebJobs.Script/Http/ScriptInvocationRequestTransformer.cs	Adds transformer to pass script invocation context through HTTP proxy
src/WebJobs.Script/Http/DefaultHttpProxyService.cs	Integrates new transformer and adds invocation context to HTTP requests

[kshyju]

If it's not FunctionTimeoutException, what exception the caller is sending to this method?

[jviau]

Yeah this seems odd, the method is OnTimeoutExceptionAsync, but we are now checking if it was called for a non-timeout? Additionally, the log here just assumes that if timeoutException is null, then the channel must be shutting down? Will that always be true? How future proof is that assumption?

[satvu]

I've only reproduced this twice, but timeoutException has ended up null in this method - still digging into why that is, but from my observation of the two events is as described in the comment. Let me continue to investigate this.

[jviau]

My main concern is the potential breaking change by switching to a WorkerShutdownException. But maybe my assumption this exception makes it to extensions is wrong?

[jviau]

Yeah this seems odd, the method is OnTimeoutExceptionAsync, but we are now checking if it was called for a non-timeout? Additionally, the log here just assumes that if timeoutException is null, then the channel must be shutting down? Will that always be true? How future proof is that assumption?

[satvu]

The new approach:

To avoid a breaking change with Durable, we will introduce an exception that derives from the existing FunctionTimeoutException, and looks like this: FunctionAbortException : FunctionTimeoutException. This will only be used when the exception passed in is null or FunctionTimeoutException to preserve the current behavior for all other logs passed in (in case other components depend on any other type exceptions this method sets results to).

In these cases, in the Shutdown method, we create a new FunctionAbortException to fail executions. For the FunctionTimeoutException cases, we additionally pass in the original FunctionTimeoutException as an inner exception (?).

Since durable already has handling for FunctineTimeoutException in some places, this will ensure that the resulting behavior is as desired by the extension team. This also covers other extensions that perform any specific behavior based on this long-existing exception.

Edit: The new exception type is also being added to the WebJobs SDK (Azure/azure-webjobs-sdk#3139) so extensions can handle this type of failure with more granularity.