Skip to content

Fail in-flight invocations when worker channel shuts down #11159

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: dev
Choose a base branch
from

Conversation

satvu
Copy link
Member

@satvu satvu commented Jul 2, 2025

Issue describing the changes in this PR

resolves #10936

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

  • Backporting to the in-proc branch is not required
    • Otherwise: Link to backporting PR
  • My changes do not require documentation changes
    • Otherwise: Documentation issue linked to PR
  • My changes should not be added to the release notes for the next release
    • Otherwise: I've added my notes to release_notes.md
  • My changes do not need to be backported to a previous version
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • My changes do not require diagnostic events changes
    • Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
  • I have added all required tests (Unit tests, E2E tests)

Additional information

Additional PR information

@satvu satvu changed the title Worker channel crash Fail in-flight invocations when worker channel restarts Jul 2, 2025
@fabiocav fabiocav requested a review from Copilot July 2, 2025 23:35
Copilot

This comment was marked as outdated.

@satvu satvu changed the title Fail in-flight invocations when worker channel restarts Fail in-flight invocations when worker channel shuts down Jul 8, 2025
@satvu satvu requested a review from Copilot July 9, 2025 01:18
Copilot

This comment was marked as outdated.

@satvu satvu marked this pull request as ready for review July 10, 2025 20:38
@satvu satvu requested a review from a team as a code owner July 10, 2025 20:38
[InlineData(3, true)]
[InlineData(1, false)]
[InlineData(3, false)]
public async Task Shutdown_FailsInFlightInvocations(int numberOfInvocations, bool hasFailureException)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Prior to this change, calling TryFailExecutions (replaced by Shutdown) would not fail executions without an exception being passed in. This test makes sure that in-flight invocations are failed whether or not an exception is passed in.

@satvu satvu requested review from kshyju, fabiocav and Copilot July 16, 2025 01:44
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR modifies the worker channel shutdown behavior to better handle in-flight invocations when a worker channel encounters a fatal error. The key change is replacing the TryFailExecutions method with a more descriptive Shutdown method that properly fails in-flight invocations with a specific exception type.

Key changes:

  • Replace TryFailExecutions method with Shutdown method across all worker channel implementations
  • Introduce WorkerShutdownException to provide more specific error information for failed invocations
  • Update HTTP proxy service to properly handle worker shutdown scenarios during request retries

Reviewed Changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
src/WebJobs.Script/Workers/Rpc/IRpcWorkerChannel.cs Updates interface to replace TryFailExecutions with Shutdown method
src/WebJobs.Script.Grpc/Channel/GrpcWorkerChannel.cs Implements new Shutdown method with WorkerShutdownException
src/WebJobs.Script/Exceptions/WorkerShutdownException.cs Adds new exception type for worker shutdown scenarios
src/WebJobs.Script/Http/RetryProxyHandler.cs Updates HTTP retry logic to handle worker shutdown exceptions
src/WebJobs.Script/Http/ScriptInvocationRequestTransformer.cs Adds transformer to pass script invocation context through HTTP proxy
src/WebJobs.Script/Http/DefaultHttpProxyService.cs Integrates new transformer and adds invocation context to HTTP requests

@satvu satvu requested review from kshyju and liliankasem July 17, 2025 20:25
@@ -29,6 +29,13 @@ public async Task OnTimeoutExceptionAsync(ExceptionDispatchInfo exceptionInfo, T
{
FunctionTimeoutException timeoutException = exceptionInfo.SourceException as FunctionTimeoutException;

// this seems to happen when the worker channel is already shutting down. Ex. One timeout is being handled and another comes in during shutdown.
if (timeoutException is null)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's not FunctionTimeoutException, what exception the caller is sending to this method?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this seems odd, the method is OnTimeoutExceptionAsync, but we are now checking if it was called for a non-timeout? Additionally, the log here just assumes that if timeoutException is null, then the channel must be shutting down? Will that always be true? How future proof is that assumption?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've only reproduced this twice, but timeoutException has ended up null in this method - still digging into why that is, but from my observation of the two events is as described in the comment. Let me continue to investigate this.

Copy link
Contributor

@jviau jviau left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My main concern is the potential breaking change by switching to a WorkerShutdownException. But maybe my assumption this exception makes it to extensions is wrong?

@@ -29,6 +29,13 @@ public async Task OnTimeoutExceptionAsync(ExceptionDispatchInfo exceptionInfo, T
{
FunctionTimeoutException timeoutException = exceptionInfo.SourceException as FunctionTimeoutException;

// this seems to happen when the worker channel is already shutting down. Ex. One timeout is being handled and another comes in during shutdown.
if (timeoutException is null)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah this seems odd, the method is OnTimeoutExceptionAsync, but we are now checking if it was called for a non-timeout? Additionally, the log here just assumes that if timeoutException is null, then the channel must be shutting down? Will that always be true? How future proof is that assumption?

@satvu
Copy link
Member Author

satvu commented Jul 23, 2025

The new approach:

To avoid a breaking change with Durable, we will introduce an exception that derives from the existing FunctionTimeoutException, and looks like this: FunctionAbortException : FunctionTimeoutException. This will only be used when the exception passed in is null or FunctionTimeoutException to preserve the current behavior for all other logs passed in (in case other components depend on any other type exceptions this method sets results to).

In these cases, in the Shutdown method, we create a new FunctionAbortException to fail executions. For the FunctionTimeoutException cases, we additionally pass in the original FunctionTimeoutException as an inner exception (?).

Since durable already has handling for FunctineTimeoutException in some places, this will ensure that the resulting behavior is as desired by the extension team. This also covers other extensions that perform any specific behavior based on this long-existing exception.

Edit: The new exception type is also being added to the WebJobs SDK (Azure/azure-webjobs-sdk#3139) so extensions can handle this type of failure with more granularity.

@satvu satvu requested review from kshyju and jviau July 24, 2025 00:16
@satvu satvu added the blocked label Jul 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Worker restart due to function timeout does not fail current executions
5 participants