-
Notifications
You must be signed in to change notification settings - Fork 272
Fix GetWorkerStopChannel() for Local Activities #1965
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix GetWorkerStopChannel() for Local Activities #1965
Conversation
…use this to avoid potential race condition
test/integration_test.go
Outdated
| ctx, cancel := context.WithCancel(context.Background()) | ||
| defer cancel() | ||
| localActivityFn := func(ctx context.Context) error { | ||
| // There is a slight delay between when workerStopChannel closes and when backgroundContextCancel is canceled |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well the delay should be the graceful shutdown time no?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
workerStopChannel should be closed during graceful shutdown and then the context should be cancelled when the graceful period elapse or the shutdown finished
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hmm, looking at the graceful shutdown case, there is no delay.
but looking at TestLocalActivityWorkerShutdownNoHeartbeat, which I realized is non-graceful shutdown, we need a sleep. Otherwise, both LAs are able to complete before shutdown stops the LA.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hm for graceful shutdown there should be a delay between the channel closing and the context closing. The context shouldn't close during the graceful shutdown period
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was getting confused because the tests are doing different things. Added some comments to the tests to hopefully avoid any future confusion
For the non-graceful test, we want the LA to return context canceled err in order to test that we don't heartbeat on failure.
For the graceful tests, we want the LA to wait for shutdown to start before we complete, so when ctx is canceled doesn't matter in these tests
The context shouldn't close during the graceful shutdown period
I've separately confirmed that this is true.
What was changed
Plumb WorkerStopChannel through for Local Activities
Why?
Looks like this was never implemented to begin with
Checklist
Closes GetWorkerStopChannel no longer indicates worker shutdown for Local Activities #1963
How was this tested:
Changed a few tests to use this to avoid a race condition