Kannan/temp task started verification timer #9017
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rkannan82
changed the base branch from
main
to
implement_history_server_worker_heartbeat
rkannan82
changed the base branch from
implement_history_server_worker_heartbeat
to
worker_chasm_reschedule_activities_on_lease_expiry
rkannan82
changed the base branch from
worker_chasm_reschedule_activities_on_lease_expiry
to
main
- Removed time from all events that are passed to transition. It was meant for debugging purposes. Technically, chasm can record this when it invokes the transition function. - Moved dynamic config into the chasm config file. - Removed redundant task validation for pure task.
- Method name is now package internal.
We can add this back when we make the frontend handler changes. The frontend handler will use this flag when forwarding the heartbeat to history service.
Blindly creates an activity timer. When the timer fires, ignores it. Why? To prevent task from getting orphaned, we need to create a timer when the task is started. Later, when the worker heartbeat arrives with this task attached, we can verify and cancel the timer.
The RecordHeartbeatRequest was missing the top-level WorkerInstanceKey field, causing requests to be routed to the wrong shard. The handler then uses the WorkerInstanceKey from the nested FrontendRequest, resulting in 'Shard is owned by X but not by Y' errors.
When a worker's lease expires (no heartbeats received), the CHASM executor now reschedules all activities that were running on that worker. This prevents activities from being orphaned when workers die. Changes: - Add ForceRescheduleActivity API to history service - Update LeaseExpiryTaskExecutor to call history service for each activity - Add metrics: chasm_worker_activities_rescheduled, chasm_worker_reschedule_errors - Retry logic with configurable max retries (3) and timeout (5s) - Error types: 'retry' for transient failures, 'permanent' for exhausted retries
chasmworker.HistoryModule requires HistoryClient which is only available in history service. Remove it from ChasmLibraryOptions (shared by all services) to prevent frontend startup failures.
Split worker module: - Module: shared by all services (for tdbg/archetypeID) - HistoryModule: history-only (provides HistoryClient for rescheduling) HistoryClient is now optional via fx.In, allowing the module to work in frontend/matching while only enabling rescheduling in history.
rkannan82
force-pushed
the
kannan/temp_task_started_verification_timer
branch
from
d303235 to
13f2a36
Compare
rkannan82
force-pushed
the
kannan/temp_task_started_verification_timer
branch
from
b69589e to
10e2bca
Compare
rkannan82
force-pushed
the
kannan/temp_task_started_verification_timer
branch
from
10e2bca to
c5ce545
Compare
- Skip non-retryable errors (NotFound, FailedPrecondition, InvalidArgument) in activity rescheduling - Remove debug logging spam from heartbeat handler - Fix heartbeat timeout timer to not re-schedule (return before clearing status bit)
- STARTED activities were being skipped by ResetActivity (no-op) - Added special handling to reset STARTED → SCHEDULED state - Clears StartedEventId, StartedTime, etc. so activity can be picked up - Increments stamp to invalidate responses from dead worker - Regenerates activity task for immediate dispatch - Added worker_chasm: prefix to all logs for easy filtering
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changed?
Describe what has changed in this PR.
Why?
Tell your future self why have you made these changes.
How did you test it?
Potential risks
Any change is risky. Identify all risks you are aware of. If none, remove this section.