[orchagent] Event-based Retry Strategy #3699
Draft
+415
−43
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What I did
Implemented
RetryCacheWhy I did it
The original orchagent performs a brute-force retry strategy for all the previously failed tasks in its cache, which has the risk of blocking the normal workflow when the accumulated failures are scaled, since the event-handling loop and the retry logic is in the same thread.
After observed the delay of normal event handling by previously accumulated failed tasks, we dived into the retry strategy and came to several conclusions. Firstly, not all failures deserve an immediate retry, since its constraint may have not been resolved yet. Secondly, it's worth analyzing the failure reason, and record it together with the failed task, as its constraint. Thirdly, we need a in-time notification mechanism to detect when the constraint is resolved and notify the related tasks to trigger their retry. Lastly, we should make sure the retry stage would not starve the normal event workflow, it can't last too long.
That means, we build up a event-based retry logic, instead of blindly repeating all the pending tasks in the cache in the main thread. The key is to skip unnecessary failure retries when you know they would fail again, and come up with these skipped failures in-time when they have a chance to succeed, instead of forgetting them forever.
How I verified it
Details if related