[Core feature] Task retry support in Flyte Connectors #3265
Conversation
|
Bito Automatic Review Skipped - Draft PR |
machichima
requested review from
cosmicBboy,
davidmirror-ops,
eapolinario,
kumare3,
pingsutw,
samhita-alla and
wild-endeavor
as code owners
machichima
changed the title
-e Signed-off-by: machichima <[email protected]>
machichima
force-pushed
the
6465-connector-retry
branch
2 times, most recently
from
e0e9e35 to
464aa9a
Compare
Signed-off-by: machichima <[email protected]>
machichima
force-pushed
the
6465-connector-retry
branch
from
5b31352 to
2a4df52
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #3265 +/- ##
===========================================
- Coverage 85.12% 47.51% -37.62%
===========================================
Files 312 215 -97
Lines 26972 22497 -4475
Branches 2952 2952
===========================================
- Hits 22960 10689 -12271
- Misses 3182 11236 +8054
+ Partials 830 572 -258 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: machichima <[email protected]>
| def runner_for_async(rpm: int, total: int): | ||
| loop = asyncio.get_event_loop() | ||
| return loop.run_until_complete(helper_for_async(rpm, total)) | ||
| return asyncio.run(helper_for_async(rpm, total)) |
There was a problem hiding this comment.
Create a fresh event loop each time and prevent using existing event loop. This fixes the CI test fail: https://github.com/flyteorg/flytekit/actions/runs/17783377366/job/50554009811
Signed-off-by: machichima <[email protected]> Signed-off-by: Atharva <[email protected]>
3 participants
Tracking issue
Part of flyteorg/flyte#6465
Why are the changes needed?
Currently, our Connector cannot retry when task fail, which results in a loss of features for end users migrating from a plugin to a connector
What changes were proposed in this pull request?
map internal_error to RETRYABLE_FAILED
How was this patch tested?
See test in flyteorg/flyte#6486
Setup process
Screenshots
Check all the applicable boxes
Related PRs
flyteorg/flyte#6486
Docs link
Summary by Bito
This pull request enhances the Flyte connectors by adding task retry functionality for internal error failures and improved error handling, mapping 'INTERNAL_ERROR' to 'RETRYABLE_FAILED'. It also optimizes async runner functions in the test suite for better performance and reliability, significantly boosting the user experience for migration from plugins to connectors.