Task retries on specific User Errors

[adartaud]

Hello,

From what I understood, we can setup task retries policies, depending on wether these exceptions are "System" or "User" scoped.
We are running mostly spark tasks in our K8s cluster, and we have identified some "transient" exceptions that can happen during our Spark jobs, that can be simply "recovered" from the Flyte UI. These exceptions are raised as "User Errors" by Flyte since they are raised from within our code, although they should be treated as system errors.

We would like our tasks to have a retry mechanism on these specific exceptions, without actually setting a retry on all User Exceptions. I did not find any configuration that would allow this. One possible workaround I thought of was to actually catch these exceptions ourselves and re-raise them as FlyteScopedSystemException or FlyteSystemException so that they would be treated as System exceptions with their retry policy here: https://github.com/flyteorg/flytekit/blob/5503ee5e232fdbc633af39c7f4539a04906102fc/flytekit/exceptions/scopes.py#L203

Does this workaround seems reasonable ? Do you have a recommended way of handling this ?

Thanks,

[davidmirror-ops]

@adartaud
Yes, that workaround seems reasonable. You may want to try with FlyteSystemException as FlyteScopedSystemException is marked as deprecated

[kumare3]

We are also working on something new that might make this really really configurable. Would you be open to having a chat?

[adartaud]

Sure with pleasure.