Improving Launchplans and Backfill UX

[cosmicBboy]

Objective

The objective of this RFC is to improve the UX around launch plans and the ability to perform backfills both programmatically and on the UI. Currently, activating/deactivating launch plans needs to be done with flytectl and flytekit.SynchronousClient. There's no way to do this on the UI. Furthermore, if a user wants to perform a backfill, flyte provides a CLI command, pyflyte backfill to create an imperative workflow that launches backfill launchplans based on some user-defined time window.

The aim of this RFC is to improve the UX around how users interact with launchplans and backfilling workloads.

[zeryx]

Launch Plans IMO are excellent locations to setup dedicated REST infrastructure. I know we have an undocumented openAPI spec for everything, however a user being able to configure and define something like a fastAPI endpoint (with cors, etc) that could accept arbitrary payloads would be really exciting.

[dennisobrien]

The existing pyflyte backfill functionality is good and something that wasn't obvious to me as a new Flyte user. I do think providing backfill functionality in the UI would be a great feature as well.

For the existing pyflyte backfill there are a few pain points I run into:

• There is a default limit of 100 tasks (or subworkflows?) within a backfill. So I often need to break up a backfill command into segments.
• If a backfill fails and I try to run the same backfill (or even the same launch plan but with different arguments) I get an error "workflow with different structure already exists" with no clear instructions on how to fix it.
• Parallelism seems to just use the default value of 25 instead of the value max_parallelism set in the launch plan. It would be nice to allow setting max_paralleism in pyflyte backfill by allowing an optional value to --parallel.

For the backfill features in the UI, it would be great to have some sort of support for these as well.

• Allow the user to override the 100 tasks limit.
• Allow the user to easily cancel or delete an existing failed backfill. Or running a backfill referencing the same launch plan just works.
• Allow the user to control parallelism in the backfill.
• Allow the user to set the failure policy of the backfill.

[kumare3]

@dennisobrien for

There is a default limit of 100 tasks (or subworkflows?) within a backfill. So I often need to break up a backfill command into segments.
This can be configure and can be increased. This is just a default safety setting

If a backfill fails and I try to run the same backfill (or even the same launch plan but with different arguments) I get an error "workflow with different structure already exists" with no clear instructions on how to fix it.
This seems like a bug, the regenerated workflow may not be identical. Would you be open to sharing an example?

Parallelism seems to just use the default value of 25 instead of the value max_parallelism set in the launch plan. It would be nice to allow setting max_paralleism in pyflyte backfill by allowing an optional value to --parallel
There is a --parallel flag already , which make the backfill run as a fanout. I like the idea of making --limit-parallelism flag. You are right, it will not use the parallelism from the downstream launchplan. This is a simple addition

For the UI,
Can you define the experience? That would be awesome!

[dennisobrien]

@kumare3 Sure. I'll try outlining how I think this should work.

UI for launching a backfill

The UI for launching a backfill should be similar to the UI for launching a workflow and should support the features available in the pyflyte backfill CLI.

• Launch plan (required) - This is possibly pre-filled depending on how the user got to this UI.
• Launch plan version (optional) defaults to the latest version
• Start date - Calendar selection
• End date - Calendar selection
• Parallel or Serial - Selects one mode or the other. Defaults to serial.
• Failure policy - "Fail Fast" (default) or "No Fail Fast".

How to launch a Backfill from the UI

A backfill is always associated with a launch plan, so it seems like there are three possible places backfill functionality could go into the UI:

1. In the Workflow UI: When a user clicks "Workflows" then on a workflow in the UI, currently the available action is "Launch Workflow" where the user specifies the workflow version, launch plan, and inputs. In addition to the "Launch Workflow" we would add "Launch Backfill" which would spawn the "Launch Backfill" UI.
2. In the Launch Plans UI: When a user clicks "Launch Plans" then on a launch plan in the UI, currently there is no specific action. Here we would add "Launch Backfill" which would spawn the "Launch Backfill" UI.
3. In a new Backfill UI: In addition to the existing navigation options "Project Dashboard", "Workflows", "Tasks", and "Launch Plans", this would add a "Backfills" category. From this page the action button is "Launch Backfill" which would spawn the "Launch Backfill" UI.

Managing existing backfills

I don't have any recommendations here mainly because I don't understand how this is supposed to work currently. From the pyflyte backfill documentation:

The backfill command generates and registers a new workflow based on the input launchplan to run an automated backfill. The workflow can be managed using the Flyte UI and can be canceled, relaunched, and recovered.

But I don't see any option to cancel a workflow. I do see options "Recover" and "Relaunch" for a failed backfill, but nothing to cancel.