Airflow 3.0.x : Bring back from Airflow 2.x "Status of all previous DAG runs" column in DAGS main page

[gbonazzoli]

Description

Airflow 3 seems to have now far better foundations in comparison to Airflow 2.

I have only one big missing thing that was present in Airflow 2 and that I can not find in Airflow 3.

It is the "Status of all previous DAG runs" column in DAGS main page.

Use case/motivation

Bring back the functionality, I don't thing that will break some UI style guides. And there is a lot of free space to accomodate it.

It was very quick every morning to look at the red ones, to see what happened.

It wa a matter to click on the red circle !!!

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[jedcunningham]

Have you seen the status bars on the dag list? For example:

Not sure you have the quick nav to only the failures that way, but you do have the visual of which had problems (plus, its recent, not "since the beginning of time").

edit: Oh, and I just noticed there is a "failed" filter at the top!

[gbonazzoli]

The failed filter at the top it's a global filter on DAGS, it is very useful but it does not guide you to the list of the errors in that specific DAG.

The new "status bars" are very cool and useful but they do not count (how many errors happened to that DAG) and do not guide you to the list of the errors in that specific DAG.

One of my use case is to launch backup every night (hundreds of launchs), and at the next morning the operators assess what went wrong.

They click on the "old red circle" see at the error and if it was a shutdown PC they simply relaunch the backup, if it was a bad block discovered they call the user in order to change the disk, and so on.

So the red circle is for me the amount of task we need to assolve every morning.

like this:

This morning, nothing to do !!!

[tirkarthi]

cc: @bbovenzi @pierrejeambrun

[bbovenzi]

We removed the status of all dag runs or task instances because if you run Airflow for long enough it becomes noise. But the current bar chart is limited to the last 14, which depending on a dag, doesn't capture enough for the daily activity. I am open to iterating on the design to give more useful information like a "Last 24 hours" summary of a dag

[gbonazzoli]

@bbovenzi

I agree with you, sometimes the information carried by the coloured circles is irrelevant, but sometimes it is not.

What about having a configuration directive in airflow.cfg to make parametric the order in which to display the blocks on the web UI ?

Something like:

web_ui_dags_page = "#Dag #Schedule #DAG_Runs_Status #Next_Dag_Run #Last_Dag_Run #Tags"

[pierrejeambrun]

I'm not in favor of adding more conf options for the UI. Ideally the UI should solve this problem by its own with an appropriate user interface, allowing to do what's needed to solve this use case. (define a column order or any other solution).

The main reason is that I wouldn't want to see a multiplicity of such params web_ui_dags_page web_ui_ti_page web_ui_asset_page and so on.

[pierrejeambrun]

Maybe what we want is a way to filter failed task instances, over a temporal window + state + list of dag ids ?

For instance this would allow you to find all the failed tasks for your given dags over the last 24 hours. We can also imagine a more complexe filtering options for the dag list.

[bbovenzi]

Yes, I would like to explore improving the UI and filtering functionality before adding new UI config. But something like "List all dags with activity in the last 24 hours" we could then show useful more useful information in the card list than today

[gbonazzoli]

Time range filtering of dags is a great idea !

And with this filtering active, the numbers calculated on "Status of filtered DAG runs" is not noisy anymore !

in my opinion with such a filter in place it will better the user experience in comparison to Airflow 2!

If I can suggest, time range filter in Grafana is something to take inspiration at: