Global gantt chart (across all DAGs)

[alejandrofm]

Description

Hi, now we have a Gantt view inside each DAG which helps to understand the overlapping/idle times/etc on each one. The request is for a global Gantt across all DAGs

Use case/motivation

This will allow knowing where Airflow and all the services that depend on it will have idle times, maybe look at the chart by tag. Optimizations on a lot of fronts come to mind when I think of this feature.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[bbovenzi]

I would very much like to improve out cross-dag views as part of AIP-38. We introduced the Dag Dependency graph, but it would be great to do more.

[DataDune]

It would also help in cross checking are all depending dags properly scheduled. I'd love to see done.

[FranArenas]

It would be interessant if it would be capable of non only tracking the start point of each dag, but also showing the mean duration time on each one

[danieldean]

This feature would be very much welcomed to determine when admittedly limited server resources have spare capacity available to slot in additional dags or shuffle around existing ones. I was quite surprised to find it had not made it's way into Airflow yet.

In the meantime I have been able to make my own gantt by connecting to Airflow via the API using a popular data visualisation tool. I have it setup so I can select a specific date but showing an average would also be possible. At the moment it does not show dependencies but this is also possible.

If I have available time I will take a look to see how this might be added into Airflow itself.

[bbovenzi]

I still think this is a good feature and can be included in the new homepage in Airflow 3

[guan404ming]

Hi @bbovenzi, I am interested in this issue. Could I help with this?
Also, would like to ask do you have any recommendation package to implement this? I do see some gantt chart package but most of them are not open source or are lack of maintenance. Or we should implement it by ourselves maybe using chart.js?
Thanks in advance!

[bbovenzi]

Yeah this is fairly complex.

First, we have a separate issue to reimplement the dag specific gantt view: #44672 Which would be the same as we had in 2.x, a custom gantt chart that is lined up with the grid view and shares the same scrolling

For across dags, I do wonder if a gantt chart is the best visualization or if a google calendar style view would be better. Maybe something like https://github.com/jquense/react-big-calendar. I think we can start with just using the list dag_runs endpoint. And later on we can add more events like showing when dags are scheduled.