Showing DAGs in folder like structure

[msumit]

Description

The DAGs main page looks messy when we've too many dags. What if we can create a soft foldering around it? I mean a UI-only folder structure to group DAGs. There could be 2 possible ways to group (or maybe more). The first way is grouping via subpaths under DAGS_FOLDER where the actual dag file is present. The second way is via tags. There could be a toggle button at the top to get the flat list structure as we've at present.

@bradarndt @ryanahamilton what do you guys think?

Use case/motivation

This feature would help keep the DAGs main page neat and clean. It would also motivate the users to arrange the dags as per their functionalities.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

+1

[potiuk]

Also this discussion might be relevant #18551 - where users expressed an interest in seeing "related" files. This seems quite connected (especially if we can assume that subpaths can be used for groupping and showing relevant files).

This sounds to me like a small AIP waiting to be written

[msumit]

#18551 is an interesting ask, but after DAG serialization I think it's quite difficult to achieve.

[potiuk]

#18551 is an interesting ask, but after DAG serialization I think it's quite difficult to achieve.

True. unless we serialize more :)

[erdos2n]

@potiuk we could serialize the relative path of the dag_folder? Thoughts?

[potiuk]

This is more complex. WebServer does not need (and should not need) access to the dag folder, so you'd have to serialize the whole folder - which might containe also other DAG files for example and a bit too much to serialize.

After thinking a bit more, I think the only "sane" way to add extra files is to specify what to serialize together with the dag - like a glob of the files to add to serialized form. In which case it will be very diffeent from what @msumit idea is with the folder structre - which will be quite different task (and should be discussed in #18551).

[msumit]

I believe implementing 18551 shouldn't be Airflow's responsibility. This is something that can be supported by Airflow SAAS vendors or even Git-based solutions. For example, years back we built a DAG Explorer to achieve something very similar.

[potiuk]

I believe implementing 18551 shouldn't be Airflow's responsibility. This is something that can be supported by Airflow SAAS vendors or even Git-based solutions. For example, years back we built a DAG Explorer to achieve something very similar.

I think you are right @msumit . I have no very strong opinion there. But you could theoretically even write a custom plugin to do that, to pull your "linked" sources" from somewhere else - not necessarily from Airflow DB - but straight from the source of the data. Serializing those external "data" to the database just to show it in the UI seems like huge waste.

[msumit]

@potiuk totally agree with you.

[draev]

Who is in charge ? Should we wait somewhen for upgrading UI and be able to group DAGs in folder structure ? Have more than 500 DAGs and it became more difficult to manage it.

[potiuk]

Whoever picks up the task. It can be you for example. This is an Open-Source project with > 2.500 contributors - anyone can pick up such task and implement it.

[draev]

Fully agree with you, but taking in consideration the duration of this open ticket, i thought that somebody possible started to do something.

As well, I see that this issue is not so popular.. maybe there is another way to organize your DAGs (except tags), I just missed out 🤔

[ammarchalifah-bolt]

I'm very interested with this feature. Any update with this?