RBAC access rights filtering by dag tags

[xuejunteo]

Description

Is there a way to auto generate permissions based on tags?

e.g.
I want to automatically grant access to user based on his role to a specific list of tags

currently only the can dag read on and can dag write on is auto populated when there is a new dag

it would be good if i can do something like can dag read on tag <tag name> and can dag write on tag <tag name>

thanks in advance!

Use case / motivation

I have multiple teams using the same airflow server and access rights granting has been very tedious as it is currently done by adding individual dags manually to the access rights of each team

Related Issues

#6489 (comment)

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[sudarshan2906]

Any update on this? I want to do something similar to this. To give access to a group of dags. Is there any other way to do this. Since this is not taken by anyone?

[kaxil]

Temporarily added it to 2.1 Milestone to revisit this at a later time.

Assigning permissions to a group of DAGs is a good future -- I am not yet sure if "Tags" are the best way for it for the reason James already mentioned. Probably better would be some sort of "wildcard" on permissions -- not sure at the moment.

Also, we should just have a single way to add/edit permissions via UI & API -- not via DAGs themselves -- I know currently we allow it but it is "anti-pattern" in my opinion and is already causing some headaches.

[alete89]

This is like a key feature. I can't believe we currently don't have a way to set permissions to a group of DAGs. Tags may not be the best, but I think it could work, like "You can only see DAGs tagged with data-science tag".
Since this has not been added into 2.1, when are we expecting 3.0 to be launched? Can't we expect this to be merged sooner?

[malthe]

I agree that conflating tags (or other metadata that primarily supports the user interface) with security is a bad thing.

If the mechanism was pluggable somehow, then one strategy could be that the first directory level of the DAGs repository was mapped to a group or "container" – and RBAC roles could then in turn be assigned to those groups. It's a simple strategy that would also fit nicely in the situation where you have multiple remote repositories contributing to the DAGs repository as the first directory level.

The interface should of course be generic enough to support other strategies (including one based on tags).

[potiuk]

This is like a key feature. I can't believe we currently don't have a way to set permissions to a group of DAGs. Tags may not be the best, but I think it could work, like "You can only see DAGs tagged with data-science tag".

My personal opninon - and this is not a universal truth, but my persona view on this is, that this is not really whether we want to use tags or something else. I think main reason is because in essence multi-tenancy feature of Airflow does not have a chance to be working (yet!). By providing "access per-group" UI feature, we would give people a completely false impression, that you can isolate those DAG groups from each other. I personally believe it would be a very bad idea to tell people they can do it, while under the hood they would miss the isolation.

In short - you CAN't currently isolate groups of DAGs in single instance of Airflow. You simply CAN't. it's impossible by design, and we are aiming to iterate and improve the design in the future to make it possible, but it will take a while (and likely will only be fully available in Airflow 3). There are multiple reasons why this kind of RBAC control is a UI-only feature (and can't be easily made into "execution isolation" - while "Role" separation makes sense as UI-only feature (Admin can access different functionality than User via UI).

I personally think introducing groups of DAGs is only really useful (and secure) if we will be able to isolate DAG groups not only via "UI" capabilities but also via "DAG writing" capabilities. I think the reason why you want to introduce this in UI is tha there is something that you want to prevent people doing between the groups (viewing, running, etc.). I think the main reason to give UI access for group is to give it to the same people who can write the dags. For me pretty much all UI "Can see DAGs in this group" access should map 1-1 to "Can write DAGS in this group". It's a bit simplistic view, I can - of course - imagine other reasons for permission separation, but to be honest - giving people the way to "view and operate" DAGs via UI "per-group", and not doing the same for "write code in DAGs" gives a very bad false-sense of security. And I think we should only introduce it when we can do it "full-stack". Otherwise people will be tempted to use current airflow as multi-tenant, where it is in fact not multi-tenant (yet!).

Currently the only way to have multi-tenancy is to have separate airflow instances for each tenant. Full stop. By introducing DAG grouping without going full-stack we give people wrong impression that the story is different. It's not.

Since this has not been added into 2.1, when are we expecting 3.0 to be launched? Can't we expect this to be merged sooner?

I think it will take a while. If you feel the need that you MUST separate the DAGs in the UI, I'd strongly encourage you to implement single instance od Airflow per "group". This is far more robust, already proven, It provides better isolation between groups - actually it's infinitely bigger - comparing to trying to run them in single instance which provides no isolation. Better yet, it provides also capability of isolating the workloads between the different groups (which is completely not possible if you run them in single instance and one group can submit million tasks influencing the other). It can be nicely integrated into any authentication mechanism you have in the way that only users belonging to certain group can have access to each instance. And it can be easily automated to create/drop new "instances" automatically for different groups (been there, done that with nice Terraform templates) - even if you need to manage many of those.

Yeah it uses more resources, but you can also reuse a lot of those resources if you put those instances in auto-scaling K8S clusters. And you can create separate schema for each "group" in the same database to utilise your DB server better.