Open
Description
Description
This is a follow-up to the changes made in kedro-plugins#962.
Once PR #1056 is merged, users will be able to use the --groupby=namespace
option when converting a Kedro pipeline to an Airflow DAG. This allows all nodes in the same namespace to run together as a single Airflow task.
Currently, the DAG runs:
kedro run --node node1,node2,node3,...
This isn't ideal when a namespace contains many nodes.
To improve this, the Airflow DAG should be updated to run the namespace directly instead of listing individual nodes:
kedro run --namespace=namespace_name
This will make the DAG cleaner and easier to work with.