Skip to content

Use kedro run --namespace in Airflow DAG for grouped node execution #1059

Open
@DimedS

Description

@DimedS

Description

This is a follow-up to the changes made in kedro-plugins#962.

Once PR #1056 is merged, users will be able to use the --groupby=namespace option when converting a Kedro pipeline to an Airflow DAG. This allows all nodes in the same namespace to run together as a single Airflow task.

Currently, the DAG runs:

kedro run --node node1,node2,node3,...

This isn't ideal when a namespace contains many nodes.

To improve this, the Airflow DAG should be updated to run the namespace directly instead of listing individual nodes:

kedro run --namespace=namespace_name

This will make the DAG cleaner and easier to work with.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions