Explore and add static checks for DAGs for early detection of common issues

[omkar-foss]

Description

As per users' feedback in the Airflow Debugging Survey 2024, 48.3% of respondents chose early issue detection during execution as one of their top 2 choices.

Use case/motivation

Goal of this issue:

• Enhance early detection of DAG issues to minimize dev time
• Some sort of static analysis similar to Ruff checks for DAGs
• Runtime analysis of DAGs if feasible can also be done to save dev time

Related issues

Parent Issue: #40975

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[omkar-foss]

In the context of this issue, DAG linting as mentioned by this article (shared by @potiuk on slack) can also be explored:

https://medium.com/@snir.isl/mastering-airflow-dag-standardization-with-pythons-ast-a-deep-dive-into-linting-at-scale-1396771a9b90

Inspired by above article, may be we can have commands like airflow dag lint <dagname>.py for checking common DAGs issues and airflow dag grade <dagname>.py for grading DAGs based on their code quality.

[potiuk]

Love that idea. We could even had some way of checking for "best practices" - like not using DB while parsing etc. This might also be then used as part of the upgrade-check mechanism that we are planning for Airflow 2-> 3 migration - see #41641 cc: @Lee-W

[Lee-W]

I feel it's a bit different. 🤔 But for not using DB while parsing, that's something we should check in the upgrade check. But the "best practices" thing would probably be something else. Probably integrating with ruff or building our own linter would be better.

[Dev-iL]

Good to see new airflow-specific Ruff rules being added (AIR3##). Nice work @Lee-W!

[zach-overflow]

This is definitely something I'd find immensely useful. In addition to the article linked in the earlier comments, there are some decent prior attempts at doing something similar with static DAG code checks (see links below); however, I think the major benefit for making this feature native to Airflow is that it can rely directly on the built-in DAG parsing logic.

Some of the examples I was referring to:

• pylint-airflow: which is not maintained, but defines some reasonable lint checks as custom pylint checkers.
• airflint: this might be useful as a reference, but it is worth noting that this is an auto-formatter rather than a simple validation tool.

[Lee-W]

We could probably start discussing what the best practice is after Airflow 3.0 is released (best practice now might not reflect the latest best practice we want then)

[omkar-foss]

We could probably start discussing what the best practice is after Airflow 3.0 is released (best practice now might not reflect the latest best practice we want then)

Yes that makes sense, let's add support for best practices linting for Airflow 3.x onwards. I just took a quick opinion and got a good concensus for 3.x onwards on slack.

[omkar-foss]

@Lee-W would you be interested in picking up this issue? Because I suppose you have good context on Airflow static checks by adding Ruff rules for it, among other things.