Added Fastapi endpoint for reserialize_dags #47943

shubham-pyc · 2025-03-19T06:14:38Z

What
Added REST ENDPOINT to replicate CLI functionality of airflow dags reserialize

Updated endpoint /public/dags/manage/reserialize

boring-cyborg · 2025-03-19T06:14:42Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: [email protected]
Slack: https://s.apache.org/airflow-slack

bugraoz93

Thanks for the changes! Added suggestions

airflow/api_fastapi/core_api/datamodels/dags.py

bugraoz93 · 2025-03-19T07:48:54Z

airflow/api_fastapi/core_api/datamodels/dags.py

+    def validate_bundle_names(cls, value):
+        """Validate bundle names format and check for duplicates."""
+        manager = DagBundlesManager()
+        all_bundles = list(manager.get_all_dag_bundles())


I am not sure if we need to init and load all the DAG bundles for each request. Maybe we can have a method in it to get_all_dag_bundles_by_name()

bugraoz93 · 2025-03-19T07:51:11Z

airflow/api_fastapi/core_api/routes/public/dags.py

+
+
+@dags_router.post(
+    "/manage/reserialize",


Suggested change

"/manage/reserialize",

"/reserialize",

I think we don't need additional manage. It would be simpler

In that case you won't be able to have a DAG with ID: reserialize

because we have an API router with /public/dags/{dag_id}

bugraoz93 · 2025-03-19T07:56:14Z

airflow/api_fastapi/core_api/routes/public/dags.py

+        )
+
+    except Exception as e:
+        session.rollback()


This could maybe go to finally?

bugraoz93 · 2025-03-19T08:10:05Z

airflow/api_fastapi/core_api/routes/public/dags.py

+    except Exception as e:
+        session.rollback()
+        raise HTTPException(
+            status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,


I'm not sure about this broad status code. It should also be documented in openapi spec.

Co-authored-by: Bugra Ozturk <[email protected]>

jedcunningham

First off, thanks for opening a comprehensive first PR! It looks great.

Unfortunately, we can't add an endpoint that will reserialize all of the DAGs in the api-server process. If you look at our security model, we state that DAG files are not necessary for the scheduler and api-server (it says webserver, but that just hasn't been updated yet 😄), and most importantly, that user provided code should not be executed in those components.

https://github.com/apache/airflow/blob/main/docs/apache-airflow/security/security_model.rst#dag-files-not-needed-for-scheduler-and-webserver

If you look at how the reparse a file feature works, you'll see it needs to communicate with the dag processor via the db in order to carry out reparsing of a specific file. We will need to follow a similar path to do a "reparse everything" feature.

I have a few ideas on how we could do this, but with feature freeze on Airflow 3 being tomorrow (really closer to the end of today 😄), unfortunately, this will have to wait for 3.1 so we have a bit of time to iterate on it.

shubham-pyc · 2025-03-19T23:03:05Z

@jedcunningham, thank you for your clarifications. I will review the security feature and its functionality. It makes sense that user-defined code is not present with the webserver and scheduler, as they can be deployed on separate machines.

I will go through the code to understand how the file is re-parsed, and we can discuss your ideas on improving its implementation.

On a side note, do you have any updates on the 3.1 release timeline?

jedcunningham · 2025-03-19T23:05:42Z

On a side note, do you have any updates on the 3.1 release timeline?

Unfortunately, not at this time. We can merge new features once 3.0 is out though, so the period of time where we cannot get new things like this merged will be approximately 1 month long.

shubham-pyc · 2025-03-20T00:08:21Z

@jedcunningham thanks for the clarification. Is it okay if I keep working on this feature and it just gets merged next month?
I am open to addressing other open issues but any feature would only be able to get merged next month.

shubham-pyc · 2025-03-21T10:04:04Z

@jedcunningham i looked into the code for reparse a file feature

For this feature if we fetch all the file locations of bundles and create a batch insert DagPriorityParsingRequest. Will it work? Since this would create a parsing results for all of our bundles.

file_locations = session.scalars(select(DagModel.fileloc).where(DagModel.name.in_(list(bundles_to_process))))
        # Process each bundle
        parsing_requests = [DagPriorityParsingRequest(fileloc=fileloc) for fileloc in file_locations]
        
        session.add_all(parsing_requests)

lemme know your thoughts

pierrejeambrun

Yes we need to use DagPriorityParsingRequest to trigger the reparsing of appropriate files. There is no DagModel.name I guess you mean DagModel.bundle_name ?

Also people can use no bundle I think. (bundle_name is None). Therefore if this endpoints is only accessible providing the bundles_to_process such setup will not be able to use this feature. Maybe using the dagfolder. (there can be multiple dag folders), no args at all and we reparse 'everything'.

This leaves much room for optimization. We really should use a cache across argument resolution, outlet_events, and inlet_events. Refactoring can be done later though; this works for now.

closes apache#47499 `TriggerDagRunOperator` requires direct DB access to trigger DAG runs, which is not feasible under AIP-72 / Task SDK. Several approaches were considered: 1. **New Airflow API provider** - Move `TriggerDagRunOperator` to a new provider built on top of the HTTP provider and Airflow API. - This would allow DAG authors to configure a dedicated Airflow connection. - However, the Public API currently requires API tokens and does not support username/password authentication, making setup cumbersome. 2. **Pass Task Token to API** - Use Task Token from the controller DAG to authenticate API calls. - This would allow triggering DAGs in the same deployment using the execution API. - However, Task-identity tokens have not been implemented yet. 3. **Handle via Task Execution API (Chosen Approach)** - Raise a `TriggerDagRunOperator` exception. - The Task Runner catches this and invokes a new endpoint in the Task Execution API to trigger the target DAG. Since (2) is not yet available, this PR/commit implements (3) as a temporary solution.

) * Add ability to pull XCom when passing multiple map_indexes * Add comment about LazyXComSequence and orignal description * Apply review comments * Update task_sdk/src/airflow/sdk/execution_time/task_runner.py Co-authored-by: Ash Berlin-Taylor <[email protected]> * Fix review comments --------- Co-authored-by: Ash Berlin-Taylor <[email protected]> Co-authored-by: Amogh Desai <[email protected]>

…47938) Fix AttributeError

Using scalars() silently drops the second object.

* refactor: remove type casting in AppflowHook update_flow method * chore: update mypy-boto3-appflow dependency to version 1.37.0 * chore: update mypy-boto3-appflow dependency to version 1.37.0 * chore: update mypy-boto3-appflow dependency to version 1.37.0 --------- Co-authored-by: Simon Prydden <[email protected]> Co-authored-by: Kalyan R <[email protected]>

)

* Add next run assets data to Dag Schedule component * Update popover bg colors

If we update next run after a backfill, for a dag with catchup enabled, the scheduler may create old runs that are before start date (in the period between the last backfill run and the start date of the dag).

The `src` folder is a folder, not Python package and the file __init__.py there is just wrong :)

…he#48100)

) Signed-off-by: Maciej Obuchowski <[email protected]>

…#48016)

We are now including node_modules into our wheel/sdist files, which makes them rather large. This brings them back down. This was a regression with the core move.

shubham-pyc · 2025-03-24T01:01:57Z

Because the rebase messed up codefile I have created a new pull request with latest changes and per suggestions.

#48174

I am closing this pull request

Added Fastapi endpoint for reserialize_dags

f5584d1

shubham-pyc requested review from bbovenzi, pierrejeambrun, ryanahamilton, jscheffl and ephraimbuddy as code owners March 19, 2025 06:14

boring-cyborg bot added area:API Airflow's REST/HTTP API area:UI Related to UI/UX. For Frontend Developers. labels Mar 19, 2025

Merge branch 'main' into feature/dags_reserialize_api

a05d187

bugraoz93 reviewed Mar 19, 2025

View reviewed changes

Update airflow/api_fastapi/core_api/datamodels/dags.py

863287b

Co-authored-by: Bugra Ozturk <[email protected]>

jedcunningham requested changes Mar 19, 2025

View reviewed changes

pierrejeambrun reviewed Mar 21, 2025

View reviewed changes

uranusjr and others added 12 commits March 22, 2025 13:57

Rewrite db usage in asset decorator operator (apache#47896)

0069a10

This leaves much room for optimization. We really should use a cache across argument resolution, outlet_events, and inlet_events. Refactoring can be done later though; this works for now.

Fix typos in task sdk (apache#47945)

33a8a97

Fix k8s test: Ensure api-server healthy before any API calls (apache#…

0d76a19

…47938) Fix AttributeError

Correctly select multiple columns with execute() (apache#47928)

e87f2a6

Using scalars() silently drops the second object.

Fix broken task mapping due to XCom refactor (apache#47944)

8bb4567

Add asset event extra to events list in asset details page (apache#47935

a77786d

)

Add next run to asset expression (apache#47932)

0b67ad4

* Add next run assets data to Dag Schedule component * Update popover bg colors

docs: enhance airflowLocalSettings value description (apache#47855)

6e867ef

Next run date should not be updated by backfills (apache#47922)

94828de

If we update next run after a backfill, for a dag with catchup enabled, the scheduler may create old runs that are before start date (in the period between the last backfill run and the start date of the dag).

jedcunningham and others added 11 commits March 22, 2025 14:24

Remove accidentally committed file (apache#48096)

e60fe40

Do not allow semicolon in partition_clause (apache#48098)

5561546

make docker swarm service name customizable (apache#47957)

79ed02b

Remove breeze's top-level init file (apache#48101)

37798a0

The `src` folder is a folder, not Python package and the file __init__.py there is just wrong :)

Fix progress retrieval when parallel airflow-core tests are run (apac…

bb3aeff

…he#48100)

serialize http transports contained in composite transport (apache#47444

c185e81

) Signed-off-by: Maciej Obuchowski <[email protected]>

Add DNS endpoint support for GKE Hooks and Operators (apache#48075)

ca7c54f

Update pre-commit setup docs to suggest uv (apache#48074)

0e72df3

Add clarification for sql injection in Airflow security model (apache…

7f55d29

…#48016)

Smaller distribution files (apache#48089)

23e8fce

We are now including node_modules into our wheel/sdist files, which makes them rather large. This brings them back down. This was a regression with the core move.

Remove password extra - nothing uses those deps (apache#48027)

2f0e455

shubham-pyc requested review from ashb, kaxil, amoghrajesh, bolkedebruin, dstandish, eladkal, potiuk, Fokko, mik-laj, hussein-awala, mobuchowski, vincbeck, josh-fell, o-nikolas, uranusjr, rawwar and XD-DENG as code owners March 22, 2025 20:24

shubham-pyc closed this Mar 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Added Fastapi endpoint for reserialize_dags #47943

Added Fastapi endpoint for reserialize_dags #47943

Uh oh!

shubham-pyc commented Mar 19, 2025 •

edited

Loading

Uh oh!

boring-cyborg bot commented Mar 19, 2025

Uh oh!

bugraoz93 left a comment

Uh oh!

Uh oh!

bugraoz93 Mar 19, 2025

Uh oh!

bugraoz93 Mar 19, 2025

Uh oh!

shubham-pyc Mar 19, 2025

Uh oh!

bugraoz93 Mar 19, 2025

Uh oh!

bugraoz93 Mar 19, 2025

Uh oh!

jedcunningham left a comment

Uh oh!

shubham-pyc commented Mar 19, 2025

Uh oh!

jedcunningham commented Mar 19, 2025

Uh oh!

shubham-pyc commented Mar 20, 2025

Uh oh!

shubham-pyc commented Mar 21, 2025 •

edited

Loading

Uh oh!

pierrejeambrun left a comment •

edited

Loading

Uh oh!

shubham-pyc commented Mar 24, 2025

Uh oh!

Uh oh!



		@dags_router.post(
		"/manage/reserialize",

Added Fastapi endpoint for reserialize_dags #47943

Added Fastapi endpoint for reserialize_dags #47943

Uh oh!

Conversation

shubham-pyc commented Mar 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

boring-cyborg bot commented Mar 19, 2025

Uh oh!

bugraoz93 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

bugraoz93 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

bugraoz93 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

shubham-pyc Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

bugraoz93 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

bugraoz93 Mar 19, 2025

Choose a reason for hiding this comment

Uh oh!

jedcunningham left a comment

Choose a reason for hiding this comment

Uh oh!

shubham-pyc commented Mar 19, 2025

Uh oh!

jedcunningham commented Mar 19, 2025

Uh oh!

shubham-pyc commented Mar 20, 2025

Uh oh!

shubham-pyc commented Mar 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shubham-pyc commented Mar 24, 2025

Uh oh!

Uh oh!

shubham-pyc commented Mar 19, 2025 •

edited

Loading

shubham-pyc commented Mar 21, 2025 •

edited

Loading

pierrejeambrun left a comment •

edited

Loading