-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Add support for async callables in PythonOperator #60268
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
9124c9a to
cd4ec26
Compare
ebf2873 to
0b341e6
Compare
…epends on modified comms supervisor which cannot be backported to older Airflow versions Add support for async callables in PythonOperator (apache#59087) * refactor: Implemented BaseAsyncOperator in task-sdk * refactor: Now PythonOperator extends BaseAsyncOperator * refactor: Also implement BaseAsyncOperator in common-compat provider to support older Airflow versions --------- Co-authored-by: Jason(Zhe-You) Liu <[email protected]> (cherry picked from commit 9cab6fb)
providers/common/compat/src/airflow/providers/common/compat/standard/operators.py
Outdated
Show resolved
Hide resolved
providers/common/compat/src/airflow/providers/common/compat/standard/operators.py
Outdated
Show resolved
Hide resolved
providers/common/compat/src/airflow/providers/common/compat/standard/operators.py
Outdated
Show resolved
Hide resolved
providers/standard/src/airflow/providers/standard/example_dags/example_python_decorator.py
Outdated
Show resolved
Hide resolved
providers/standard/src/airflow/providers/standard/example_dags/example_python_operator.py
Outdated
Show resolved
Hide resolved
…andard/operators.py Co-authored-by: Kaxil Naik <[email protected]>
24536ae to
cdecb12
Compare
…andard/operators.py Co-authored-by: Kaxil Naik <[email protected]>
597f08a to
f829c3e
Compare
…nstead of NotImplementedError
kaxil
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Almost there, once newsfragment is added we are good
And the newfragment ;-) |
|
@kaxil If newsfragment is ok for you then I think it's finally ready to merge. |
|
Static check is failing, worth looking at https://github.com/apache/airflow/blob/main/airflow-core/newsfragments/54505.significant.rst or other examples |
This PR is related to the discussion I started on the devlist and which allows you to natively execute async code on PythonOperators.
There is also an AIP for this: https://cwiki.apache.org/confluence/display/AIRFLOW/%5BWIP%5D+AIP-98%3A+Rethinking+deferrable+operators%2C+async+hooks+and+performance+in+Airflow+3
Below an example which show you how it can be used with async hooks:
This PR will fix additional remarks made by @kaxil on the original PR which has been reverted.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.