Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move Asset user facing components to task_sdk #43773

Open
wants to merge 28 commits into
base: main
Choose a base branch
from
Open

Move Asset user facing components to task_sdk #43773

wants to merge 28 commits into from
+902 −733

Conversation

Lee-W
Copy link
Member

@Lee-W Lee-W commented Nov 7, 2024 •
edited
Loading

Why

As part of AIP-72

Closes: #43619

What

  • Move "Asset", "AssetAll", "AssetAny", "Dataset", "Model", "@asset" from Airflow Core to task_sdk
  • Import Asset from task_sdk for common.compat provider
  • Fix ImportError handling in common.compat provider
    • Add a temporary workaround for this error for the following providers (which should be removed after common.compat provider change in this PR has been released)
      • providers/openlineage
      • providers/google
      • providers/amazon

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Lee-W Lee-W added the legacy api Whether legacy API changes should be allowed in PR label Nov 7, 2024
@Lee-W Lee-W force-pushed the move-asset-to-task-sdk branch 6 times, most recently from c7ecee5 to 5c8d6e2 Compare November 7, 2024 10:02
@Lee-W Lee-W marked this pull request as ready for review November 7, 2024 11:40
@Lee-W Lee-W requested a review from uranusjr November 18, 2024 13:29
@Lee-W
feat(task_sdk): Move asset from Airflow core to task_sdk
04da447
@Lee-W
feat(task_sdk): Move assets.metadata to task_sdk.definitions.asset
59e3616
@Lee-W
fix(providers/amazon): fix common.compat provider ImportError handling
26577f5
@Lee-W
fix(providers/google): fix common.compat provider ImportError handling
764e1a2
@Lee-W
fix(providers/openlineage): fix common.compat provider ImportError ha…
21e3401 
…ndling
@Lee-W
fix(provider/common/compat): fix common.compat provider ImportError h…
b795f89 
…andling
@Lee-W
feat(task_sdk): expose Model
f9f420e
@Lee-W
docs(nesfragements): update how asset module should be imported
96127db
@Lee-W
fix(task_sdk): fix 2_10 compatibility
ae2345d
@Lee-W
feat(common.compat): use version to decide how to import assets inste…
0b562ea 
…ad of exception
@Lee-W
feat(providers/common.compat): use airflow version instead of excepti…
d32a75b 
…on to return compat method
@Lee-W
refactor(providers/common/compat): extract airflow version to __init__
33538b1
@Lee-W
fix(providers): use version compare to decide whether to import asset
dc70d7a
@Lee-W
feat(decorators/asset): move @asset to task_sdk
65f02f2
@Lee-W
refactor(asset): rename _AssetAliasCondition as AssetAliasCondition
7da3dfe
@Lee-W
feat(task_sdk): make airflow.sdk.definitions.decoratos a package
0113884
@Lee-W
Revert "feat(task_sdk): make airflow.sdk.definitions.decoratos a pack…
6dfb09b 
…age"

This reverts commit 324efc0.
@Lee-W
feat(task_sdk): move asset related logic in airflow.sdk.definitions.d…
1e77746 
…ecorators to airflow.sdk.definitions.asset.*
@Lee-W
refactor(task_sdk): move @asset to airflow.sdk.definitions.asset.deco…
87b09a4 
…rators
@Lee-W
test(providers/amazon): remove unnecessary compat handling
2f9d913
@Lee-W
test(providers/google): remove unnecessary compat handling
c61a4be
@Lee-W
test(openlineage): remove unnecessary compat handling
f1c10d7
@Lee-W
fix(provider/openlineage): fix how asset compat is handled
aecc6ce
@Lee-W
feat(task_sdk/asset): expose extract_event_key
4377a36
@Lee-W
test(providers/google): change Asset import back to common.compat
c239ec3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bolkedebruin bolkedebruin Awaiting requested review from bolkedebruin bolkedebruin is a code owner

@potiuk potiuk Awaiting requested review from potiuk potiuk is a code owner

@ashb ashb Awaiting requested review from ashb ashb is a code owner

@jedcunningham jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

@mobuchowski mobuchowski Awaiting requested review from mobuchowski mobuchowski is a code owner

@eladkal eladkal Awaiting requested review from eladkal eladkal is a code owner

@o-nikolas o-nikolas Awaiting requested review from o-nikolas o-nikolas is a code owner

@ryanahamilton ryanahamilton Awaiting requested review from ryanahamilton ryanahamilton is a code owner

@bbovenzi bbovenzi Awaiting requested review from bbovenzi bbovenzi is a code owner

@pierrejeambrun pierrejeambrun Awaiting requested review from pierrejeambrun pierrejeambrun is a code owner

@jscheffl jscheffl Awaiting requested review from jscheffl jscheffl is a code owner

@ephraimbuddy ephraimbuddy Awaiting requested review from ephraimbuddy ephraimbuddy is a code owner

@XD-DENG XD-DENG Awaiting requested review from XD-DENG XD-DENG is a code owner

@uranusjr uranusjr Awaiting requested review from uranusjr uranusjr is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
airflow3.0:breaking Candidates for Airflow 3.0 that contain breaking changes area:API Airflow's REST/HTTP API area:lineage area:providers area:Scheduler including HA (high availability) scheduler area:serialization area:task-execution-interface-aip72 AIP-72: Task Execution Interface (TEI) aka Task SDK area:webserver Webserver related Issues kind:documentation legacy api Whether legacy API changes should be allowed in PR provider:common-compat
Projects
AIP-72 - Task Execution Interface and SDK
Status: In Progress
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Move Asset user-facing components to task_sdk
2 participants
@Lee-W @uranusjr