Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ability to subclass from both Input and Output allowing for passthrough IO #1109

Closed
wants to merge 2 commits into from

Conversation

elliotgunton
Copy link
Collaborator

Pull Request Checklist

Description of PR
This PR makes it possible to subclass Input and Output for the new decorators in HEP0001, and then pass an output from a script template call directly into the next step or task. i.e. instead of

@w.dag()
def my_dag():
    output_task = give_output()
    take_input(
        PassthroughIO(
            my_str=output_task.my_str,
            my_int=output_task.my_int,
            my_artifact=output_task.my_artifact,
        )
    )

you can do

@w.dag()
def my_dag():
    output_task = give_output()
    take_input(output_task)

Note this functionality is limited to script-decorated functions only for now, as other template types would need an even more involved approach where this code had already become complex enough, as we are reconstructing an Output from a step or task object.

@elliotgunton elliotgunton added type:enhancement A general enhancement semver:minor A change requiring a minor version bump labels Jun 20, 2024
@elliotgunton elliotgunton added this to the HEP0001 - new decorators milestone Jun 20, 2024
Makefile Outdated Show resolved Hide resolved
Copy link
Collaborator

@sambhav sambhav left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would prefer if we stay consistent with the functionality around our new templates. I think this will cause a lot of confusion if it only works for script templates, no that they look exactly the same as other templates. Also how would this work for script template stubs?

@elliotgunton
Copy link
Collaborator Author

I think this will cause a lot of confusion if it only works for script templates

Fair point, I wanted to see if it was possible for script templates first. As for stubs it should work fine as long as they are decorated as script templates - we only need the signature, which is why it's fairly easy to do for script templates as we can access template.source. Haven't looked into other template types extensively

@elliotgunton elliotgunton marked this pull request as draft June 21, 2024 08:48
This change means users can use the same object output from one
step and use it as input for the next step, without having to map all
the parameters

Signed-off-by: Elliot Gunton <[email protected]>
@elliotgunton
Copy link
Collaborator Author

Closing this as stale and it's only a POC which isn't high priority for now

@sambhav sambhav deleted the add-passthrough-io branch August 27, 2024 21:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
semver:minor A change requiring a minor version bump type:enhancement A general enhancement
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Arguments mapping is very verbose depending on use case
2 participants