Skip to content

explicit linking #19

Open
Open
@CJ-Wright

Description

@CJ-Wright

Currently we handle sub-pipeline linking by connecting nodes with the same name. This causes a bunch of issues, mostly because connect is a little bit broken due to how our stream type inheritance works (dask vs standard backend).

A better approach may be to explicitly request certain nodes incoming and then inherit from those. This way we don't need to use connect and the nodes will properly inherit their type at pipeline creation time. This also simplifies the linker's implementation by requiring a mapping of arg/kwarg names to input streams and a list of functions which take those args and kwargs and return dicts.

def linker(initial_inputs, list_sub_funcs):
    namespace = initial_inputs
    for sub_func in list_sub_funcs:
        namespace.update(sub_func(**namespace))
    return namespace

Note that this requires all sub_funcs to allow for arbitrary kwargs. We could also inspect the signature, but that seems more brittle.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions