Description
Currently we handle sub-pipeline linking by connecting nodes with the same name. This causes a bunch of issues, mostly because connect
is a little bit broken due to how our stream type inheritance works (dask vs standard backend).
A better approach may be to explicitly request certain nodes incoming and then inherit from those. This way we don't need to use connect and the nodes will properly inherit their type at pipeline creation time. This also simplifies the linker's implementation by requiring a mapping of arg/kwarg names to input streams and a list of functions which take those args and kwargs and return dicts.
def linker(initial_inputs, list_sub_funcs):
namespace = initial_inputs
for sub_func in list_sub_funcs:
namespace.update(sub_func(**namespace))
return namespace
Note that this requires all sub_funcs
to allow for arbitrary kwargs. We could also inspect the signature, but that seems more brittle.