Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make dask dependency completely optional? #139

Open
TomNicholas opened this issue Mar 23, 2023 · 0 comments
Open

Make dask dependency completely optional? #139

TomNicholas opened this issue Mar 23, 2023 · 0 comments

Comments

@TomNicholas
Copy link
Member

Can we make dask an optional dependency of rechunker?

IIUC conceptually rechunker doesn't explicitly require dask - rechunking plans can be executed with a variety of executors.

This has been discussed in #85 and #87 but there still seem to be hard imports of both dask and dask.array.


Context: In pydata/xarray#7019 I'm working on generalizing xarray to wrap other chunked parallel arrays, in particular cubed. We would like to be able to test xarray's handling of chunked arrays via cubed without ever importing dask. @tomwhite was able to remove the explicit dependency of cubed on dask, but cubed fundamentally depends on rechunker, which currently depends explicitly on dask. I think cubed only imports the parts of rechunker that don't explicitly depend on dask (i.e. the executors, see #87), but at the moment dask will still be installed when rechunker is installed.


Looking at the code, it seems that the dask dependence is already mostly compartmentalized into executors/dask.py. Dask is also imported in api.py, but it's mostly for isinstance() checks against dask.array.Array. We could handle those in a similar way to how xarray handles checking array types against optionally imported modules.

The only imports I'm not immediately sure how to handle are:

cc @rabernat

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant