-
Notifications
You must be signed in to change notification settings - Fork 65
Description
Originally asked at #NGFF > OME-Zarr: status of sharding support @ 💬
"I noticed ome-zarr-py now supports zarr v3 🎉
Josh pointed me to using storage options for sharding, which indeed works!
However, now I want to write a dask array with sharding, but it seems to only chunck (following the dask chunks presumably). How can I get sharding writing a dask array?"
Issue at: dask/dask#11778
Also see:
zarr-developers/zarr-python#3514
"I'm not sure what dask should do here -- calling to_zarr in this case is not unsafe if you choose compute=False and handle manually handle the execution of the tasks yourself, or if you are operating in single-threaded mode. But zarr does not provided any file locking so any concurrent writes to single shards will be unsafe. Maybe dask should emit a warning here?
We could invest in making concurrent shard writes work better but I think it's fundamentally an anti-pattern: you should not expect to write concurrently within a single file, barring exceptional circumstances."