Skip to content

async internally for read_elem and write_elem #1897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
4 tasks
ilan-gold opened this issue Mar 7, 2025 · 2 comments · May be fixed by #1902
Open
4 tasks

async internally for read_elem and write_elem #1897

ilan-gold opened this issue Mar 7, 2025 · 2 comments · May be fixed by #1902

Comments

@ilan-gold
Copy link
Contributor

ilan-gold commented Mar 7, 2025

Please describe your wishes and possible alternatives to achieve the desired result.

After #1726 people will be able to have zarr return async compatible arrays. We can update all of our internal io to pool requests together using asyncio.gather and give us more throughput, especially for cloud-based analysis.

  • Internally use async for the registered reading classes
  • Make a public async version of read_elem
  • Make a public async version of read_lazy/read_elem_lazy
  • Make a public async version of read_zarr/read_h5ad
@ilan-gold
Copy link
Contributor Author

The main issue here is the interplay between read_dispatched and the underlying methods in methods,py which call _Reader.read_elem (thus hooking back into the callback from read_dispatched). With a single set of methods.py that are all async, the functions hook into read_elem_async which can be problematic if callback is synchronous. Thus we probably need either a really clean compat layer (not sure what this would look like, another argumet? but then you just have giant if-then blocks?) or more-or-less duplciated functions

Copy link

This issue has been automatically marked as stale because it has not had recent activity.
Please add a comment if you want to keep the issue open. Thank you for your contributions!

@github-actions github-actions bot added the stale label May 12, 2025
@ilan-gold ilan-gold removed the stale label May 12, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant