JOSS review

[betolink]

Hi SlideRule! I started to review/learn how the different components work for the JOSS review openjournals/joss-reviews#4982 and thought that a discussion would be better than opening issues. So far the docs at https://slideruleearth.io/ look great, I think you should include this URL as the official documentation in all the repos. I'll go through the stack from a user's perspective and keep my questions on this thread if that's good with you all. @dshean @jpswinski

[dshean]

Thanks @betolink! Discussion sounds good. Appreciate your feedback.

[betolink]

Hi team, the documentation for SlideRule is pretty great. I like the high-level overview and the documentation on the specific functions. There are a few broken links to examples but other than that the organization is well thought.

I’m listing some questions I have about the different components of SlideRule.

Backend

1. Is there any special sauce for deploying the stack? (NASA/NSIDC credentials linked to ACLs etc?)
2. Do you have an estimated operating cost for the infrastructure?
3. Is there any awareness/coordination with other projects that may benefit from this novel approach? I’m thinking of https://github.com/icesat2py/icepyx/

H5Coro

1. Generalization: this library could be a breakthrough in scientific workflows that use HDF5 from cloud storage, especially with NASA and other agencies migrating their data to the cloud. I understand that this is tied to funding but is there a roadmap to generalize it enough so that it can be used directly in a more user friendly way (python bindings) or have it integrated with h5py/h5netcdf etc? I guess I’m trying to make the case for making H5Coro more interoperable with the PyData ecosystem to broaden its impact.
2. Comparison with other libraries (Kerchunk like in this example. https://github.com/lsterzinger/2022-esip-kerchunk-tutorial/blob/main/01-Create_References.ipynb) things have moved since H5Coro started and I’d be pretty curious about having a comparison with Kerchunk for some selected datasets. Down the road this would be important so users can decide which tool would be better suited for their needs and access patterns.

Widget

1. Simpler version with sensible defaults: I feel like the UI is a bit overwhelming for the uninitiated in the ancient art of photon counting. Perhaps you can have just the most important controls visible and collapse the rest into an “advanced” mode? This is not as important if the target audience is only academic people.

2. Ability to export the subsetted ROI: I think there is a great opportunity to provide a subsetter, I love plotting my region of interest but what about if I want to download the data? (thinking of NetCDF and/or GeoParquet or even this new https://copc.io/ format from the creators of the Laz format)

None of the above questions/suggestions are a blocker for the review as I could use the widget and that effectively demonstrates the stack is working and the documentation is in pretty good shape.

[jpswinski]

Hi @betolink, thank you for reviewing our work and providing such a rich set of feedback (it is really appreciated!). I will do my best to answer what I can.

Backend

1. Yes and no. The no portion is easier, so I'll start with that. The basic cluster that's needed to provide on-demand data products is fairly straightforward to set up and get running. It consists of a custom load balancer that uses HAProxy, a system monitor that uses Grafana/Prometheus/Loki, and a bunch of sliderule server nodes that do the actual data processing. I run all of it on my laptop for development, and it could easily get setup and run on a rack or some other infrastructure pretty quickly. That's not to say it wouldn't be tedious getting all the port mappings, etc., worked out. But it is straightforward. On the other hand, running in AWS is not as easy and really does require a good bit of knowledge on how to architect and deploy. For both cases (local, and cloud), we have a private repository called sliderule-build-and-deploy which has things like docker-compose files for running locally, and terraform files for deploying to AWS. If you are interested, we can make you a member of our GitHub organization so you can access that repository.

2. The costs are variable depending on the dynamic size of the cluster over time - but to run a 7-node cluster full time is about $54 per day. That's the size of the cluster we use for our public service, which has proven adequate for past demand. We temporarily scale the cluster resources for short periods when we expect higher demand (e.g., during hackweek tutorials), which increases daily costs by ~50%.

3. We highlighted icepyx in the paper under the State of the Field: ICESat-2 packages section (https://github.com/ICESat2-SlideRule/paper/blob/main/paper/paper.md#icesat-2-packages). We have collaborated with the icepyx leadership team on some SlideRule functionality, and even developed a custom Python client module to support SlideRule requests from icepyx (https://slideruleearth.io/rtd/api_reference/ipxapi.html) . We hope that icepyx and other projects will leverage the SlideRule service moving forward.

H5Coro

1. We'd love to generalize H5Coro so that more people could benefit from it, and we do have Python bindings written in C++. I've given multiple talks at different HDF5 conferences on the subject of what would be needed to make H5Coro more accessible, but have failed to get the traction needed to really make a go of it. The fundamental problem is that the HDF5 specification is not really a data format spec but a document of the internal format used by the hdf5 library. As such, the specification is extremely long, complicated, and full of one-off features that were added as different funding customers made requests for their unique case. It makes supporting the entire spec very difficult. H5Coro supports a subset of the spec that roughly matches the default behavior of the library when writing an HDF5 file using normal API calls. This works for many datasets, but not for all. I continue to have a working relationship with the HDF Group, and we are currently collaborating on a funded project to make HDF5 more cloud friendly - but it's a long road with a lot of challenges. If you are interested in pursuing this, I'd be happy to meet with you separately to discuss in more detail.

2. The main difference between H5Coro and Kerchunk/Zarr and DMR++ is that H5Coro does not use or need a metadata repository of mappings into the HDF5 file. My strong recommendation for any future project is to not use HDF5 but to use a cloud optimized format like GeoParquet (or Zarr, etc.). But for projects that must use HDF5, then the institute generating and hosting the data should also generate the index files along with them (like Zarr or DMR++). If those index files are available, then we can all benefit from them. But if they haven't generated those files, then for projects like us, we looked at the costs associated with building a pipeline to reprocess all of the ICESat-2 data every six months (per each release), and produce these index files, and then host them, and it would have been more costly than everything else we were doing put together. That's why we went with H5Coro - because it doesn't require an index file - it reads the file as is and uses a mix of caching, concurrency, and a look-ahead algorithm to greatly reduce the time it takes to read the relevant portions of the H5 file. Another issue with the index file approach is that some HDF5 datasets use internal chunk sizes that are very small (on ICESat-2 they are just hundreds of kilobytes per chunk). Even with the metadata indexes, the tools still need to be able to read multiple chunks that are near each other in the object with one read. We benchmarked some software that used indexing and read all the chunks in parallel and found that it couldn't scale when we made requests to read a few hundred thousand chunks at once (which is normal for SlideRule); it quickly resulted in hundreds of thousands of individual S3 reads (and therefore TCP/IP connections), which bottlenecked the whole system.

Widget

1. This suggestion has come up before, and a collapsible advanced mode is a great idea. I added it to our GitHub issue (Voila demo update sliderule-python#83) on the topic.

2. The UW folks have wanted this option for a while too. I added a new comment to Voila demo update sliderule-python#83 on it as well, to preserve a note about the data formats you suggested.