Reference files system support?

[emfdavid]

Super excited about the potential of rfsspec.
I tried it out with one of my HRRR kerchunk aggregations

I get the following error:

File ~/.cache/bazel/_bazel_builder/99bd544777c52d5a0d8a66ea3b05626a/execroot/ritta/bazel-out/k8-dbg/bin/build/jupyter/jupyter.runfiles/common_deps_rfsspec/site-packages/rfsspec/gcs.py:41, in RustyGCSFileSystem.cat(self, path, recursive, on_error, start, end, **kwargs)
     39     return out
     40 else:
---> 41     return self.cat_file(paths[0], start=start, end=end, **kwargs)

File ~/.cache/bazel/_bazel_builder/99bd544777c52d5a0d8a66ea3b05626a/execroot/ritta/bazel-out/k8-dbg/bin/build/jupyter/jupyter.runfiles/common_deps_rfsspec/site-packages/rfsspec/gcs.py:28, in RustyGCSFileSystem.cat_file(self, url, start, end, **kwargs)
     27 def cat_file(self, url, start=None, end=None, **kwargs):
---> 28     return gcs_cat_ranges([url], start=[start or 0], end=[end or 0], **self.kwargs)[0]

PanicException: called `Option::unwrap()` on a `None` value

Is this just a bug or a major missing feature in rfsspec to support reference file system in zarr?

[martindurant]

Yes, rfsspec can be used in zarr-like access patterns, which I'm pretty sure should include referenceFS with rfsspec as the "remote" FS backend. I'm not certain if reading the JSON will work on gcs (it is a little behind s3), but you can always .cat() the bytes.

The fo you specify looks like a zarr directory rather than a JSON file? Did you register rfsspec's gcs as the fsspec backed for "gcs"?

Probably I will need to do a little bit manually to get it to work.

Oh, I should say that the "panic" there is basically just an exception, but I haven't put in the appropriate handling logic in yet.