Reading Kerchunk datasets

[tomwhite]

I created a Kerchunk dataset by following the tutorial here: https://fsspec.github.io/kerchunk/tutorial.html#combine-multiple-kerchunked-datasets-into-a-single-logical-aggregate-dataset

I was then able to read the dataset using Xarray and Cubed:

import fsspec
import ujson
import xarray as xr

from cubed import Spec

fs = fsspec.filesystem("")  # local file system to load json from

with fs.open("../kerchunk-example/air_pressure_at_mean_sea_level_combined.json", "rb") as f:
    d = ujson.load(f)

backend_args = {"consolidated": False, "storage_options": {"fo": d, "remote_protocol": "s3","remote_options": {"anon": True}}}

spec = Spec(work_dir="tmp", allowed_mem="1GB")

ds = xr.open_dataset(
    "reference://",
    engine="zarr",
    backend_kwargs=backend_args,
    chunked_array_type="cubed",
    from_array_kwargs={"spec": spec},
    chunks={},
)

print(ds)

Output:

<xarray.Dataset>
Dimensions:                         (time0: 1440, lat: 721, lon: 1440)
Coordinates:
  * lat                             (lat) float32 90.0 89.75 ... -89.75 -90.0
  * lon                             (lon) float32 0.0 0.25 0.5 ... 359.5 359.8
  * time0                           (time0) datetime64[ns] 2020-01-01 ... 202...
Data variables:
    air_pressure_at_mean_sea_level  (time0, lat, lon) float32 cubed.Array<chunksize=(24, 100, 100)>
Attributes:
    institution:  ECMWF
    source:       Reanalysis
    title:        ERA5 forecasts

It would be good to add this to the documentation, but in the meantime this issue will do.

[tomwhite]

Note that I needed cubed-dev/cubed#297 for the example to run.

[TomNicholas]

FYI that virtualizarr will hopefully make any special treatment of kerchunk unnecessary, as then you can use this trick on kerchunk references (using e.g. the KerchunkJSONParser), and read from the ManifestStore using the same code from_zarr API cubed would use to read from any other zarr store.

[tomwhite]

Thanks. I think this is probably superceded by cubed-dev/cubed#646