IOOS Success Story with LiveOcean forecast collection

[rsignell-usgs]

This GSoC was chosen by the IOOS organization, so I just wanted to report that @peterm790's help with understanding a problem with the fill_value in led to a Kerchunk PR which was then successfully applied to an IOOS model collection -- the LiveOcean forecast collection from the NaNOOS Regional Association of IOOS.

We took 24 sample hourly LiveOcean NetCDF from ROMS, put them on the Open Storage Network and kerchunked them into a single virtual Zarr dataset. We were able to modify the metadata, adding the standard_name='time' attribute to the ocean_time variable.

Here is a notebook demonstrating access and simple visualization:

Snapshot:

[rsignell-usgs]

@peterm790 and I had a good discussion about LiveOcean yesterday and he set up a kbatch cronjob script to kerchunk the daily forecast data at 1500 UTC, about 1 hour after it arrives on Azure Blog Storage.

It worked yesterday, fingers crossed it works today!

He's also working on a "Live Ocean" explorer notebook that consumes the data, starting with this Panel-based COAWST Explorer notebook. In this notebook we standardize the dataset in the notebook (e.g. adding standard_name for time, and fixing fill values, but it would be better to fix those in the JSON of the virtual dataset!)

[peterm790]

Just to update this I have set up a new container image at https://github.com/users/peterm790/packages/container/package/pangeo which replicates the pangeo conda enviroment on qhub, so we no longer rely on the OGC demo image. When this is finished updating I will set up a more minimal image for running the liveocean kerchunk update script.

Then the liveocean source data on azure seems to have changed slightly and been extended to a 3 day forecast, so I will need to change the updating script to reflect this.

[rsignell-usgs]

I tried running my LiveOcean Explorer notebook today (which is accessing a kerchunk dataset that is updated daily) and got:

File /home/conda/users/e6aa7ecb5d1fdfdc96b013e60f37b5b6476dfafe607fc361e6c796f9050d1e70-20220801-103922-189931-116-pangeo/lib/python3.9/site-packages/numcodecs/zlib.py:40, in Zlib.decode(self, buf, out)
     37     out = ensure_contiguous_ndarray(out)
     39 # do decompression
---> 40 dec = _zlib.decompress(buf)
     42 # handle destination - Python standard library zlib module does not
     43 # support direct decompression into buffer, so we have to copy into
     44 # out if given
     45 return ndarray_copy(dec, out)

error: Error -3 while decompressing data: incorrect header check

@martindurant do you know where to point the finger?

[martindurant]

Network issue?!?

[rsignell-usgs]

Network issue when creating the JSON (earlier today) or accessing the JSON/NetCDF files (in which case I should just try again later?)

[martindurant]

I imagine the latter, unless you gzip-compressed your JSON.

[martindurant]

The file was created on the 1st but modified on the 3rd - does it need to be rescanned?

{'metadata': None,
 'creation_time': datetime.datetime(2022, 8, 1, 13, 47, 41, tzinfo=datetime.timezone.utc),
 'deleted': None,
 'deleted_time': None,
 'last_modified': datetime.datetime(2022, 8, 3, 13, 39, 46, tzinfo=datetime.timezone.utc),
 'content_settings': {'content_type': 'application/octet-stream', 'content_encoding': None, 'content_language': None, 'content_md5': None, 'content_disposition': None, 'cache_control': None},
 'remaining_retention_days': None,
 'archive_status': None,
 'last_accessed_on': None,
 'etag': '0x8DA7555A1264549',
 'tags': None,
 'tag_count': None,
 'name': 'cas6-v0-u0kb/f2022-08-03/ocean-his-0022.nc',
 'size': 644660145,
 'type': 'file'}

[rsignell-usgs]

Hmm, this works:

ds.temp[383,-1,0,150]

but this doesn't, giving the incorrect header check error:

ds.temp[384,-1,0,150]

Would looking at the referenceFileSystem object provide insight?

Is there a good way to get the referenceFileSystem object from the intake catalog?

The intake catalog looks like:

cat['LiveOcean-Archive']

LiveOcean-Archive:
  args:
    consolidated: false
    storage_options:
      fo: s3://esip-qhub-public/LiveOcean/LiveOcean_reference.json
      remote_options:
        account_name: pm2
        skip_instance_cache: true
      remote_protocol: abfs
      target_options:
        anon: true
        skip_instance_cache: true
    urlpath: reference://
  description: LiveOcean Forecast Archive
  driver: intake_xarray.xzarr.ZarrSource
  metadata:
    catalog_dir: https://raw.githubusercontent.com/USGS-python/hytest-catalogs/main

[rsignell-usgs]

@martindurant oops I didn't see your message above. Will take a look.

[martindurant]

The way to get the filesystem:

fs = fsspec.filesystem("reference",
    "fo": "s3://esip-qhub-public/LiveOcean/LiveOcean_reference.json",
    "remote_options": {"account_name": "pm2",
          "skip_instance_cache": True},
    "remote_protocol": "abfs",
    "target_options": {
      "anon": True,
      "skip_instance_cache": True
    }
)

You can look at this object's .references dictionary, and interact with its internal filesystem fs.fss[None]. Nothing obvious appeared to me when I tried this. The filesize of the requested key is as big as the reference says it should be.

[peterm790]

@rsignell-usgs yes @martindurant is correct this is because some of the liveocean netcdf's have been updated after the references were generated. I haven't set this up to auto update in part because of struggles with kbatch but also thought it would be better for it to run on azure rather than aws.

I can quickly rerun it to update the references but it will break again when the files update tomorrow.

[rsignell-usgs]

@peterm790 , what are these "kbatch struggles" of which you speak?

[rsignell-usgs]

@parkermac, are the LiveOcean NetCDF files being overwritten periodically with newer versions?

[peterm790]

@peterm790 , what are these "kbatch struggles" of which you speak?

I suspect it might be a permissions problem but when I run a kbatch job it will just hang as 'running'. I have tried to issue a new API Token to solve this but that didn't seem to help, nor is the API token ever reflected as being used.

[martindurant]

It is conceivable that kerchunk should include a checksum/uuid with each referenced file to make sure you know when content changes. I'll put it on the list, but I don't see it as very high priority.