s3 object storage featureCollection collection spec update

[mylesmc123]

Using TDS 5.6, I would like to update an existing FeatureCollection's collection spec attribute - where the data's location is defined. I've tried a couple different patterns and failed with various errors.

This is an attempt to update block storage to object storage data locations. The working block storage pattern looks like this:

<collection spec="/cbofs/**/.*nos.cbofs.fields.*\.nc" olderThan="5 min"/>

I have tried updating using the object storage location with the following syntax:

<collection spec="cdms3://object.site.com/datapub-prod?coops/access/cbofs/**/.*nos\.cbofs\.fields.*\.nc#delimiter=/" olderThan="5 min"/>

This does load the catalog and allows me to access the feature collection's landing page:

However, on Selection of the service OpenDAP

I ge tthe following error:

Error {
code = 500;
message = "Cannot read field "proto" because "localState" is null";
};

I have deleted the cache file to attempt to remediate the error, but no luck, the fmrc.log gives me multiple 'FileNotFound' errors:

Caused by: java.util.concurrent.ExecutionException: java.io.FileNotFoundException: cdms3:object.site.com/datapub-prod?coops/access/cbofs/2024/01/nos.cbofs.fields.n005.20240111.t00z.nc#delimiter= (No such file or directory)

However, I these files do exist in the bucket, and I can add them via a datasetScan rather than FeatureCollection.

Any ideas for getting this working?

[lesserwhirls]

I do not believe FeatureCollections work with object storage at this point (although I believe some work was done while I away from Unidata). I'd be happy to take a look over the next week if you know of a public bucket that is available for use.

[mylesmc123]

Hey @lesserwhirls , Thanks for picking this up, it is much appreciated.

Based on the src code at https://github.com/Unidata/tds/blob/main/tds/src/test/content/thredds/tds-s3.xml

I see an aggregation in there for GOES16 netCDF, could we use that to also test featureCollections? AWS Explorer URL: https://noaa-goes16.s3.amazonaws.com/index.html#ABI-L1b-RadC/

[lesserwhirls]

I have not explored it myself, but it's possible the code needed to do a forecastModelRunCollection NcML aggregation works with object stores (maybe not with scanFmrc), whereas using a featureCollection element with a <collection spec...> child does not.

[mylesmc123]

For this featureCollection I am using a forecastModelRunCollection via the featureCollection attrs:

featureType="FMRC"
harvest="true"

It also has a netcdf tag setting up some variables to have coordinates and standard names compliant with CF-1.6.

In the case that <collection spec...> does not work with FMRC types of featureCollections from object storage, is there any alternative method?

I just need to search out netcdf files recursively and scan periodically for new files.

[lesserwhirls]

At this point I don't think it is possible. I'll likely need to make a few code modifications to both netCDF-Java and the TDS get it working. Is there a public bucket with the data of interest that I could use for development? If so, would you mind sharing your NcML modifications you use to make the data CF compliant?

[mylesmc123]

The bucket I'm using isn't directly public yet. However, I have found an AWS public bucket that serves the same data in a slightly different folder structure: https://noaa-ofs-pds.s3.amazonaws.com/index.html

My bucket just has this dataset archived further back in time and is separated into year/month/day folders, whereas this public bucket has flatter folders of YYYYMMDD.

The catalog entry for a featureCollection using this public data, as we wish to use it, should be as attached in the catalog xml snippet. I have updated the regex on the feature spec attribute to use the public bucket's data.

<dataset name="CBOFS Forecast Aggregation" ID="CBOFS">
    <metadata inherited="true">
        <documentation xlink:href="http://tidesandcurrents.noaa.gov/ofs/cbofs/cbofs.html"
                       xlink:title="Chesapeake Bay Operational Forecast System Home Page"/>
        <authority>gov.noaa.nos.co-ops</authority>
        <serviceName>aggServices</serviceName>
        <dataType>Grid</dataType>
        <dataFormat>NetCDF</dataFormat>
    </metadata>
    <featureCollection name="Aggregated CBOFS Fields Forecast" featureType="FMRC"
                       harvest="true" path="model-cbofs-agg">
        <!--
                             Inside the featureCollection, but outside the protoDataset, we define the NcML that happens
            before the aggregation.  To get aggregated, we must have grids, so we turn the bed params
            into grids by giving them a psuedo coordinate in Z.  If we don't do this, they will not be
            aggregated.
        -->
        <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2">
            <attribute name="title" value="CBOFS - Chesapeake Bay Operational Forecast System - NOAA CO-OPS - POM"/>
            <attribute name="summary" value="NOAA's National Ocean Service (NOS) has upgraded its Chesapeake Bay Operational Forecast System (CBOFS). The new higher resolution CBOFS is now based on a three-dimensional ROMS model that runs on NOAA's High Performance Computers (HPC). In addition to providing water level nowcast and forecast guidance, the new CBOFS now also provides currents, water temperature and salinity as well as interpolated winds from National Weather Service products. CBOFS runs four times per day and generates 6-hour nowcasts and 48-hour forecast guidance. CBOFS products include time series graphics at station locations and aerial animations of the whole Chesapeake Bay for all five parameters (wind, water level, currents, temperature and salinity)." />

            <!--Change conventions from COARDS to CF-1.6 -->
            <attribute name="Conventions" value="CF-1.6"/>
            <!-- Curvilinear grids need "coordinates" attribute for each data variable-->
            <!--The "positive" attribute is only for the z coordinate variable, so remove it from depth and elevation-->
            <variable name="zeta">
                <attribute name="coordinates" value="lon_rho lat_rho ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="sea_surface_height"/>
                <remove name="positive" type="attribute"/>
            </variable>
            <variable name="s_rho">
                <attribute name="standard_name" value="ocean_s_coordinate"/>
                <attribute name="positive" value="up"/>
            </variable>
            <variable name="s_w">
                <attribute name="standard_name" value="ocean_s_coordinate"/>
                <attribute name="positive" value="up"/>
            </variable>
            <variable name="mask_rho">
                <attribute name="coordinates" value="lon_rho lat_rho"/>
            </variable>
            <variable name="f">
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="coordinates" value="lon_rho lat_rho"/>
            </variable>
            <variable name="h">
                <attribute name="coordinates" value="lon_rho lat_rho"/>
                <attribute name="missing_value" type="double" value="2.0"/>
                <attribute name="coordinates" value="lon_rho lat_rho"/>
                <attribute name="missing_value" type="double" value="2.0"/>
                <attribute name="standard_name" value="sea_floor_depth"/>
            </variable>
            <variable name="mask_u">
                <attribute name="coordinates" value="lon_u lat_u"/>
            </variable>
            <variable name="mask_v">
                <attribute name="coordinates" value="lon_v lat_v"/>
            </variable>
            <variable name="Uwind">
                <attribute name="coordinates" value="lon_rho lat_rho ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="grid_eastward_wind"/>
            </variable>
            <variable name="Vwind">
                <attribute name="coordinates" value="lon_rho lat_rho ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="grid_northward_wind"/>
            </variable>
            <variable name="Pair">
                <attribute name="coordinates" value="lon_rho lat_rho ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="surface_air_pressure"/>
            </variable>
            <variable name="temp">
                <attribute name="coordinates" value="lon_rho lat_rho s_rho ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="sea_water_temperature"/>
            </variable>
            <variable name="salt">
                <attribute name="coordinates" value="lon_rho lat_rho s_rho ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="sea_water_salinity"/>
            </variable>
            <variable name="u">
                <attribute name="coordinates" value="lon_u lat_u s_rho ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="grid_eastward_sea_water_velocity"/>
            </variable>
            <variable name="v">
                <attribute name="coordinates" value="lon_v lat_v s_rho ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="grid_northward_sea_water_velocity"/>
            </variable>
            <variable name="w">
                <attribute name="coordinates" value="lon_v lat_v s_w ocean_time"/>
                <attribute name="missing_value" type="double" value="0.0"/>
                <attribute name="standard_name" value="vertical_sea_water_velocity"/>
            </variable>
        </netcdf>
        <!-- Using a public bucket -->
        <collection spec="cdms3://noaa-ofs-pds?cbofs.*/cbofs.*\.fields\..*\.nc#delimiter=/" olderThan="5 min"/>
        <update recheckAfter="15 min" />
        <fmrcConfig regularize="true" datasetTypes="Best"/>
    </featureCollection>
</dataset>

[lesserwhirls]

Great, thank you! I'll use this to see what is needed to support FMRC featureCollections with object storage.

[mylesmc123]

Hello @lesserwhirls , I was just wanting to give this issue a nudge post shutdown and thanksgiving break. It looks to be an important feature for migrating NCEI's THREDDS instances to the cloud. It also looks to be very important for us to be able to migrate our ERDDAP instance too, as we have many dependencies on THREDDS serving data to our ERDDAP directly. I thank you and Unidata developers for all the support.

[lesserwhirls]

Thank you for checking-in. This is a feature I want to have in place for the next release (of which I don't have an ETA yet).