CESM catalog generation- input path does not exist

[aradhakrishnanGFDL]

Opening the issue referenced here in MDTF as this relates to the noaa-gfdl/catalogbuilder

From @emaroon

GFDL catalog builder option: I tried using the GFDL catalog builder today, but after customizing the output_path_template, I cannot get past an error that says "Input path does not exist. Adjust configuration" despite triple checking the input_path.

[emaroon]

Thanks for opening the issue over here @aradhakrishnanGFDL! And thank you for helping me figure this one out since my POD development is stuck until I get catalogs working.

Since posting yesterday, I discovered that I had input_path correct in the config.yaml, but it was incorrect in the python script. So, once I corrected input_path in the python script to be identical to input_path in config.yaml, the catalog python script completed! Though, then it produced a csv file with only the header titles and no information on the files. The json looks similarly sparse.

Here's the setup I've got. I started from an example script that was in the docs for the gfdl catalog builder:

import catalogbuilder
from catalogbuilder.scripts import gen_intake_gfdl
import sys, os 

#This is an example call to run catalog builder using a yaml config file.
package_dir = os.path.dirname(os.path.abspath(__file__))
configyaml = '/home1/emaroon/mdtf/gfdlcatalogbuilder/config.yaml'
input_path = "/adhara_a/emaroon/mdtf/inputdata/model/cesm2/b.e21.BHIST.f09_g17.CMIP6-historical.001/mon"
output_path = "/home1/emaroon/mdtf/gfdlcatalogbuilder/catalog"

def create_catalog_from_config(input_path=input_path,output_path=output_path,configyaml=configyaml):
    csv, json = gen_intake_gfdl.create_catalog(input_path=input_path,output_path=output_path,config=configyaml)
    return(csv,json)

if __name__ == '__main__':
    create_catalog_from_config(input_path,output_path,configyaml)

Here's config.yaml as I've modified it to fit where I've put model output, again starting from what you've provided:

headerlist: ["activity_id", "institution_id", "source_id", "experiment_id",
                  "frequency", "realm", "table_id",
                  "member_id", "grid_label", "variable_id",
                  "time_range", "chunk_freq","platform","dimensions","cell_methods","standard_name","path"]

output_path_template: ['NA','NA','NA','NA','NA','source_id','NA','frequency']

output_file_template: ['NA','NA','NA','NA','experiment_id','member_id','variable_id','frequency']

#######################################################

input_path: "/adhara_a/emaroon/mdtf/inputdata/model/cesm2/b.e21.BHIST.f09_g17.CMIP6-historical.001/mon/"
output_path: "/home1/emaroon/mdtf/gfdlcatalogbuilder/" # ENTER NAME OF THE CSV AND JSON, THE SUFFIX ALONE. e.g catalog (the builder then generates catalog.csv and catalog.json. This can also be an absolute path)

I'm using CESM output on my local server. Here's the directory and list of the five files that I'm trying to get into a catalog:

(gfdlcatalog) [emaroon@adhara mon]$ pwd
/adhara_a/emaroon/mdtf/inputdata/model/cesm2/b.e21.BHIST.f09_g17.CMIP6-historical.001/mon

(gfdlcatalog) [emaroon@adhara mon]$ ls
b.e21.BHIST.f09_g17.CMIP6-historical.001.SALT.mon.nc  b.e21.BHIST.f09_g17.CMIP6-historical.001.SHF.mon.nc  b.e21.BHIST.f09_g17.CMIP6-historical.001.TAREA.mon.nc
b.e21.BHIST.f09_g17.CMIP6-historical.001.SFWF.mon.nc  b.e21.BHIST.f09_g17.CMIP6-historical.001.SST.mon.nc

Here's what the catalog.csv looks like:

activity_id,institution_id,source_id,experiment_id,frequency,realm,table_id,member_id,grid_label,variable_id,time_range,chunk_freq,platform,dimensions,cell_methods,standard_name,path

I'm assuming that the output_path_template or output_file_template are wrong? What am I not understanding here? Thanks for your help!

[aradhakrishnanGFDL]

Hi @emaroon, your config file looks perfect. Ofcourse, since the catalog builder relies on a couple more things that the input directory does not provide, it didn't work straight away.

I may have to suggest some workarounds before making things less hardwired.
I was able to generate a sample catalog with the following changes

1- Checkout 89-cesm branch which is in an open PR. I originally had some minor fixes for the only cesm test case we used with @bitterbark's help. I will merge the PR once your tests pass too.

2- We don't have a way to get realm from the directory structure in this case. If it's in the netcdf header, we can lean on that in a newer PR. I also have some assumptions around "time-series" and "time-averages" as the structure is different for GFDL, and we only enable "ts".

So, here is the suggested directory structure for your run, as a workaround. (ts/mon/ocean)

/adhara_a/emaroon/mdtf/inputdata/model/cesm2/b.e21.BHIST.f09_g17.CMIP6-historical.001/ts/mon/ocean/

3- Now, lets adjust the yaml config

config can be found here

output_path_template https://github.com/aradhakrishnanGFDL/sandbox/blob/cats-cam/cats/emaroon/cesm-template.yaml#L31
output_file_template https://github.com/aradhakrishnanGFDL/sandbox/blob/cats-cam/cats/emaroon/cesm-template.yaml#L36

4- Here is the notebook I used to test

https://github.com/aradhakrishnanGFDL/canopy-cats/blob/main/notebooks/emaroon_sample.ipynb

(note that the standard name is NA, but if the netcdf header has this or the long_name, then the builder should populate it. I used a quick and easy dummy dataset)

Let me know if you're able to get past the initial hurdles with these steps. Thanks for trying this out!

[emaroon]

Success! I think! Following your detailed instructions and using the cesm-89 branch, I was able to get a catalog to build from CESM output! The five files appear in the csv with most of the fields populated. And, it looks like when I try to run MDTF with the catalog, that I reach a new issue. Still trying to figure this next error out, but I suspect that the catalog is now not the problem. (Though I could be wrong.) If I cannot figure it out with a little more poking on my own, I'll post on that issue back on the MDTF github.

The only question I have is that it looks like some of the fields that I set in the output_file_template (like member_id) do not populate in the csv file, even when I set them in output_file_template - but it looks like that in your notebook's example csv output too, so I'm guessing that I shouldn't worry about that?

Thanks again for your help!

[aradhakrishnanGFDL]

Good catch! I’ll make a note of this benign issue. You’re right, this can’t affect the current MDTF tests with CESM data.