searching by nominal_resolution?

[rsignell-usgs]

I was hoping to search public CMIP6 datasets for all 10km datasets (or 25km datasets) using the searching for datasets approach:

# form query dictionary
query = dict(experiment_id=['historical', 'ssp245', 'ssp585'],
             table_id='Amon', nominal_resolution='10km',
             variable_id=['tasmax'])
# subset catalog and get some metrics grouped by 'source_id'
col_subset = col.search(require_all_on=['source_id'], **query)
col_subset.df.groupby('source_id')[['experiment_id', 'variable_id', 'table_id', 'nominal_resolution']].nunique()

but this didn't work. I guess that although this is part of the standard vocabulary, it's not part of the json, and I guess not part of the CSV files that the json was created from, right?

Could I just add it by modifying the code here? https://github.com/pangeo-data/pangeo-cmip6-cloud/blob/master/myconfig.py

[rsignell-usgs]

ping @jbusecke

[jbusecke]

Hey @rsignell-usgs, sorry for the delay. Just came back from vacation.

but this didn't work. I guess that although this is part of the standard vocabulary, it's not part of the json, and I guess not part of the CSV files that the json was created from, right?
Correct. Currently we only use these attributes as searchable 'facets' (i.e. columns in our catalog csv)

Could I just add it by modifying the code here? https://github.com/pangeo-data/pangeo-cmip6-cloud/blob/master/myconfig.py

I doubt it. a) I am currently completely revamping the CMIP6 data ingestion using pangeo-forge, and will still have to work out the cataloging (e.g. adding rows to the csv)

To me 'nominal resolution' is an "extra" facet, since it is not needed to uniquely identify a dataset (rather its a function of source_id). But I see that the official CMIP6 search page includes it as a filter criterion:

I think it would be worth a discussion over here (my newest attempt on automating the ingestion). Could you be so kind and raise an issue over there?

In the meantime maybe this workaround gets you to where you need to be:

import xarray as xr
from xmip.utils import google_cmip_col
col = google_cmip_col()

# create a smaller set with all relevant source_ids (You might have to modify this for your usecase)
cat = col.search(
    grid_label='gn',
    table_id='Amon', # make sure this matches the climate component you are looking for e.g. atmosphere here
)
df = cat.df
df_trimmed = df.groupby('source_id', group_keys=False).apply(lambda row: row.iloc[:1,:]) #pick only the first entry for each source_id (no matter what member, experiment, variable itis)

nominal_res_dict = {}
for store in df_trimmed['zstore']:
    ds = xr.open_dataset(store, engine='zarr')
    nominal_res_dict[ds.attrs['source_id']] = ds.attrs.get('nominal_resolution')
nominal_res_dict

Which got me:

{'ACCESS-CM2': '250 km',
 'ACCESS-ESM1-5': '250 km',
 'AWI-CM-1-1-MR': '100 km',
 'AWI-ESM-1-1-LR': '250 km',
 'BCC-CSM2-HR': '50 km',
 'BCC-CSM2-MR': '100 km',
 'BCC-ESM1': '250 km',
 'CAMS-CSM1-0': '100 km',
 'CAS-ESM2-0': '100 km',
 'CESM1-1-CAM5-CMIP5': '100 km',
 'CESM1-WACCM-SC': '250 km',
 'CESM2': '100 km',
 'CESM2-FV2': '250 km',
 'CESM2-WACCM': '100 km',
 'CESM2-WACCM-FV2': '100 km',
 'CMCC-CM2-HR4': '1x1 degree',
 'CMCC-CM2-SR5': '100 km',
 'CMCC-CM2-VHR4': '25 km',
 'CMCC-ESM2': '1x1 degree',
 'CanESM5': '500 km',
 'CanESM5-CanOE': '500 km',
 'E3SM-1-1': '50 km',
 'FGOALS-g3': '250 km',
 'FIO-ESM-2-0': '10000 km',
 'GFDL-CM4': '1x1 degree',
 'GFDL-CM4C192': '50 km',
 'GFDL-ESM2M': '250 km',
 'GFDL-ESM4': '1x1 degree',
 'GISS-E2-1-G': '250 km',
 'GISS-E2-1-G-CC': '250 km',
 'GISS-E2-1-H': '250 km',
 'GISS-E2-2-G': '250 km',
 'GISS-E2-2-H': '250 km',
 'HadGEM3-GC31-HM': '50 km',
 'HadGEM3-GC31-LL': '250 km',
 'HadGEM3-GC31-LM': '250 km',
 'HadGEM3-GC31-MM': '100 km',
 'ICON-ESM-LR': '250 km',
 'IITM-ESM': '250 km',
 'MCM-UA-1-0': '250 km',
 'MIROC-ES2H': '250 km',
 'MIROC-ES2L': '500 km',
 'MIROC6': '250 km',
 'MPI-ESM-1-2-HAM': '250 km',
 'MPI-ESM1-2-HR': '100 km',
 'MPI-ESM1-2-LR': '250 km',
 'MPI-ESM1-2-XR': '50 km',
 'MRI-AGCM3-2-H': '25 km',
 'MRI-AGCM3-2-S': '25 km',
 'MRI-ESM2-0': '100 km',
 'NESM3': '250 km',
 'NorCPM1': '250 km',
 'NorESM1-F': '250 km',
 'NorESM2-LM': '250 km',
 'NorESM2-MM': '100 km',
 'SAM0-UNICON': '100 km',
 'TaiESM1': '100 km',
 'UKESM1-0-LL': '250 km'}

in a reasonable time.

The key here is to not crawl the full catalog, but only load one dataset per source_id (see my comment above).