POD development: how to add a new variable

[bitterbark]

What is the recommended path for developers who need to use a new variable?

• I see the directory data/ has the historic fieldlist_*.jsonc and also a bunch of tables. It seems that adding to
  fieldlist_NCAR is noticed by the framework. Do we need to also add to the tables? Is it our responsibility to add to other models fieldlists?

• Is there pressure level extraction? If so, how to access it?
  In the POD I'm implementing, I need geopotential height at 500mb. There is
  already a zg = geopotential_height, defined in fieldlist_NCAR.jsonc
  as follows:
  "Z3": {
  "standard_name": "geopotential_height",
  "units": "m",
  // note: 4d name is 'Z3' but Z500 = height at 500 mb, etc.
  "scalar_coord_templates": {"plev": "Z{value}"},
  "ndim": 4
  },
  It looks like the scalar_coord_templates should recognize an inputdata/model
  netcdf file named Z500, and it looks like this was used in the EOF pod.
  Emulating its settings.jsonc results in errors (it is depricated, right?)

• In the pod's settings.jsonc file, if we have non-standard dimensions in the file, how do we describe them in the dimensions specifications?

• My specific error
  When I just asked for varlist "zg", "standard_name" : "geopotential_height", "dimensions": ["time", "lat", "lon"]
  I get the error
  WARNING: Deactivating <Update with current development efforts #2.blocking_neale:zg> due to translation failure ("Queried standard name 'geopotential_height' with an unexpected set of axes frozenset({'X', 'T', 'Y'}) not in convention 'NCAR'.").
  I don't understand what the frozenset axes are or why they don't correspond to the dimensions I specify.

Since it did't work with the (3d) zg = "standard_name" : "geopotential_height", I made a new variable:
"zg500": "standard_name" : "geopotential_height_500mb",

It seems to get farther:
Querying <Z500 (=geopotential_height_500mb)>
No data found for <Z500 (=geopotential_height_500mb)>.
Request for <#2.blocking_neale:zg500 (=Z500) @ 1day> failed; finding alternate vars.
No alternates available for <#2.blocking_neale:zg500>.
WARNING: Execution of POD blocking_neale couldn't be completed:

The file in question is named inputdata/model/blocking_cam5/day/blocking_cam5.Z500.day.nc
float Z500(time, lat, lon) ;
Z500:units = "m" ;
Z500:long_name = "Geopotential Z at 500 mbar pressure surface" ;
Z500:cell_methods = "time: mean

I'm attaching the following files, let me know if you want to look at something else!
input_dani.jsonc.txt
settings.jsonc.txt
out.txt

This will go on my list of improved documentation for new developers based on my experiences.
Thanks!

[jkrasting]

Thanks Dani. I was wondering the same thing. @tsjackson-noaa - do we have documentation on this specific task?

[jkrasting]

I am also facing this same problem now that @wrongkindofdoctor's static variable fix (#185) gets us past the static field issue in @chiaweh2's tropical sea level POD:

### Framework: initialize CESM2_omip1_r1i1p1f1_gn
WARNING: Deactivating <#2.tropical_pacific_sea_level:zos> due to translation failure ("Standard name 'sea_surface_height_above_geoid' not defined in convention 'NCAR'.").

[tsjackson-noaa]

Hi @bitterbark;

1. The CMOR tables were moved to /data as part of PR Include CMOR tables as subtree, not submodule #108; the rationale for doing so is described there. The fieldlist_* files were moved to /data from /src because they were better described as "supporting data for the framework" instead of "framework source code"; there's no ironclad reason to have them in either location.
   Updating the fieldlist_* files is still a manual process (e.g. Add fieldlist entries for new variables requested by Gen 2 PODs #134, which addresses the issue @jkrasting mentioned above), but a more scalable and systematic way of dealing with this information is definitely desirable. The limiting factor is the availability of up-to-date and comprehensive variable name information from the modeling centers we want to support (e.g. as would be used by CMOR).

2. The fieldlist_* and POD settings files specify different information. The entry from the fieldlist you quoted is telling the framework how to translate a POD's request for zg on a pressure to NCAR's convention: NCAR's variable name for "zg at X mb" is 'Z' + str(X).

The request for a variable on a pressure level is still made in a varlist entry in the POD's settings file, using the scalar_coordinates attribute. This use of the scalar_coordinates attribute to denote array slices is in keeping with the CF conventions.

This is what is done in the EOF_500hPa POD: we request zg on a pressure level described by the dimension coordinate plev taking the value 500 (in units of hPa, which were specified in plev's definition.) Because we haven't implemented conversion from hybrid sigma coordinates yet, this POD uses a workaround where it defines zg_hybrid_sigma (i.e., geopotential_height on hybrid sigma levels lev) as an alternate to be used if zg (i.e., geopotential_height on pressure levels plev) is not found.

3. You'll need to provide more context as to what's meant by "non-standard dimensions." For the purposes of preprocessing, we need to identify horizontal, vertical and time coordinates. The requirements we currently impose on these are documented here; obviously we need better support for other coordinates (hybrid sigma, tripolar grids for ocean PODs, etc...)
   Variables that depend on coordinates beyond those (e.g. wavelength, or a land use category) can be handled by defining the corresponding dimension coordinate with an axis type of OTHER, but this functionality hasn't been tested since it hasn't been needed by any POD so far.

4. See second paragraph of response to # 2. The framework doesn't interpret the name z500 in the settings file as a 500 mb level; PODs are free to give arbitrary names to their variables. The error arises because the framework is still interpreting your request as being for the the full 3D geopotential_height variable, but you've only listed 2 space + 1 time dimensions. The way to fix this is by specifying the level you want in the scalar_coordinate attribute, as described above.

[wrongkindofdoctor]

@drewmresnick There is no 'Z500' variable file in the daily QBOi.EXP1.AMIP.001 test data. There is a Z250 variable file. The monthly data contains the full 4D Z field (Z3) if you want to try to extract a specific level.

[drewmresnick]

@wrongkindofdoctor I have been working with the file for z250, not z500 - unless you are seeing somewhere that I wrote z500? Although at least based off my understanding of the error (and I could be wrong) is the issue not arising from querying of the temperature variable (T)?

[wrongkindofdoctor]

@drewmresnick My bad--I read down too far and the variable settings caught my eye. There is no 'Z/plev' dimension in the t250, z250, or any of the other files for the variables listed in your settings file, so this may be causing the error. I suggest omitting the 'plev' block in the data section (or add a 4-D variable with a Z-axis to analyze if your POD works on them).

While not the source of this specific error, the settings.jsonc file in the branch you linked to shows z500, so you may need to update your file. If your local branch is up-to-date, you'll need to push your changes to the remote so we can view them.

[drewmresnick]

Apologies - I thought I had pushed the changes. Updated settings file can be found here.

[wrongkindofdoctor]

@drewmresnick I'm not sure which standard you are using for the run, but assuming that you left the default setting as 'CESM' in your default_test.jsonc file (or whatever file you are using to run the framework), the settings.jsonc file needs to be corrected to reflect this format. The 'T250' variable is also missing from the fieldlist_CESM.jsonc file referenced by the framework, so it needs to be added. I have a branch with these changes that made it past the errors you listed that you can reference here.

Note that your flow_dep_diags branch is pretty far behind the NOAA-GFDL/develop branch. The fieldlist format in the develop branch supports the variable name format in the current version of your settings.jsonc file (i.e., you can list the variable name directly instead of using the varname : "[name]" syntax supported by the older framework version of the file in your flow_dep_diags branch. You would have to add T250 and Z250 (as well as any other variables not included that you want to test) to the appropriate field list file(s) (fieldlist_NCAR.jsonc in this example).

Since you'll need to sync your branch with the develop branch before we review the PR anyway, I recommend the following steps:

1. make a backup copy of your repo on your machine
2. update your remote and local develop branches
3. rebase flow_dep_diags onto the develop branch
4. fix any formatting issues in your settings.jsonc file, and make sure the variable names match the format you are using by referencing the appropriate fieldist file
5. add Z250 and T250 to the appropriate fieldlist file