Tools for Hydrodynamic Model Output Extraction
A Python package providing utilities for processing, interpolating, and converting hydrodynamic ocean model NetCDF datasets.
There are many hydrodynamic modeling frameworks in use today by the oceanographic community. All models are output to a NetCDF format, but the structure, dimensions, variables, attributes, naming conventions, coordinate systems, and masking rules can all vary significantly, making it difficult for users to extract meaningful information from these complex systems.
This package attempts to support a common methodology for interacting with these datasets by abstracting the nuances of each modeling system into distinct modules and unifying them with a single API.
NOTE: To date, this package has been tested to work with National Ocean Service (NOS) Operational Forecast Systems (OFS) (e.g. CBOFS, DBOFS, NGOFS, etc.), however with minor tweaks it should also work with output from non-NOS OFS models.
The following ocean modeling frameworks are presently supported:
Regional Ocean Modeling System (ROMS)
"ROMS is a free-surface, terrain-following, primitive equations ocean model widely used by the scientific community for a diverse range of applications...In the horizontal, the primitive equations are evaluated using boundary-fitted, orthogonal curvilinear coordinates on a staggered Arakawa C-grid. The general formulation of curvilinear coordinates includes both Cartesian (constant metrics) and spherical (variable metrics) coordinates. Coastal boundaries can also be specified as a finite-discretized grid via land/sea masking. As in the vertical, the horizontal stencil utilizes a centered, second-order finite differences."
Description from https://www.myroms.org/
Finite-Volume Community Ocean Modeling System (FVCOM)
"FVCOM is a prognostic, unstructured-grid, finite-volume, free-surface, 3-D primitive equation coastal ocean circulation model developed by UMASSD-WHOI joint efforts. The model consists of momentum, continuity, temperature, salinity and density equations and is closed physically and mathematically using turbulence closure submodels. The horizontal grid is comprised of unstructured triangular cells and the irregular bottom is presented using generalized terrain-following coordinates."
Description from http://fvcom.smast.umassd.edu/fvcom/
Princeton Ocean Model (POM)
"POM is a sigma coordinate (terrain-following), free surface ocean model with embedded turbulence and wave sub-models, and wet-dry capability."
Description from http://www.ccpo.odu.edu/POMWEB/index.html
Hybrid Coordinate Ocean Model (HYCOM)
"The HYbrid Coordinate Ocean Model is a primitive equation ocean general circulation model that evolved from the Miami Isopycnic-Coordinate Ocean Model (MICOM) developed by Rainer Bleck and colleagues. Vertical coordinates in HYCOM remain isopycnic in the open, stratified ocean. However, they smoothly transition to z coordinates in the weakly-stratified upper-ocean mixed layer, to terrain-following sigma coordinate in shallow water regions, and back to level coordinates in very shallow water."
Description from https://www.hycom.org/attachments/067_overview.pdf
Additionally, a generic rectilinear module has been created to support
any model output whose coordinate system conforms to a rectilinear grid
and whose depth coordinates reflect standard depths/z-levels, however
this module does not support vertical interpolation.
- Support for ROMS, FVCOM, POM, HYCOM model output
- Interpolate staggered horizontal coordinates (i.e. ROMS Arakawa-C grid rho/eta/xi) to common coordinates before further processing
- Apply spatially-varying rotation angle to u/v current components to obtain true-north/true-east values before further processing
- Interpolate sigma (bathymetry-following) vertical coordinates to a given depth-below-surface, respecting the appropriate vertical transformation, if any
- Given an approximate target grid resolution, generate a regular grid definition conforming to the model domain's bounding box (or optionally to a predefined set of bounding rectangles), and output regular grid definition to CF-compliant NetCDF
-
All modules in
thyme/models/are written specifically for NOAA hydrodynamic ocean models and should be used with caution. The modules were written by referencing NOS model metadata, NOS model guidance, and roms, pom, fvcom, and hycom model documentation. -
To develop a custom module to support a new model, use one of the existing modules (fvcom.py, hycom.py, pom.py, rectilinear.py, or roms,py) as a template, and place the new python module file alongside the others under the
thyme/model/folder -
Development of a custom module may be required to support:
- Model output options specific to your organization
- Different variable names and dimensions
- Different vertical coordinates
- Different variable masks
- Different vertical or time varying horizontal masks
- Different date and time format
- If the model has already been converted to a regular or rectilinear grid
- If the model output's vertical coordinate system uses standard depth levels versus sigma
In order to convert model output files to a regular grid, an index file must be supplied at runtime. The purpose of the index file is to persist information that does not change between model runs in order to reduce the overall processing time per cycle run. The information stored in the index file includes the output grid definition and general metadata about the model itself. Index files for FVCOM-based models with a hybrid (generalized) coordinate system additionally store interpolated vertical coordinate values for each output grid cell.
When generating a grid definition, the user has a choice between using the full extent of the model's domain or supplying a shapefile containing one or more polygons defining subgrids (sub-regions/subdomains) to which the grid definition will conform. If no subgrid shapefile is specified, the resulting index file will define a regular grid (with NoData mask) matching the full model domain (extent). Otherwise, if a subgrid shapefile is specified, the resulting index file will define a regular grid (with NoData mask) matching the unified extent of all subgrid polygons that intersect the model domain, along with information identifying the grid cell ranges that correspond with each subgrid. Note that when supplying a subgrid shapefile, all subgrid polygons must be rectangular, congruent with each other, and adjacent to one another.
A subgrid index is intended to be used to subset the model output into smaller geographic areas (i.e. tiles), which in turn results in smaller output file sizes. Optionally, for a subgrid index file, the user may specify the name of an attribute field within the subgrid shapefile that uniquely identifies each subgrid (tile). If no field is specified, each polygon's FID value is used as the identifier. This identifier can be used to construct unique filenames.
Additionally, a land mask polygon shapefile can be supplied when generating an index file. If supplied, any output grid cells whose centroid intersects a land polygon will be masked in the final grid definition.
One index file must be created per ocean forecast system for each combination of target resolution and extent (whether using the model's full domain extent or subgrid definition).
Once a model index file is created, it can be reused indefinitely for that model/resolution/extent/land mask until any of those properties change. For example, if an FVCOM-based model has a hybrid (generalized) vertical coordinate system that is modified at some point (i.e., sigma values are changed), any associated index files will need to be regenerated using a new model output file in the updated format.
Generally, a new index file is required:
- For each new model
- For each desired output grid resolution
- For each desired set of subgrids
- If the subgrid polygons change
- If the subgrid attribute identifier changes
- If the land mask shapefile changes
- If the underlining model changes (e.g., new geographic extent, change to FVCOM sigma coordinates, etc.)
This codebase is written for Python 3 and relies on the following python packages:
- gdal
- netCDF4
- numpy
- scipy
- shapely
The GDAL Python bindings used by this package require system libraries to be
present, so it usually can't just be installed using pip install gdal.
We recommend installing GDAL either through a package manager (e.g.
conda, apt, yum, pacman) or by compiling from scratch.
Miniconda is probably
the easiest method.
Once gdal has been installed, thyme can be installed using pip:
pip install thymeTo generate a new index file for an FVCOM-based model using the default grid extent, a ~500 meter target resolution, and a shoreline shapefile defining land areas to be masked:
from thyme.model import fvcom
native_model_file = fvcom.FVCOMFile('/path/to/existing_fvcom_file.nc')
model_index_file = fvcom.FVCOMIndexFile('/path/to/new_index_file.nc')
try:
native_model_file.open()
model_index_file.open()
model_index_file.init_nc(native_model_file, 500, 'my_fvcom_model', '/path/to/shoreline_shapefile.shp')
finally:
model_index_file.close()
native_model_file.close()To generate a new index file for a ROMS-based model using a subgrid shapefile (with fieldname 'id' used to identify subgrid areas) and a ~300m target resolution (with no shoreline mask shapefile specified):
from thyme.model import roms
native_model_file = roms.ROMSFile('/path/to/existing_roms_file.nc')
model_index_file = roms.ROMSIndexFile('/path/to/new_index_file.nc')
try:
native_model_file.open()
model_index_file.open()
model_index_file.init_nc(native_model_file, 300, 'my_roms_model', None, '/path/to/subgrid_shapefile.shp', 'subgrid_id_fieldname')
finally:
model_index_file.close()
native_model_file.close()To interpolate u/v current components from a ROMS-based model to a
regular grid defined in an existing model index file, at a depth of 4.5
meters below surface, for time index 0, and store the resulting u/v
values in two numpy.ma.masked_array objects:
from thyme.model import roms
native_model_file = roms.ROMSFile('/path/to/existing_roms_file.nc')
model_index_file = roms.ROMSIndexFile('/path/to/existing_index_file.nc')
try:
native_model_file.open()
model_index_file.open()
(u_with_mask, v_with_mask) = native_model_file.uv_to_regular_grid(model_index_file, 0, 4.5)
# u_with_mask and v_with_mask now contain 2D numpy masked arrays
finally:
model_index_file.close()
native_model_file.close()This project uses pytest for unit testing.
To run the test suite:
pip install pytest
pytest -vv- Erin Nagel (UCAR), [email protected]
- Jason Greenlaw (ERT), [email protected]
This work, as a whole, is licensed under the BSD 2-Clause License (see LICENSE), however it contains major contributions from the U.S. National Oceanic and Atmospheric Administration (NOAA), 2017 - 2019, which are individually dedicated to the public domain.
This repository is a scientific product and is not official communication of the National Oceanic and Atmospheric Administration, or the United States Department of Commerce. All NOAA GitHub project code is provided on an "as is" basis and the user assumes responsibility for its use. Any claims against the Department of Commerce or Department of Commerce bureaus stemming from the use of this GitHub project will be governed by all applicable Federal law. Any reference to specific commercial products, processes, or services by service mark, trademark, manufacturer, or otherwise, does not constitute or imply their endorsement, recommendation or favoring by the Department of Commerce. The Department of Commerce seal and logo, or the seal and logo of a DOC bureau, shall not be used in any manner to imply endorsement of any commercial product or activity by DOC or the United States Government.
This software has been developed by the National Oceanic and Atmospheric Administration (NOAA)/National Ocean Service (NOS)/Office of Coast Survey (OCS)/Coast Survey Development Lab (CSDL) for use by the scientific and oceanographic communities.
CSDL wishes to thank the following entities for their assistance:
- NOAA/NOS/Center for Operational Oceanographic Products and Services (CO-OPS)
- Canadian Hydrographic Service (CHS)
- Teledyne CARIS