Addition of improved MODIS ice surface temperature observation converter program #858
Conversation
…r and move from cice subdirectory to MODIS subdirectory
…conventions, HDF file preprocessing to netcdf, and upload of example data files
There was a problem hiding this comment.
Pull Request Overview
This PR introduces an improved converter for MODIS ice surface temperature observations by cleaning up and standardizing the original converter program. Key changes include:
- Addition of a new Python script to convert MODIS HDF observation files to netCDF.
- Integration of geopandas-based CRS transformation to extract standardized lat/lon arrays.
- Adjusted output variable transformations and file renaming to support the new converter version.
Files not reviewed (4)
- observations/obs_converters/MODIS/MOD29E1D_to_obs.f90: Language not supported
- observations/obs_converters/MODIS/MOD29E1D_to_obs.nml: Language not supported
- observations/obs_converters/MODIS/work/input.nml: Language not supported
- observations/obs_converters/MODIS/work/quickbuild.sh: Language not supported
Comments suppressed due to low confidence (1)
observations/obs_converters/MODIS/shell_scripts/MOD29E1D_hdf_to_netcdf.py:44
- [nitpick] The output file is currently hardcoded to 'test.nc', which seems temporary. Please revert to using the intended production filename format.
post_file.to_netcdf('test.nc')
observations/obs_converters/MODIS/shell_scripts/MOD29E1D_hdf_to_netcdf.py
Outdated
Show resolved
Hide resolved
observations/obs_converters/MODIS/shell_scripts/MOD29E1D_hdf_to_netcdf.py
Outdated
Show resolved
Hide resolved
|
There is some super out of date, documentation for things that don't work, and slightly comical documentation in the MODIS directory on main: https://docs.dart.ucar.edu/en/latest/observations/obs_converters/MODIS/readme.html https://docs.dart.ucar.edu/en/latest/observations/obs_converters/MODIS/MOD15A2_to_obs.html Has the title 
okie dokie 
true/not true? Why have a bunch of docs for this unsupported tool. I'll go ahead an pull out https://docs.dart.ucar.edu/en/latest/observations/obs_converters/MODIS/readme.html in this request, and put MOD15A2_to_obs weirdness into a different issue. |
This is not referenced by any docs, seems to be out-of-date (does not list all modis docs)
|
@mollymwieringa Is observations/obs_converters/MODIS/MOD29E1D_to_obs.f90 replacing At the moment modis_ist_to_obs_netcdf.f90 is in this branch, but I believe this should be removed and your MODIS/MOD29E1D_to_obs.f90 is what people should be using? Cheers, |
Taken from comments in code. Added links from list of converters (not sure why we have two of these)
|
Added documentation page in a2c140c Viewable for the pull request: |
There was a problem hiding this comment.
This is a pretty large (47MB) file to commit, and it is output from the converter, so not needed. I'd like to take this out of the git history. This will mean rewriting the history on this branch.
There was a problem hiding this comment.
Should I delete this file in the pull request (as a new commit)? Or would you need me to leave it in order to take it out of the git history?
| import rioxarray as rxr | ||
| import xarray as xr | ||
| import geopandas as gpd | ||
| import numpy as np |
There was a problem hiding this comment.
Can you send along what versions of these packages you are using.
With the following:
pip freeze
affine==2.4.0
attrs==25.3.0
certifi==2025.4.26
click==8.1.8
click-plugins==1.1.1
cligj==0.7.2
GDAL==3.10.3
geopandas==1.0.1
nump==5.5.5.5
numpy==2.2.5
packaging==25.0
pandas==2.2.3
pyogrio==0.10.0
pyparsing==3.2.3
pyproj==3.7.1
python-dateutil==2.9.0.post0
pytz==2025.2
rasterio==1.4.3
rioxarray==0.19.0
shapely==2.1.0
six==1.17.0
tzdata==2025.2
xarray==2025.4.0
I get
File "rasterio/_base.pyx", line 312, in rasterio._base.DatasetBase.init
rasterio.errors.RasterioIOError: '../data/MOD29E1D_2000_02_24.EXAMPLE.hdf' not recognized as being in a supported file format.
gdalinfo can print file info ok:
gdalinfo ../data/MOD29E1D_2000_02_24.EXAMPLE.hdf
Driver: HDF4/Hierarchical Data Format Release 4
Files: ../data/MOD29E1D_2000_02_24.EXAMPLE.hdf
Size is 512, 512
Metadata:
ALGORITHMPACKAGEACCEPTANCEDATE=12-2004
...
There was a problem hiding this comment.
These are the versions from the environment I used to write that program:
rioxarray== 0.18.2
xarray==2025.3.1
geopandas==1.0.1
numpy== 1.26.4
There was a problem hiding this comment.
@hkershaw-brown note to self, need to use conda to get gdal hdf
(geo-env) hkershaw:shell_scripts > conda env export --from-history
name: geo-env
channels:
- defaults
dependencies: - gdal
- python=3.11
- rasterio
- rioxarray
- geopandas
- libgdal-hdf4
prefix: /Users/hkershaw/miniconda3/envs/geo-env
python MOD29E1D_hdf_to_netcdf.py MOD29E1D.A2000055.061.2020037150613.hdf 2020 07 08
|
Hi Helen, Yes, Best,
|
awesome thanks Molly, I've give those a spin. |
… into ice_converters
unnecessary addition to the pull request
… into ice_converters
|
@hkershaw-brown I went ahead and removed the old There are some additional commits for cleaning up some extraneous files that made it in during my committing-- I don't think you need to worry about them (they are not in the branch anymore). Let me know if I missed anything! |
| pre_file['lat'] = ({'x','y'}, lat) | ||
| pre_file['lon'] = ({'x','y'}, lon) |
There was a problem hiding this comment.
I think x,y is backwards here or in MOD29E1D_to_obs.f90 looking at the output netcdf file:
netcdf test {
dimensions:
y = 1906 ;
x = 1328 ;
variables:
double tsfc(y, x) ;
tsfc:_FillValue = NaN ;
double y(y) ;
y:_FillValue = NaN ;
double lat(x, y) ;
lat:_FillValue = NaN ;
double lon(x, y) ;
lon:_FillValue = NaN ;
double x(x) ;
x:_FillValue = NaN ;
DART/observations/obs_converters/MODIS/MOD29E1D_to_obs.f90
Lines 118 to 121 in 420be65
Hi Molly, Are you working on this branch today? If not I think I can rebase to rm the obs_seq.mod29e1d file from the commit it came in (a71f303) and leave the rest of the history alone. I'll need to force push so just checking that you don't have work in progress on this branch before I do that. Cheers, |
|
I can stay off this branch today! I'll fix the (x, y) allocation issue once you're done. |
I ended up just git rm obs_seq.mod29e1d. Rebasing or using git filter-repo was rewriting to much history to be worth worrying for this file (it is not crazy big). |
…n MOD29E1D hdf5 to netcdf reader
…df to netcdf reader
Description:
The converter program written by Yongfei Zhang for transcribing MODIS ice surface temperature observations has been cleaned and standardized; the module has also been moved from the
cicesubdirectory of DART'sobs_converterprograms to theMODISsubdirectory. Changes toquickbuild.shand theinput.nmlassociated with MODIS have been implemented to facilitate the new version of the converter,MOD29E1D_to_obs.An example preprocessing script to convert the MODIS IST hdf files to netcdf files with standardized lat/lon information and variables names is introduced in the
shell_scriptsdirectory. Note that this program requiresrio_xarray,xarray,numpy, andgeopandaspackages to ascertain the lat/lon information associated with the 4km EASE grid on which the observations are provided. There are other potential solutions to this preprocessing requirement that may be more general; they have not been explored for this pull request.Types of changes
Documentation changes needed?
Tests
MOD29E1D_to_obsbuilds successfully fromquickbuild.sh. The python preprocessing scriptMOD29E1D_hdf_to_netcdf.pyconverts the example daily observation file (from Earthdata's HTTPS file system for MODIS IST observations)data/MOD29E1D_2000_02_24.EXAMPLE.hdfto a netcdf file. When that file is fed into the namelist forMOD29E1D_to_obs, the program writes anobs_seq.mod29e1doutput file with expected observation, error, and qc values (O 100,000 observations total).Checklist for merging
Checklist for release
Testing Datasets