bug: OPAWS radar converter

[hkershaw-brown]

🐛 🐝

Describe the bug

OPAWS radar converter bug
Required by dart users, does not work as expected (or maybe at all)

1. Build the opaws exectuables
2. x.oban would run to completion with the test data & namelists from https://code.google.com/archive/p/opaws/downloads
3. The PARAMETERS namelist is not completely read, and the default values of some variables are used, which leads to the following non-fatal error for the namelist read, followed by the fatal error for "invalid value of namelist parameter: radar_data_format"

hkershaw@derecho3:/glade/derecho/scratch/hkershaw/DART/ForBrett/runs$ ./x.oban oban.nml.foray.orig 

PROGRAM OBAN:  running version Version 2.1.1:  Updated 24 June 2013 [DCD]                                                                                      
PROGRAM OBAN:  Namelist/Input file: oban.nml.foray.orig                                                                                                             

Problem reading namelist: PARAMETERS                                        Error:        5010
! INPUT NAMELIST values for the OBAN


 ======================= BEGIN PARAMETER DICTIONARY  ===========================

                                      nx -->        121
                                      ny -->        121
                                      nz -->          1
                                    xmin -->      0.0000    
                                    ymin -->      0.0000    
                                    zmin -->      0.0000    
                                      dx -->      2.0000    
                                      dy -->      2.0000    
....
 ======================== END PARAMETER DICTIONARY  ============================



Checking namelists for required information...

 ERROR -- invalid value of namelist parameter:  radar_data_format

Error Message

Problem reading namelist: PARAMETERS Error: 5010
ERROR -- invalid value of namelist parameter: radar_data_format

Which model(s) are you working with?

none, opaws converter

Version of DART

Not applicable
OPAWS "Version 2.1.1: Updated 24 June 2013 [DCD]"

Have you modified the code?

Yes I have modified the code and makefile to build on mac and Derecho
https://github.com/NCAR/opaws.git

Build information

Please describe:

1. Derecho & my mac
2. gfortran 12.4.0 & 13.2.0

[braczka]

Just filling in pieces of problem first reported reported by Yi Dai:

1. Documentation could use a better description of compiling options in "makefile" to allow for compiling to work with Derecho, for example. Example edits that work for derecho:

	# netCDF4 libs - you need to fill in the blanks
	#
	NETCDFINC = -I$NETCDF/include
	NETCDFLIB = -L$NETCDF/lib -lnetcdff
	
	# netCDF3 libs - you need to fill in the blanks
	#
	#NETCDFINC = -I/   /    /netcdf3-32-gfortran/include
	#NETCDFLIB = -L/   /    /netcdf3-32-gfortran/lib -lnetcdf 

#=============================================================================
	# Fortran and C compiler information - various configurations are setup, try and find one close
	# 
	# Generic gfortran is left to default.
	
	#=====>> Gfortran 
	# 
	FC   = gfortran -fallow-argument-mismatch -m64 -g -O0  -ffixed-line-length-132 -Wunused -Wuninitialized
	CC   = gcc
        CFLAGS = -m64 -c -g -I. -DLONG32 -DUNDERSCORE -DLITTLE -Wunused -Wuninitialized

2. Using the test data with the OPAWS download (test_opaws) does not work correctly during conversion process:
   a) The namelist variables radar_data_format, nfld, and nswp are not passed correctly into the code. Get the following error:

                       Radar data format -->       -999
                     No. of radar fields -->          0
                            No. of files -->          0
 
 ======================== END PARAMETER DICTIONARY  ============================
 
 
 
Checking namelists for required information...
 
 ERROR -- invalid value of namelist  parameter:  radar_data_format

This can be corrected if manually entering desired namelist values in oban_namelist.f90 file and recompiling. Fix is not ideal.

integer :: radar_data_format = 1 
integer :: nfld = 2                        
integer :: nswp = -1 

b) Path to the test foray radar files is incorrect within namelist file (oban.nml.foray)

Need to replace '../../KCYS_foray/ncswp_*.nc' with './foray_data/ncswp_*.nc'

[hkershaw-brown]

Python version of OPAWS: https://github.com/louiswicker/pyOPAWS
This links to https://github.com/hhuangwx/opaws which is apparently an export to Github of the
google code project https://code.google.com/archive/p/opaws/

[hkershaw-brown]

NCAR fork here: https://github.com/NCAR/opaws

[braczka]

@hkershaw-brown Good find --- at least there is a github equivalent to the the google project code. It looks like the Python version of OPAWS is the most recent and maintained version, however, it also looks like it has lost capabilities to convert the DORADE formatted radar files....... only converts the CFradial.

Maybe I missed it, did the google code project provide and forward links to the github and pyOPAWS versions?

[braczka]

... but pyOPAWS appears to convert the NEXRAD Level II radar data format directly to obs_seq -- which is what Yi Dai is looking for. Alternative to NEXRAD Level II conversion to DORADE (using LROSE), then DORADE conversion to obs_seq using OPAWS.

[hkershaw-brown]

@hkershaw-brown Good find --- at least there is a github equivalent to the the google project code. It looks like the Python version of OPAWS is the most recent and maintained version, however, it also looks like it has lost capabilities to convert the DORADE formatted radar files....... only converts the CFradial.

Maybe I missed it, did the google code project provide and forward links to the github and pyOPAWS versions?

No it didn't give a link, I just poked around github and found Lou Wicker's pyOPAWS.

[hkershaw-brown]

@braczka the namelist read is failing is because oban.nml.foray & oban.nml.sweep given in the test_data directory have entries that are not in the code.

hkershaw@derecho3:/glade/derecho/scratch/hkershaw/DART/ForBrett/runs$ diff oban.nml.foray ../test_data/test_opaws/oban.nml.foray
78a79,82

use_clutter_mask = .false. ! .true. if clutter mask based on ppi stats should be implemented
cm_min_refl_avail = 75.0 ! minimum reflectivity availability (%) for clutter mask
cm_min_refl = -5.0 ! minimum reflectivity (dBZ) for clutter mask
cm_max_refl_sd = 10.0 ! maximum reflectivity standard deviation (dBZ) for clutter mask
79a84,86
cm_min_vel_sd = 7.5 ! minimum velocity standard deviation (m/s) for highway detection in clutter mask
cm_max_vel = 2.0 ! maximum velocity absolute magnitude (m/s) for standard clutter detection in clutter mask
cm_max_vel_sd = 2.0 ! maximum velocity standard deviation (m/s) for standard clutter detection in clutter mask
81a89,90
cm_refl_fname_ppi = "REF" ! name of reflectivity field in "ppi stats" file
cm_vel_fname_ppi = "VEL" ! name of velocity field in "ppi stats" file
102a112
ncgen_command = 'ncgen', ! path/executable for local "ncgen" command

Maybe the test data is not for a different version of the code, but user beware using this code.

[hkershaw-brown]

updated bug description with the problem

[braczka]

updated bug description with the problem

Thanks @hkershaw-brown -- that was a misleading error code, as radar_data_format namelist variable was seemingly unrelated to core issue of mismatch between namelist values ( within oban.nml.sweep and oban.nml.foray) and what the source code supports, which is concerning. The source code does not include max/min reflectivity and velocity namelist options.

Recommend the removal of unsupported namelist values as mentioned above, before executing conversion. This provides a different answer (radar obs_seq file is different, as different namelist values are passed into code) than previous fix suggestion of hardcoding namelist values in oban_namelist.f90. The latter approach is not recommended.

Bigger picture is that google archive version is a decent test case, but is deprecated, and newer pyOPAWS is probably better alternative for most radar data applications.

[hkershaw-brown]

Added test_data directory with fixed oban.nml.foray namelist:
https://github.com/NCAR/opaws
https://github.com/NCAR/opaws/tree/master/test_data

[hkershaw-brown]

Opening re standup conversions, on dart documentation for opaws.

[hkershaw-brown]

Note the original issue reported by the user of OPAWS was a compilation error.
The code read_dorade.c includes malloc.h (and the file malloc.h Copyright (c) 1999 Apple Computer, Inc. All rights reserved.)
This is why the code would not compile on Derecho (Linux)

There is no need to include this or have the malloc.h on Linux or mac. This is fixed in https://github.com/NCAR/opaws e677ddcbbff78f4e7f91b92829d7d69c940e0bf4