GDAS v17 fails in debug mode

[DusanJovic-NOAA]

When I run GDAS v17 configuration on WCOSS2 using a debug build, model fails with this error:

WARNING from PE     0: Unused line in INPUT/MOM_input : V_IC_VAR = "v"

[nid003770.cactus.wcoss2.ncep.noaa.gov](http://nid003770.cactus.wcoss2.ncep.noaa.gov/) 11206: forrtl: error (73): floating divide by zero                                                                                                                                                                                        
Image              PC                Routine            Line        Source            
libpnetcdf.so.4.0  00001506342B62EC  for__signal_handl     Unknown  Unknown
libpthread-2.31.s  00001506339A6910  Unknown               Unknown  Unknown
ufs_model          000000000EDBF3DD  mom_oda_incupd_mp         612  MOM_oda_incupd.F90
ufs_model          000000000DB40B79  mom_diabatic_driv        1224  MOM_diabatic_driver.F90
ufs_model          000000000DAD7D43  mom_diabatic_driv         407  MOM_diabatic_driver.F90
ufs_model          000000000C04372B  mom_mp_step_mom_t        1665  MOM.F90
ufs_model          000000000C023495  mom_mp_step_mom_          998  MOM.F90
ufs_model          000000000BF6A45C  mom_ocean_model_n         638  mom_ocean_model_nuopc.F90
ufs_model          000000000BEEAF1C  mom_cap_mod_mp_mo        1911  mom_cap.F90
ufs_model          0000000000CE912F  _ZNK5ESMCI13Metho         377  ESMCI_MethodTable.C

[DeniseWorthen]

@DusanJovic-NOAA Can you check your MOM_input for the variable names for the module MOM_oda_incupd namelist and then compare it w/ the names of the variables in your mom6 increment file (usually named mom6_increment.nc).

[DusanJovic-NOAA]

@DusanJovic-NOAA Can you check your MOM_input for the variable names for the module MOM_oda_incupd namelist and then compare it w/ the names of the variables in your mom6 increment file (usually named mom6_increment.nc).

Yes, I'll check it.

[DusanJovic-NOAA]

@DusanJovic-NOAA Can you check your MOM_input for the variable names for the module MOM_oda_incupd namelist and then compare it w/ the names of the variables in your mom6 increment file (usually named mom6_increment.nc).

The variable names in the namelist are the same as the names in increment file

[DusanJovic-NOAA]

I think the block of code where 'divide by zero' error happens should be moved inside the if (G%mask2dT(i,j) == 1) then
block, something like this:

diff --git a/src/ocean_data_assim/MOM_oda_incupd.F90 b/src/ocean_data_assim/MOM_oda_incupd.F90
index f174bf14a..d708c1a6a 100644
--- a/src/ocean_data_assim/MOM_oda_incupd.F90
+++ b/src/ocean_data_assim/MOM_oda_incupd.F90
@@ -599,19 +599,19 @@ subroutine apply_oda_incupd(h, tv, u, v, dt, G, GV, US, CS)
   tmp_val1(:) = 0.0
   tmp_t(:,:,:) = 0.0 ; tmp_s(:,:,:) = 0.0 ! diagnostics
   do j=js,je ; do i=is,ie
-    ! account for the different SSH
-    sum_h1 = 0.0
-    do k=1,nz
-      sum_h1 = sum_h1+h(i,j,k)
-    enddo
-    sum_h2 = 0.0
-    do k=1,nz_data
-      sum_h2 = sum_h2+h_obs(i,j,k)
-    enddo
-    do k=1,nz_data
-      tmp_h(k) = ( sum_h1 / sum_h2 ) * h_obs(i,j,k)
-    enddo
     if (G%mask2dT(i,j) == 1) then
+      ! account for the different SSH
+      sum_h1 = 0.0
+      do k=1,nz
+        sum_h1 = sum_h1+h(i,j,k)
+      enddo
+      sum_h2 = 0.0
+      do k=1,nz_data
+        sum_h2 = sum_h2+h_obs(i,j,k)
+      enddo
+      do k=1,nz_data
+        tmp_h(k) = ( sum_h1 / sum_h2 ) * h_obs(i,j,k)
+      enddo
       ! get temperature increment
       do k=1,nz_data
         tmp_val2(k) = CS%Inc(1)%p(i,j,k)

[DeniseWorthen]

I wonder why you're getting this debug failure? The new (uncommitted yet) SFS test also uses the ODA and I've tested it extensively in debug both before and after the MOM6 update.

[DusanJovic-NOAA]

I wonder why you're getting this debug failure? The new (uncommitted yet) SFS test also uses the ODA and I've tested it extensively in debug both before and after the MOM6 update.

For some grid points h_obs(i,j,:) is zero (for every k), so sum_h2 is also zero.

[DeniseWorthen]

Ok, so in the mom6_increment I'm using for the SFS test, the h_anl has vertical minimum of 1.05 in the ocean (Tmask=1).

[DusanJovic-NOAA]

Debug build of GDAS v17 configuration finished successfully on wcoss2 without any floating point exception.

@jiandewang Can you please check with MOM6 developers/code managers to see if the above change is correct. Thanks.

[DeniseWorthen]

There also might be something wrong w/ the increment file you're using...maybe it should not contain 0 values by design, but for some reason the one you're using does.

Still, the change to allowing this only over Tmask=1 is probably good practice in any case.

[DeniseWorthen]

I'm working on the C1172 RT configuration and I can confirm that in debug mode, the above failure in MOM_oda_incupd.F90 also occurs.

[jiandewang]

Debug build of GDAS v17 configuration finished successfully on wcoss2 without any floating point exception.

@jiandewang Can you please check with MOM6 developers/code managers to see if the above change is correct. Thanks.

See mom-ocean/MOM6#1681