Adaptive_inflate_mod NaN issue?

[hkershaw-brown]

Expand on "the error in the adaptive_inflate_mod persists though; there is more work to be done in terms of exploring which files are given to check_attribtues with various namelist configurations and if this is suitable"

There is a fix for the previous inability to check for NaNs as inconsistent attribute values (i.e Fill, missing_value) in the check_attributes_value_r4/r8 subroutines on dev branch nan_check_attributes

The error in the adaptive_inflate_mod persists though; there is more work to be done in terms of exploring which files are given to check_attribtues with various namelist configurations and if this is suitable

Originally posted by @mjs2369 in #970

[hkershaw-brown]

@mjs2369 when you get a chance can you confirm the details of this bug.
I think this is check_attributes not checking restarts and mean and sd and inflation files, but I am not sure.

[mjs2369]

The subroutine check_attributes in the io_filenames_mod is intented to check that cf-convention attributes (i.e. fill_value, missing_value, scale_factor, add_offset, unit, long_name, short_name) are consistent across restarts.

In this subroutine, check_attribute_value_r4/r8 are called to check that the values of missing_value and fill_value are the same across restarts.

bug-fix #984 fixes the issue in check_attribute_value_r4/r8 where it was not actually catching if these values were different if the attribute in the given netcdf file was a NaN (see bug-fix for more details).

The original error that kicked off the investigation of this was that my ROMS_Rutgers test case was erroring out with a floating point exception in a minval function at this point in the adaptive_inflate_mod due to certain compilers not being able to do arithmetic operations with a NaN with debugging flags on -

DART/assimilation_code/modules/assimilation/adaptive_inflate_mod.f90

Line 1457 in c4d5cae

minmax_mean(1) = minval(ens_handle%copies(ss_inflate_index, :))

Program received signal SIGFPE: Floating-point exception - erroneous arithmetic operation.

Backtrace for this error:
#0  0x14cad4d6d8ff in ???
#1  0x4f527c in __adaptive_inflate_mod_MOD_get_minmax_task_zero
	at /glade/derecho/scratch/masmith/DART11.17.0/DART/assimilation_code/modules/assimilation/adaptive_inflate_mod.f90:1457

I originally thought that #984 would "fix" this problem by having it error out in check_attribute_value_r4/r8 instead with the more appropriate message:

ERROR FROM:
  source : io_filenames_mod.f90
  routine: check_attribute_value_r4
  message:   variable attribute, _FillValue in state   9.99999993E+36  does not match _FillValue   NaN  in input_priorinf_sd.nc

However, this was not the case; the ROMS_Rutgers test case was still erroring out with the floating point exception in the adaptive_inflate_mod.

This is because check_attributes is not actually being called on the files input_priotinf_sd or input_priorinf_mean, which is where the FIll_Value was set to NaN in the ROMS_Rutgers test case; it is actually only called on the set of files to output (file_info_output) which is just a subset of CURRENT_COPIES:

DART/assimilation_code/modules/assimilation/filter_mod.f90

Line 2687 in e08756e

call set_filename_info(file_info_output, 'output', ens_size, CURRENT_COPIES )

This is confirmed by adding print statements that show that the only filenames that are passed check_attributes for this 3 ens member test case are:
ens/up_cawo1.nc ens/up_cawo2.nc ens/up_cawo3.nc output_mean.nc output_sd.nc output_priorinf_mean.nc output_priorinf_sd.nc output_postinf_mean.nc output_postinf_sd.nc

Therefore, If you have NaNs in your input_priorinf_x (or use NaN as the FillValue), DART will still error out in the adaptive_inflate_mod.

It seems that ultimately, the intent of the subroutine check_file_info_variable_shape (which is the subroutine that ultimately call check_attributes, is to just verify this for the output files "to prevent overwriting unwanted files":

DART/assimilation_code/modules/io/io_filenames_mod.f90

Lines 322 to 327 in 66fc08d

	!-------------------------------------------------------------------------------
	!> Check that the netcdf files variables have the correct shape
	!> to prevevent overwriting unwanted files
	subroutine check_file_info_variable_shape(file_info, ens_handle)

Would we also want to perform these checks for input files? Or is it not an issue if the input files use different fill_values, missing_values, etc.?

But it seems to me that we should at least error out in a more helpful way if there are NaNs in the input inflation files.