Issues masking fields in the CMEPS mediator

[miguelsolanocordoba]

Background
The weather team at Sofar Ocean has been working on a coupled atmosphere-wave model application, based on FV3-SHiELD for the atmosphere/land and WaveWatchIII for the wave component (as a start - we are working on adding MOM6 as the ocean component). For atmosphere-wave coupling, we are passing surface neutral winds, air-density and air-sea temperature differences from atmos to waves, and the Charnock paramter from waves to atmosphere. We were previously using a simple ESMF/NUOPC mediator, but quickly ran into the issue of only being able to run coupled at the atmosphere step (2.5min for our global model), so we decided to move to CMEPS as our mediator. We've had some success with the new coupling (we can now run at lower coupling frequencies, among other advantages), but have 1 outstanding masking issue (explained below).

Description
I noticed that some of the default masking options are changed depending on application specific uses:

    !----------------------------------------------------------
    ! Default masking: for each component, the first element is
    ! when it is the src and the second element is when it is
    ! the destination
    !----------------------------------------------------------

    nullify(is_local%wrap)
    call ESMF_GridCompGetInternalState(gcomp, is_local, rc)
    if (ChkErr(rc,__LINE__,u_FILE_u)) return 

    allocate(defaultMasks(ncomps,2))
    defaultMasks(:,:) = ispval_mask
    if (is_local%wrap%comp_present(compocn)) defaultMasks(compocn,:) = 0
    if (is_local%wrap%comp_present(compice)) defaultMasks(compice,:) = 0
    if (is_local%wrap%comp_present(compwav)) defaultMasks(compwav,:) = 0
    if ( coupling_mode(1:3) == 'ufs') then
       if (is_local%wrap%comp_present(compatm)) defaultMasks(compatm,:) = 1
    endif
    if ( trim(coupling_mode) == 'hafs') then
       if (is_local%wrap%comp_present(compatm)) defaultMasks(compatm,1) = 1
    endif
! comment this out, so that default masks are set to ispval if missing from esm_run.config
!    if ( trim(coupling_mode(1:5)) == 'sofar') then
!       if (is_local%wrap%comp_present(compatm)) defaultMasks(compatm,1) = 1
!       if (is_local%wrap%comp_present(compwav)) defaultMasks(compwav,1) = 1
!    endif
    if ( trim(coupling_mode) /= 'cesm') then       
       if (is_local%wrap%comp_present(compatm) .and. trim(atm_name(1:4)) == 'datm') then
          defaultMasks(compatm,1) = 0
       end if
    end if

It seems odd to hard-code some of these "default" values, as pointed out in this (still open) issue: #93

Nevertheless, I modified our forked CMEPS branch to take in the srcMaskValues and dstMaskValues at runtime in the configuration file (*.config), to allow for further testing. Using the default mask (ispval) or masking land in the (source) wave model doesn't work.

 (module_med_map: med_map_routehandles_initfrom_field) creating RH fillv_bilnr for atm to wav srcMask =    -987987 dstMask =    -987987
 (module_med_map: med_map_routehandles_initfrom_field) creating RH fillv_bilnr for wav to atm srcMask =          1 dstMask =    -987987

For some unknown reason, we can't seem to mask fields over land in the mediator and it seems some NaN values are getting passed in the Charnock paramter values to the atmosphere, which is causing it to blow-up. We could "hack" these values if necessary, but I was wondering if anyone had run into this issue before? What would be the proper way to mask sea/land/ice values in the mediator for an atmosphere-wave configuration?

Thanks in advance for any feedback.

[mvertens]

@miguelsolanocordoba - which mode are your using -

• UFS or CESM (this will determine if you are using esmFldsExchange_ufs_mod.F90 or esmFldsExchange_cesm_mod.F90).
• What components are you using?
• What mapping are you using for wav->atm
• You can only mask source/destination mapping - if it is wav-> atm or atm-> wav.
• I'm cc'ing @DeniseWorthen on this since I think she added this capability to CMEPS.

[DeniseWorthen]

@miguelsolanocordoba The only mapfillv_bilnr use that I am aware of in CMEPS is for the UFS HAFs application. That is a regional application with specific uses because of non-overlapping domains (non-overlapping in this case is not just that the land mask doesn't match---the domains themselves extend out past each other).

So it would be helpful to understand which field exchange you are using, if your domains are global, if your ATM and WAV land masks match (or you need to fill in non-mapped points with nstod) etc.

[miguelsolanocordoba]

Thanks for the quick reply @mvertens and @DeniseWorthen.

We used the UFS HAFs application as our starting point, so we follow their application closely (i.e. we essentially use the esmFldsExchange_hafs_mod.F90 routine). Here is some of the information they are asking for:

• We use a global domain, with ATM and WAV model components. The ATM has an embedded land and ocean mixed layer model.
• We follow UFS HAFs in using billinear mapping.
• ATM and WAV masks do NOT match. For example, the ATM has "sea" points over lakes, while the WAV masks those. WAV is 1/4 deg while the ATM is C768 (~ 13 km), so there are differences due to the different resolutions around coastlines.

Seems like the solution here is to fill non-mapped points with nstod, correct? Could you provide more details on how that would get implemented?

Let me know if I missed any details, or if more information would be helpful.

[DeniseWorthen]

@miguelsolanocordoba Thanks for the additional information. Obviously details matter, and I don't know anything about FV3-SHieLD. But for UFS, we have a regularly tested configuration of global ATM-WAV only, with a C96 ATM and a 1-deg rectilinear WW3 domain, which sounds somewhat comparable. For that configuration we send 10m winds and bottom temperature to WAV and z0 to ATM (I don't believe CESM has any configuration where WAV -> ATM). We're mapping to WAV with bilinear and from WAV with bilinear+nstod.

Our ATM-WAV configuration is global and so uses the esmFldsExchange_ufs_mod.F90. (It's possible that is a better choice for you than the HAFS setup.) As of Monday, I've made updates which are applicable here---the srcMasking we're now using for the ATM and how we're mapping various other fields. I haven't had a chance to push those changes back up to ESCOMP though, so you'd have to look at NOAA-EMC's fork of CMEPS.

[miguelsolanocordoba]

We're mapping to WAV with bilinear and from WAV with bilinear+nstod.

@DeniseWorthen, thank you for your feedback. Just wanted to come back to this, and point out that the suggestion above solved our issue, and all atm-wav fields seem to be working correctly.

[DeniseWorthen]

@miguelsolanocordoba Thanks for the update.