TF5: Initial coupled evaluation with CAM-SE_L58/MOM6

[cecilehannay]

The SSC wants an evaluation in coupled mode of SE dycore with MOM6 within 6-9 months. The main goal of the exercise is to test feasibility.

We had a discussion at the CAM7 meeting on how to proceed.

• We agreed that a good starting point would be to use:
  cam_dev with L58 with reordering but without PUMAS
  as in run
  f.e21.FWscHIST.ne30_L48_BL10_cam6_3_041_control.hf.001 #53.

• This will be a combined effort between AMWG and OMWG.

AMWG: Cecile, Julio, Peter, Isla, Adam, Rich...
OMWG: Keith, Frank, Gustavo, Gokhan, ...

Please email Cecile to have your name added to the list in case we set up side meetings.

[adamrher]

Hi all, I was looking over this paper
https://agupubs.onlinelibrary.wiley.com/doi/10.1029/2020MS002052

And it reminded me that FV/FV3 dycore has additional interpolations from the staggered winds at edges (I thought this was called the c-grid, but they refer to it as d-grid in the paper) to the grid centers. It looks like from their Figure 1 this occurs in the dp_coupling layer. Anyhow, CAM-SE does not need to center their winds since they are natively at grid centers; and is yet another reason the winds mapped to the OCN grid are less diffusive in SE relative to FV.

[cecilehannay]

On Friday, Gustavo helps me to set a new sandbox to start the Coupled evaluation with MOM6.

For this run, I will use zm bugfix (as in #100) and the mods to fix numerical noise near topography (as in #98). This run will also use the nuopc driver.

I am wondering if we should also do a new L58 control with the combined mods from #100 and #98. This new control would also use the nuopc driver. And potentially we could use FHIST instead of FWscHIST (this was discussed several times before but there was never a firm commitment one way or the other).

I don't know if the mods for the upper boundary conditions are ready for the L58. It could be in the next run.

[adamrher]

Re: FHIST vs FWscHIST, I've shown that the Tropical cold point is a bit warmer in FHIST, compared with FWscHIST. Compared with ERAI, I believe this colder cold point is an improvement over FHIST. I haven't figured out why yet, although I've confirmed it's not O3 since they use the same prescribed O3 file.

FWscHIST has 4 more active tracers than FHIST:

12  CH4       CH4                                                                                                                               dry
13  N2O       N2O                                                                                                                               dry
14  CFC11     CFC11                                                                                                                             dry
15  CFC12     CFC12    

My instinct is that we should include CH4 and N2O in our configuration going forward, but prescribe CFCs from a file. @dan800? Or could the CFC differences be responsible for the better cold point?

[JulioTBacmeister]

We should just use FWscHIST and BWscHIST for these experiments. I don't think we should worry for these runs about recoding what is fully prescribed and what advected after being specified as near surface mixing ratio.

[cecilehannay]

Based on Julio's response, I will start two runs:

• a FWscHIST at L58 with mods from f.c6_3_41.FWscHIST.ne30_L58_exner.002 #98 and f.c6_3_41.FWscHIST.ne30_L58.zm2_fix.001 #100 + nuopc
• a BWsc1850 with MOM with the same atm as the run above

These runs will use the CAM_CONFIG_OPTS:
"-phys cam_dev -chem waccm_sc_mam4 -nlev 58"

I expect that the BWsc1850 will need some tuning to get an acceptable RESTOM. We will look at the coupled simulation as soon as we have enough years to evaluate the RESTOM.

At the same time, we can continue to investigate:

• new configurations (that include CH4 and N2O and prescribed CFCs) as Adam suggests. - upper boundary conditions in L58 that Francis is working on.

Does it sound like a plan?

[adamrher]

sounds good. The only wrinkle is that in a near future PR, I'm going to switch -phys cam_dev to not invoke -microphys mg3, and instead run w/ -microphys mg2. This is for computational expediency. So you may want to add a -microphys mg2 flag to your CONFIG_OPTS.

Note also that in user_nl_cam, you should not need to specify mg_micro_version or mg_dograupel/hail namelists, as these should automatically set to the appropriate values according to the -microphys flag.

[cecilehannay]

Hi Adam,

To make sure I understand this well.

First, I want to clarify that I was planning to run with cam6_3_41 which still use the mg2. So I think it doesn't make any difference for this tag to use:
-phys cam_dev -chem waccm_sc_mam4 -nlev 58
or
-phys cam_dev -microphys mg2 -chem waccm_sc_mam4 -nlev 58
Correct me if I am wrong.

Are you suggesting that I use a more recent tag than cam6_3_41 and to use: -microphys mg2? If there is a benefit to move to a more recent tag, I could update the sandbox I created with Gustavo.

[adamrher]

I'm saying use cam6_3_041 but specify -microphys mg2. In the 041 code, cam_dev turns on mg3, and we want to revert back to mg2.

[cecilehannay]

I thought cam_dev was turning on mg3 starting in the cam6_3_046.

[adamrher]

no. cam_dev turns on mg3. In 046 it turns on mg3+PUMAS.

[cecilehannay]

Ok. So I will run with:
-phys cam_dev -microphys mg2 -chem waccm_sc_mam4 -nlev 58

Are there big differences between mg2 and m3?

[adamrher]

no, it doesn't seem to make a difference at 1deg.

[cecilehannay]

Just an update on where the coupled simulation stands.

F case
We completed a new control in standalone mode with the configuration we want to use for the coupled simulation (includes zm bugfix + the mods to fix numerical noise near topography. ADF and AMWG Diags are available. Maybe @swrneale and @PeterHjortLauritzen could gibe an update about these mods at the CAM7 meeting.

B case
After a few hiccups, we were able to compile the configuration with CAM-SE L58 + MOM, but it crashes in the cam/src/chemistry/utils/modal_aero_wateruptake.F90 with the error:

2150: setting radius to dry radius=  6.964271883218964E-002
2150: ccm kohlerc - no real(r8) solution found (quartic)
2150: roots = (NaN,NaN) (NaN,NaN) (NaN,NaN) (NaN,NaN)
2150: p0-p3 = -1.028222055345795E-006                     NaN  0.000000000000000E+000
2150:  3.044105941181562E-003
2150: rh=  0.672564388259088     
2150: setting radius to dry radius=  6.964271883218964E-002
2150:MPT ERROR: Rank 2150(g:2150) received signal SIGSEGV(11).

See: /glade/scratch/hannay/b.cesm3_cam041_mom.B1850MOM.ne30_L58_t061.001
@tilmes: do you have any advice? Thanks.

[adamrher]

looks like it's crashing in the initialization phase (nstep=0). This may be related to my re-ordering, since I call this module for nstep=0. Can you try to run w/o the cam_dev flag to see if it gets past nstep=0?

[cecilehannay]

Will do.

[jedwards4b]

Here is further output I see in DEBUG mode:

388: Sponge layer del4 coefficient defaults based on model top location:
388:
388:forrtl: severe (408): fort: (3): Subscript #3 of the array DATA_Q has value -1 which is less than the lower bound of 1
388:
388:Image PC Routine Line Source
388:cesm.exe 000000001035705F Unknown Unknown Unknown
388:cesm.exe 000000000417DB1A tracer_cnst_mp_ge 331 tracer_cnst.F90

[jedwards4b]

I added tracer_cnst_specifier = 'O3','OH','NO3','HO2','HALONS'
to user_nl_cam and can now run with -phys cam6
I'll try again with cam_dev

[jedwards4b]

Running with cam_dev it still fails

0: Number of interpolation points claimed by more than one element:            0
0: max number of elements which claimed the same interpolation point:           1
2549:MPT ERROR: Rank 2549(g:2549) received signal SIGFPE(8).

The run is here if you want to have a look: /glade/scratch/jedwards/PFS.ne30pg3z58_t061.B1850MOM.cheyenne_intel.20220420_102035_iylw2m

[jedwards4b]

Even though I still have the DEBUG flag on I'm not getting any debug info.

[adamrher]

can you point me to which job number has the error so I can find the right log file?

[adamrher]

nrmind found it here: cesm.log.3845642.chadmin1.ib0.cheyenne.ucar.edu.220420-140715

I'm going to have to experiment with this configuration, adding in debugging statements.

[PeterHjortLauritzen]

I don't think this is the issue but can you try and turn off interpolation of output in the atmosphere (I have had problems on Izumi when using interpolated output - model just stops without any info), i.e. remove

interpolate_output = .true.,.true.
interpolate_nlat = 192,192
interpolate_nlon = 288,288

from user_nl_cam.

[adamrher]

it runs fine for cam6, it's only a cam_dev problem. I suspect it's because I'm not calling cam_diagnostics for nstep=0, which sets the cam_out array for diag_export. That's my first thought -- it's def. a problem even if it's not the source of this problem.

[adamrher]

Following up on the mtg, do we want to invoke -co2_cycle or not for these runs? We can punt this to later, but at some point we should probably figure this out.

[klindsay28]

These initial runs do not have BGC in MOM, so my 2 cents is to punt for now.

[adamrher]

OK, thanks @klindsay28. So keep running like your running @cecilehannay!

[cecilehannay]

We completed the first coupled simulation: #107

Results were presented at the CAM7 meeting. (The presentation requires access to the CAM7 drive).

The simulation (#107) exhibits excessive fresh water flux.
The topic will be discussed in: #109

We are locking the current discussion.