Creating a minimal dev environment (with FMS history writing) #1858

barlage · 2023-08-09T22:17:53Z

barlage
Aug 9, 2023

I'm attempting to set up a bare-bones development environment, specifically one that can run on <= 8 processors and output every timestep on the native model grid using the global model. I guess my first question should be: is this even possible? My assumption is the answer is no if model_configure: quilting = .true. since the model and write component each need 6 processors (is this a correct assumption?). So, currently I'm trying to get FMS to write the tile output since it seems to not need additional processors.

I start with effectively the control_p8 regression test setup (model_configure: dt_atmos=720; nhours_fcst=1). Turning quilting off (model_configure: quilting = .false.), the model will run with 6 processors. However, I can't seem to get more than one output time (the first timestep) in the output files. I've stripped down diag_table to produce just the 2D 2-m temperature. I've tried several combinations in the file section. My last attempt looks something like this:

resource_testing
2021 03 22 06 0 0

"h%4yr%2mo%2dy%2hr%2mi",  12,  "minutes",  1,  "minutes",  "time", 1, "days"

"gfs_sfc",     "t2m"     ,    "tmp2m",     "h%4yr%2mo%2dy%2hr%2mi",  "all",  .false.,  "none",  2
"gfs_sfc",     "slmsksfc",    "land" ,     "h%4yr%2mo%2dy%2hr%2mi",  "all",  .false.,  "none",  2

which in my understanding of the FMS documentation (http://noaa-gfdl.github.io/FMS/group__diag__manager.html) should include one day of output into a file. Since my simulation length is one hour and time step is 12 minutes, I expect 5 times in the output file. I end up with the following, which is the correctly named file, but it only has the first time in the file.

$ ncdump -v time h_2021_03_22_06_12.tile3.nc          
netcdf h_2021_03_22_06_12.tile3 {
dimensions:
        grid_xt = 96 ;
        grid_yt = 96 ;
        time = UNLIMITED ; // (1 currently)
variables:
        float grid_xt(grid_xt) ;
                grid_xt:units = "degrees_E" ;
                grid_xt:long_name = "T-cell longitude" ;
                grid_xt:axis = "X" ;
        float grid_yt(grid_yt) ;
                grid_yt:units = "degrees_N" ;
                grid_yt:long_name = "T-cell latitude" ;
                grid_yt:axis = "Y" ;
        float time(time) ;
                time:units = "minutes since 2021-03-22 06:00:00" ;
                time:long_name = "time" ;
                time:axis = "T" ;
                time:calendar_type = "GREGORIAN" ;
                time:calendar = "gregorian" ;
        float tmp2m(time, grid_yt, grid_xt) ;
                tmp2m:_FillValue = 9.99e+20f ;
                tmp2m:missing_value = 9.99e+20f ;
                tmp2m:units = "K" ;
                tmp2m:long_name = "2m temperature" ;
                tmp2m:cell_methods = "time: point" ;
        float land(time, grid_yt, grid_xt) ;
                land:_FillValue = 9.99e+20f ;
                land:missing_value = 9.99e+20f ;
                land:units = "numerical" ;
                land:long_name = "sea-land-ice mask (0-sea, 1-land, 2-ice)" ;
                land:cell_methods = "time: point" ;

// global attributes:
                :NumFilesInSet = 1 ;
                :title = "resource_testing" ;
                :grid_type = "regular" ;
                :grid_tile = "N/A" ;
data:

 time = 12 ;
}

I assume I'm missing a setting somewhere in model_configure or input.nml, possibly in the &diag_manager_nml or &fms_io_nml sections. Here is my entire model_configure:

start_year:              2021
start_month:             03
start_day:               22
start_hour:              06
start_minute:            0
start_second:            0
nhours_fcst:             1

dt_atmos:                720
restart_interval:        0

quilting:                .false.
quilting_restart:        .false.
write_groups:            1
write_tasks_per_group:   1
itasks:                  1
output_history:          .true.
write_dopost:            .false.
write_nsflip:            .true.
nsout:                   1

Appreciate any feedback on if what I'm trying to do is even possible. Also wondering if there is an exhaustive list of settings with documentation for both model_configure and input.nml.

Answered by barlage

Aug 10, 2023

I got multi-time to work with setting in model_configure:
output_fh: 0.2 -1
and removing
nsout: 1

Without any real proof, I speculate that nsout might be getting reset after the first time step. This line in atmos_model.F90 seems to indicate that writing should happen for nsout>0. With the output_fh line above, the execution log looks like this(note the ---isec,seconds print):

0: PASS: fcstRUN phase 1, n_atmsteps =                0 time is        38.257919
0:  ---isec,seconds         720         720
0:   gfs diags time since last bucket empty:   0.200000000000000      hrs
0: PASS: fcstRUN phase 2, n_atmsteps =                0 time is         5.327943
1:  ncells=           5
1:  nlives=  …

View full answer

junwang-noaa · 2023-08-10T13:45:40Z

junwang-noaa
Aug 10, 2023
Maintainer

@barlage I think the way you specified the field will give you only one time step. Can you try:

"fv3_history", 12, "minutes", 1, "minutes", "time"
"fv3_history2d", 12, "minutes", 1, "minutes", "time"

"gfs_dyn", "ucomp", "ugrd", "fv3_history", "all", .false., "none", 2
"gfs_dyn", "vcomp", "vgrd", "fv3_history", "all", .false., "none", 2
..
"gfs_sfc", "crain", "crain", "fv3_history2d", "all", .false., "none", 2
"gfs_sfc", "tprcp", "tprcp", "fv3_history2d", "all", .false., "none", 2

3 replies

barlage Aug 10, 2023
Author

when you say "the way you specified the field", you mean using "fv3_history" instead of "h%4yr%2mo%2dy%2hr%2mi"?

barlage Aug 10, 2023
Author

@junwang-noaa I can try that. I started with something similar to that then added the time in the filename to see if that would help. Also, I originally did not have the "1, days" at the end. An interesting note is that if I put 6 for the output frequency (even though the time step is 12 minutes), the file will contain two times, an empty one for 6 minutes and the first time step at 12 minutes.

barlage Aug 10, 2023
Author

using this diag_table:

resource_testing
2021 03 22 06 0 0

"fv3_history"  , 12, "minutes", 1, "minutes", "time"
"fv3_history2d", 12, "minutes", 1, "minutes", "time"

###
# FV3 variables 
###

"gfs_dyn", "ucomp", "ugrd", "fv3_history", "all", .false., "none", 2

"gfs_phys",    "DSWRFI"  ,    "dswrf",     "fv3_history2d",  "all",  .false.,  "none",  2
"gfs_sfc",     "t2m"     ,    "tmp2m",     "fv3_history2d",  "all",  .false.,  "none",  2
"gfs_sfc",     "slmsksfc",    "land" ,     "fv3_history2d",  "all",  .false.,  "none",  2

I still only get the first time step:

ncdump -v time fv3_history2d.tile3.nc
netcdf fv3_history2d.tile3 {
dimensions:
        grid_xt = 96 ;
        grid_yt = 96 ;
        time = UNLIMITED ; // (1 currently)
variables:
        float grid_xt(grid_xt) ;
                grid_xt:units = "degrees_E" ;
                grid_xt:long_name = "T-cell longitude" ;
                grid_xt:axis = "X" ;
        float grid_yt(grid_yt) ;
                grid_yt:units = "degrees_N" ;
                grid_yt:long_name = "T-cell latitude" ;
                grid_yt:axis = "Y" ;
        float time(time) ;
                time:units = "minutes since 2021-03-22 06:00:00" ;
                time:long_name = "time" ;
                time:axis = "T" ;
                time:calendar_type = "GREGORIAN" ;
                time:calendar = "gregorian" ;
        float dswrf(time, grid_yt, grid_xt) ;
                dswrf:_FillValue = 9.99e+20f ;
                dswrf:missing_value = 9.99e+20f ;
                dswrf:units = "W/m**2" ;
                dswrf:long_name = "instantaneous surface downward shortwave flux" ;
                dswrf:cell_methods = "time: point" ;
        float tmp2m(time, grid_yt, grid_xt) ;
                tmp2m:_FillValue = 9.99e+20f ;
                tmp2m:missing_value = 9.99e+20f ;
                tmp2m:units = "K" ;
                tmp2m:long_name = "2m temperature" ;
                tmp2m:cell_methods = "time: point" ;
        float land(time, grid_yt, grid_xt) ;
                land:_FillValue = 9.99e+20f ;
                land:missing_value = 9.99e+20f ;
                land:units = "numerical" ;
                land:long_name = "sea-land-ice mask (0-sea, 1-land, 2-ice)" ;
                land:cell_methods = "time: point" ;

// global attributes:
                :NumFilesInSet = 1 ;
                :title = "resource_testing" ;
                :grid_type = "regular" ;
                :grid_tile = "N/A" ;
data:

 time = 12 ;
}

The reason I tried the time-based name is with something like "fv3_history" I was also getting only one time and I thought the system maybe wouldn't overwrite files and the default wasn't to put more than one time per file.

junwang-noaa · 2023-08-10T15:59:35Z

junwang-noaa
Aug 10, 2023
Maintainer

Yes, since the file name has minutes in it, I am not sure if the file can include the time other than the specified time in the name. Maybe you can try remove the minutes in the name?

1 reply

barlage Aug 10, 2023
Author

Some additional tests:
"h%4yr%2mo%2dy", 12, "minutes", 1, "minutes", "time"
still only puts one time in the file; interestingly the file name will be: h%4yr%2mo%2dy.tile3.nc

"h%4yr%2mo%2dy", 12, "minutes", 1, "minutes", "time", 1, "days"
also only puts one time in the file, but the filename will be: h_2021_03_22.tile3.nc

it seems new_file_freq must be added to the file section to get the file name to be generated correctly.

junwang-noaa · 2023-08-10T16:44:03Z

junwang-noaa
Aug 10, 2023
Maintainer

There might be some other settings. Can you take a look at: /scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_227608/control_p8_lndp_intel?

I have:

netcdf fv3_history2d.tile1 {
dimensions:
grid_xt = 96 ;
grid_yt = 96 ;
time = UNLIMITED ; // (240 currently)
pfull = 127 ;
phalf = 128 ;
...

time = 12, 24, 36, 48, 60, 72, 84, 96, 108, 120, 132, 144, 156, 168, 180,
192, 204, 216, 228, 240, 252, 264, 276, 288, 300, 312, 324, 336, 348,
360, 372, 384, 396, 408, 420, 432, 444, 456, 468, 480, 492, 504, 516,
528, 540, 552, 564, 576, 588, 600, 612, 624, 636, 648, 660, 672, 684,
696, 708, 720, 732, 744, 756, 768, 780, 792, 804, 816, 828, 840, 852,
864, 876, 888, 900, 912, 924, 936, 948, 960, 972, 984, 996, 1008, 1020,
1032, 1044, 1056, 1068, 1080, 1092, 1104, 1116, 1128, 1140, 1152, 1164,
1176, 1188, 1200, 1212, 1224, 1236, 1248, 1260, 1272, 1284, 1296, 1308,
1320, 1332, 1344, 1356, 1368, 1380, 1392, 1404, 1416, 1428, 1440, 1452,
1464, 1476, 1488, 1500, 1512, 1524, 1536, 1548, 1560, 1572, 1584, 1596,
1608, 1620, 1632, 1644, 1656, 1668, 1680, 1692, 1704, 1716, 1728, 1740,
1752, 1764, 1776, 1788, 1800, 1812, 1824, 1836, 1848, 1860, 1872, 1884,
1896, 1908, 1920, 1932, 1944, 1956, 1968, 1980, 1992, 2004, 2016, 2028,
2040, 2052, 2064, 2076, 2088, 2100, 2112, 2124, 2136, 2148, 2160, 2172,
2184, 2196, 2208, 2220, 2232, 2244, 2256, 2268, 2280, 2292, 2304, 2316,
2328, 2340, 2352, 2364, 2376, 2388, 2400, 2412, 2424, 2436, 2448, 2460,
2472, 2484, 2496, 2508, 2520, 2532, 2544, 2556, 2568, 2580, 2592, 2604,
2616, 2628, 2640, 2652, 2664, 2676, 2688, 2700, 2712, 2724, 2736, 2748,
2760, 2772, 2784, 2796, 2808, 2820, 2832, 2844, 2856, 2868, 2880 ;

0 replies

barlage · 2023-08-10T18:54:59Z

barlage
Aug 10, 2023
Author

I got multi-time to work with setting in model_configure:
output_fh: 0.2 -1
and removing
nsout: 1

Without any real proof, I speculate that nsout might be getting reset after the first time step. This line in atmos_model.F90 seems to indicate that writing should happen for nsout>0. With the output_fh line above, the execution log looks like this(note the ---isec,seconds print):

0: PASS: fcstRUN phase 1, n_atmsteps =                0 time is        38.257919
0:  ---isec,seconds         720         720
0:   gfs diags time since last bucket empty:   0.200000000000000      hrs
0: PASS: fcstRUN phase 2, n_atmsteps =                0 time is         5.327943
1:  ncells=           5
1:  nlives=          12
1:  nthresh=   18.0000000000000     
0: PASS: fcstRUN phase 1, n_atmsteps =                1 time is        11.525541
0:  ---isec,seconds        1440        1440
0:   gfs diags time since last bucket empty:   0.400000000000000      hrs
0: PASS: fcstRUN phase 2, n_atmsteps =                1 time is         0.384172
1:  ncells=           5
1:  nlives=          12
1:  nthresh=   18.0000000000000     
0: PASS: fcstRUN phase 1, n_atmsteps =                2 time is        11.265998
0:  ---isec,seconds        2160        2160
0:   gfs diags time since last bucket empty:   0.600000000000000      hrs
0: PASS: fcstRUN phase 2, n_atmsteps =                2 time is         0.375978
1:  ncells=           5
1:  nlives=          12
1:  nthresh=   18.0000000000000     
0: PASS: fcstRUN phase 1, n_atmsteps =                3 time is        10.912704
0:  ---isec,seconds        2880        2880
0:   gfs diags time since last bucket empty:   0.800000000000000      hrs
0: PASS: fcstRUN phase 2, n_atmsteps =                3 time is         0.389668
1:  ncells=           5
1:  nlives=          12
1:  nthresh=   18.0000000000000     
0: PASS: fcstRUN phase 1, n_atmsteps =                4 time is        10.927153
0:  ---isec,seconds        3600        3600
0:   gfs diags time since last bucket empty:    1.00000000000000      hrs
0:  write out restart at n_atmsteps=           4  seconds=        3600 
0:  integration length=  0.8000000    
0: PASS: fcstRUN phase 2, n_atmsteps =                4 time is         1.730532

with the nsout line above, it looks like this (note the ---isec,seconds only in the first time step):

0: PASS: fcstRUN phase 1, n_atmsteps =                0 time is        38.432412
0:  ---isec,seconds         720         720
0:   gfs diags time since last bucket empty:   0.200000000000000      hrs
0: PASS: fcstRUN phase 2, n_atmsteps =                0 time is         5.201233
1:  ncells=           5
1:  nlives=          12
1:  nthresh=   18.0000000000000     
0: PASS: fcstRUN phase 1, n_atmsteps =                1 time is        11.543172
0: PASS: fcstRUN phase 2, n_atmsteps =                1 time is         0.308594
1:  ncells=           5
1:  nlives=          12
1:  nthresh=   18.0000000000000     
0: PASS: fcstRUN phase 1, n_atmsteps =                2 time is        11.231538
0: PASS: fcstRUN phase 2, n_atmsteps =                2 time is         0.339961
1:  ncells=           5
1:  nlives=          12
1:  nthresh=   18.0000000000000     
0: PASS: fcstRUN phase 1, n_atmsteps =                3 time is        11.062545
0: PASS: fcstRUN phase 2, n_atmsteps =                3 time is         0.341714
1:  ncells=           5
1:  nlives=          12
1:  nthresh=   18.0000000000000     
0: PASS: fcstRUN phase 1, n_atmsteps =                4 time is        11.021451
0:  write out restart at n_atmsteps=           4  seconds=        3600 
0:  integration length=  0.8000000    
0: PASS: fcstRUN phase 2, n_atmsteps =                4 time is         1.662038

Is fv3_cap.F90 called even if there is no write component? Because this line may be resetting nsout to 0.

2 replies

junwang-noaa Aug 10, 2023
Maintainer

@barlage you are right! The nsout does not get set when quilt is set to .false. We are actually trying to switch to use output_fh which has the capability of outputting history files at certain frequency, at random forecast time and at each time step. We hope to clean up those legacy code of "nfhout/nfhouout_hf/nsout" etc.

barlage Aug 10, 2023
Author

Thanks @junwang-noaa. We should make sure the nsout option is removed from the model_configure documentation.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creating a minimal dev environment (with FMS history writing) #1858

{{title}}

Replies: 4 comments 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

Select a reply

Creating a minimal dev environment (with FMS history writing) #1858

barlage Aug 9, 2023

Replies: 4 comments · 6 replies

junwang-noaa Aug 10, 2023 Maintainer

barlage Aug 10, 2023 Author

barlage Aug 10, 2023 Author

barlage Aug 10, 2023 Author

junwang-noaa Aug 10, 2023 Maintainer

barlage Aug 10, 2023 Author

junwang-noaa Aug 10, 2023 Maintainer

barlage Aug 10, 2023 Author

junwang-noaa Aug 10, 2023 Maintainer

barlage Aug 10, 2023 Author

barlage
Aug 9, 2023

Replies: 4 comments 6 replies

junwang-noaa
Aug 10, 2023
Maintainer

barlage Aug 10, 2023
Author

barlage Aug 10, 2023
Author

barlage Aug 10, 2023
Author

junwang-noaa
Aug 10, 2023
Maintainer

barlage Aug 10, 2023
Author

junwang-noaa
Aug 10, 2023
Maintainer

barlage
Aug 10, 2023
Author

junwang-noaa Aug 10, 2023
Maintainer

barlage Aug 10, 2023
Author