restart issue with MYNN in WRFV4.2.2

[xqu-crypto]

A restarted run using restart files with MYNN in WRFV4.2.2 yields different wrf output from a continuous run without using restart files.

The continuous run started on 07/29/2018 00z and ran 10 days. It generates hourly wrf output and daily restart files. The run is forced by ERA5 reanalysis.

The restarted run started on 08/03/2018 00z and used the restart file generated from the continuous run. All other files (wrflow*, wrfbdy*, wrffdda*) are unchanged.

Then we computed the difference in wrfout files between the two runs for different hours using ncdiff.

1. On restart time (08/03/2018 00z), there is no difference between the two runs.

2. On all hours after the restart time, the two runs diverge. On 1 hour after the restart time (08/03/2018 01z), T2 can differ by 2 degree, and U/V can differ by 1 m/s.

The namelist.input for the restarted run is attached below.

namelist.input.txt

[weiwangncar]

@xqu-crypto Can you test the restart without using adaptive time stepping?

[xqu-crypto]

Thanks for your comment, Wei. I tested the role of adaptive time stepping in the restart issue. I switched off the adaptive time stepping, and set time_step to 30 seconds. The typical time stepping is shown as the following: 30s for domain 1, 10s for domain 2 and somewhat uneven time steps for domain 3 since step_to_output_time is set to true.

Timing for main: time 2008-07-30_00:00:03 on domain 3: 11.00377 elapsed seconds
Timing for main: time 2008-07-30_00:00:06 on domain 3: 0.11013 elapsed seconds
Timing for main: time 2008-07-30_00:00:10 on domain 3: 0.10990 elapsed seconds
Timing for main: time 2008-07-30_00:00:10 on domain 2: 94.66403 elapsed seconds
Timing for main: time 2008-07-30_00:00:13 on domain 3: 0.13542 elapsed seconds
Timing for main: time 2008-07-30_00:00:16 on domain 3: 0.10976 elapsed seconds
Timing for main: time 2008-07-30_00:00:20 on domain 3: 0.10850 elapsed seconds
Timing for main: time 2008-07-30_00:00:20 on domain 2: 0.49601 elapsed seconds
Timing for main: time 2008-07-30_00:00:23 on domain 3: 0.10858 elapsed seconds
Timing for main: time 2008-07-30_00:00:26 on domain 3: 0.10998 elapsed seconds
Timing for main: time 2008-07-30_00:00:30 on domain 3: 0.11105 elapsed seconds
Timing for main: time 2008-07-30_00:00:30 on domain 2: 0.44336 elapsed seconds
Timing for main: time 2008-07-30_00:00:30 on domain 1: 147.72408 elapsed seconds

The differences in T2, U10 and V10 due to restart are somewhat reduced compared to the runs with adaptive time stepping, but definitely non-zero.

[weiwangncar]

@xqu-crypto Just make sure that you are doing both the continuous run and restart run without adaptive time stepping. Is that correct? Would you also be interested in testing the latest version of the model without adaptive stepping?

[xqu-crypto]

Yes, I was doing both the continuous run and restart run with fixed time stepping using WRFV422. I also tested both runs with adaptive time stepping using WRFV452. For this version, the difference due to restart seems to be small in the interior but much larger near the boundaries between domain 2 and domain 3. I'll do another test with fixed time stepping using WRFV452.

[xqu-crypto]

I tested fixed time stepping with WRF452. The difference due to restart seems to be smaller than that with WRFV422 but still non-negligible over a few isolated regions.

[weiwangncar]

@xqu-crypto Thanks for testing MYNN PBL in WRF v4.5.2. The latest WRF version is 4.7.1 released in early June this year. Can we ask you to test it too?

[xqu-crypto]

I tested WRFV4.7.1 with fixed time stepping. The difference due to restart is much smaller than the early versions. In one hour after the restart, differences in T2, U10 and V10 are typically less than 0.001. In a day after the restart, differences in T2 is typically around 0.5K, and differences in U10 and V10 is typically less than 1m/s. Clearly, the impact is not negligible for long simulations.

[weiwangncar]

@xqu-crypto Thank you for testing WRF v4.7.1. Did you test your namelist with another PBL option? How did you decide the reason for the differences between the restart and continuous runs are due to MYNN (PBL+sfc) options? Could it be due to other options used in the runs?

[xqu-crypto]

At this point, I don't know exactly what caused the differences. I'll try a different PBL option.

[xqu-crypto]

I replaced MYNN with YSU. In a day after the restart, difference in T2 is less than 0.1K in most locations, and differences in U10 and V10 is less than 0.1 m/s in most locations. But in a few isolated regions, difference in T2 can reach up to 1K, and differences in U10 and V10 can reach up to 1m/s. Perhaps we should treat these differences as the rounding errors.

[weiwangncar]

@xqu-crypto Again, thanks for the tests. If you're using the same number of processors and run the model on the same computer, we should expect the results to be the same. Do you write any output less than a day? If you do, what does that compare to the continuous run? I'm interested in the files written before the stop time.
Your tests may also suggest that the problem isn't with the PBL scheme, but with one of the other physics. If you don't mind more tests, you could switch radiation option from 5 to 4 (option 4, RRTMG, is used a lot more), and LSM from NoahMP with Noah.

[xqu-crypto]

I have the hourly output. Since the differences right after the restart is very small, I assume they grow more or less linearly with time. I could do more tests but it may take some time.

[xqu-crypto]

I did a test run with RRTMG (LW and SW radiation option 4) and Noah (land surface scheme option 2) and WRFV4.7.1. After one-day simulation, difference in T2 is typically less than 0.1K, but differences in U10 and V10 can reach up to 1m/s in isolated regions.

[weiwangncar]

@xqu-crypto Can you share the namelist.input file you used to run this test? I assume you're using the same number of processors / tasks to run both the continuous and restart runs.

[xqu-crypto]

Yes, I was using the same number of processors to run both runs. Here are the namelist.input files.

namelist.input.restart.txt
namelist.input.continuous.txt