EDMFX ERA5 `ExternalDrivenTVForcing` test case slowing down

[nefrathenrici]

The buildkite test "Prognostic EDMFX ERA5 Time Varying driven in a column" has been slowing down very recently.

Each log I have looked at shows a sudden drop in SYPD, but this does not seem to happen deterministically. Most recent runs take ~20 minutes (current main).

Sample logs of this behavior: 1 2 3 4

[Julians42]

@nefrathenrici I'm not sure why this is happening - it was reasonably fast a month or two ago when I merged this in. You're welcome to change t_end to 8 hours or shorter if its slowing down ci - even though there is functionality we want to test I'm not sure its worth your time. I am mostly be using Prognostic EDMFX ERA5 monthly averaged diurnal driven in a column for experiments at the moment.

[nefrathenrici]

Thanks @Julians42 ! I don't think we need to shorten the test, it normally runs in 15-20 minutes.
I would rather try to figure out why the SYPD drops so suddenly sometimes, I suspect this is due to file I/O:

┌ Info: Progress
│   simulation_time = "1 minute, 20 seconds"
│   n_steps_completed = 8
│   wall_time_per_step = "51 milliseconds, 329 microseconds"
│   wall_time_total = "9 minutes, 14 seconds"
│   wall_time_remaining = "9 minutes, 13 seconds"
│   wall_time_spent = "410 milliseconds, 636 microseconds"
│   percent_complete = "0.1%"
│   estimated_sypd = "0.533"
│   date_now = 2025-07-23T14:28:51.599
└   estimated_finish_date = 2025-07-23T14:38:05.599
┌ Info: Progress
│   simulation_time = "2 minutes, 40 seconds"
│   n_steps_completed = 16
│   wall_time_per_step = "33 milliseconds, 158 microseconds"
│   wall_time_total = "5 minutes, 58 seconds"
│   wall_time_remaining = "5 minutes, 57 seconds"
│   wall_time_spent = "530 milliseconds, 531 microseconds"
│   percent_complete = "0.1%"
│   estimated_sypd = "0.826"
│   date_now = 2025-07-23T14:28:51.719
└   estimated_finish_date = 2025-07-23T14:34:49.719
┌ Info: Progress
│   simulation_time = "5 minutes, 20 seconds"
│   n_steps_completed = 32
│   wall_time_per_step = "23 milliseconds, 986 microseconds"
│   wall_time_total = "4 minutes, 19 seconds"
│   wall_time_remaining = "4 minutes, 18 seconds"
│   wall_time_spent = "767 milliseconds, 570 microseconds"
│   percent_complete = "0.3%"
│   estimated_sypd = "1.141"
│   date_now = 2025-07-23T14:28:51.956
└   estimated_finish_date = 2025-07-23T14:33:10.956
┌ Info: Progress
│   simulation_time = "10 minutes, 40 seconds"
│   n_steps_completed = 64
│   wall_time_per_step = "192 milliseconds, 30 microseconds"
│   wall_time_total = "34 minutes, 33 seconds"
│   wall_time_remaining = "34 minutes, 21 seconds"
│   wall_time_spent = "12 seconds, 289 milliseconds"
│   percent_complete = "0.6%"
│   estimated_sypd = "0.143"
│   date_now = 2025-07-23T14:29:03.479
└   estimated_finish_date = 2025-07-23T15:03:25.478
┌ Info: Progress
│   simulation_time = "21 minutes, 20 seconds"
│   n_steps_completed = 128
│   wall_time_per_step = "345 milliseconds, 873 microseconds"
│   wall_time_total = "1 hour, 2 minutes"
│   wall_time_remaining = "1 hour, 1 minute"
│   wall_time_spent = "44 seconds, 271 milliseconds"
│   percent_complete = "1.2%"
│   estimated_sypd = "0.079"
│   date_now = 2025-07-23T14:29:35.512
└   estimated_finish_date = 2025-07-23T15:31:07.512