Forecast steps time index error when step_hr exceeds len_hr length #1659
Description
What happened?
When the configuration of step_hr is more than `len_hr', the forecast steps behaves not in a right manner. There are indexing issues
for settings when step_hr =12:00:00 and len_hr=06:00:00
self.forecast_dt = 06:00:00
self.step_time_delta = 12:00:00
for fstep = 1, it becomes 6*1 // 12 which equals 0. So, there is no increment of step_forecast_dt
It can be seen in this zarr file
inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=24:00:00
validation_chkpt00000_rank0000.zarr
>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
'2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
'2022-10-01T00:00:00', '2022-10-01T00:00:00'],
shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
'2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
'2022-10-01T00:00:00', '2022-10-01T00:00:00'],
shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
'2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
'2022-10-01T00:00:00', '2022-10-01T00:00:00'],
shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
'2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
'2022-10-02T00:00:00', '2022-10-02T00:00:00'],
shape=(40320,), dtype='datetime64[s]')
The target_times for step 1,2, and 3 is same here
inference_script : srun uv run --offline inference --from_run_id jhty3fja --samples=10 --options training_config.forecast.num_steps=16 training_config.time_window_step=12:00:00
>>> ds["0/ERA5/1/target/times"][:]
array(['2022-10-01T00:00:00', '2022-10-01T00:00:00',
'2022-10-01T00:00:00', ..., '2022-10-01T00:00:00',
'2022-10-01T00:00:00', '2022-10-01T00:00:00'],
shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/2/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
'2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
'2022-10-01T12:00:00', '2022-10-01T12:00:00'],
shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/3/target/times"][:]
array(['2022-10-01T12:00:00', '2022-10-01T12:00:00',
'2022-10-01T12:00:00', ..., '2022-10-01T12:00:00',
'2022-10-01T12:00:00', '2022-10-01T12:00:00'],
shape=(40320,), dtype='datetime64[s]')
>>> ds["0/ERA5/4/target/times"][:]
array(['2022-10-02T00:00:00', '2022-10-02T00:00:00',
'2022-10-02T00:00:00', ..., '2022-10-02T00:00:00',
'2022-10-02T00:00:00', '2022-10-02T00:00:00'],
shape=(40320,), dtype='datetime64[s]')
>>>
This is also addressed in Issue #1595
What are the steps to reproduce the bug?
To reproduce: train a sample model with default config and use the inference command as stated above. Inspect the zarr file afterwards.
Hedgedoc link to logs and more information. This ticket is public, do not attach files directly.
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status