Shape mismatch issue

[ankitpatnala]

What happened?

I ran pre-training with forecast_mode on. forecast_offset=1 forecast_steps=0
Now on evaluating with this config

run_id=we83f1ia
epoch=24
start_date=202210011600
end_date=202211010400

I am getting a shape mismatch error afterwards.

What are the steps to reproduce the bug?

pre-train with training_mode="forecast"
validate with parameters mentioned above

Version

0.1

Platform (OS and architecture)

JUWELS Booster (Linux)

Relevant log output

0: [rank0]: Traceback (most recent call last):
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/.venv/bin/inference", line 10, in <module>
0: [rank0]:     sys.exit(inference())
0: [rank0]:              ^^^^^^^^^^^
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/src/weathergen/run_train.py", line 27, in inference
0: [rank0]:     inference_from_args(sys.argv[1:])
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/src/weathergen/run_train.py", line 65, in inference_from_args
0: [rank0]:     trainer.inference(cf, args.from_run_id, args.epoch)
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/src/weathergen/train/trainer.py", line 130, in inference
0: [rank0]:     self.validate(epoch=0)
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/src/weathergen/train/trainer.py", line 625, in validate
0: [rank0]:     write_validation(
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/src/weathergen/utils/validation_io.p
0: y", line 132, in write_validation
0: [rank0]:     ds[f"{rn}/sources"].append(source_k)
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/.venv/lib/python3.12/site-packages/zarr/core.py", line 2706, in append
0: [rank0]:     return self._write_op(self._append_nosync, data, axis=axis)
0: [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/.venv/lib/python3.12/site-packages/zarr/core.py", line 2596, in _write_op
0: [rank0]:     return self._synchronized_op(f, *args, **kwargs)
0: [rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/.venv/lib/python3.12/site-packages/zarr/core.py", line 2587, in _synchronized_op
0: [rank0]:     result = f(*args, **kwargs)
0: [rank0]:              ^^^^^^^^^^^^^^^^^^
0: [rank0]:   File "/p/project1/hclimrep/patnala1/WeatherGen/WeatherGenerator/.venv/lib/python3.12/site-packages/zarr/core.py", line 2717, in _append_nosync
0: [rank0]
0: :     raise ValueError(
0: [rank0]: ValueError: shape of data to append is not compatible with the array; all dimensions must match except for the dimension being appended

Accompanying data

No response

Organisation

JSC

[clessig]

This problem is reproducible? If yes, can you add a print( source_k.shape) here:

WeatherGenerator/src/weathergen/utils/validation_io.py

Line 108 in 2ec1b0f

and post the output.