-
Notifications
You must be signed in to change notification settings - Fork 51
Labels
data:ioIssues with the zarr output produced during inference/validationIssues with the zarr output produced during inference/validationinitiativeLarge piece of work covering multiple sprintLarge piece of work covering multiple sprintmodelRelated to model training or definition (not generic infra)Related to model training or definition (not generic infra)model:inferenceanything related to the inference step (not plotting or score computation).anything related to the inference step (not plotting or score computation).
Description
What happened?
Currently, forecast output is written after all forecast steps have been completed. This limits the number of steps we can consider, especially at very high resolutions. Hence, we should have an inference mode where the output is written after each step, or a small number of steps. (I cannot see a reason to have this in training, but it might exist). The current model.forward() function can be used--and should, to make sure we run training and inference identically-- but some minor modifications are needed. This will likely also affect the validation_io code, since we need to make sure the data is appended correctly.
@ankitpatnala @SavvasMel @Jubeku @grassesi
What are the steps to reproduce the bug?
No response
Hedgedoc link to logs and more information. This ticket is public, do not attach files directly.
No response
Metadata
Metadata
Assignees
Labels
data:ioIssues with the zarr output produced during inference/validationIssues with the zarr output produced during inference/validationinitiativeLarge piece of work covering multiple sprintLarge piece of work covering multiple sprintmodelRelated to model training or definition (not generic infra)Related to model training or definition (not generic infra)model:inferenceanything related to the inference step (not plotting or score computation).anything related to the inference step (not plotting or score computation).
Type
Projects
Status
No status