Skip to content

Performance of MultiStreams data loading and step_hrs modification #369

Open
@Sindhu-Vasireddy

Description

@Sindhu-Vasireddy

Experiment with loading the 3 available data streams via anemoi data reader:

  • opera dataset as rodeo-opera-files-2km-2013-2023-15m-v1-lambert-azimuthal-equal-area.zarr,
  • imerg dataset as nasa-imerg-grib-n320-1998-2024-6h-v1.zarr, and
  • aifs dataset as aifs-ea-an-oper-0001-mars-o96-1979-2023-6h-v8.zarr

to evaluate the performance when loading data from multiple streams, while also validating the recent changes made in the structure of the data readers.

A subtask of evaluating the impact of the step_hrs parameter during training to be performed, and results presented for each of the streams by modifying its values with 1,7 etc... .

Inclusion other streams of observation datasets such as Fesom, ICON to evaluate the performance will be considered post completion of intial experiment.

Metadata

Metadata

Labels

modelRelated to model training or definition (not generic infra)

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions