ERA5 daily dataset

[lopsided]

Hi,

I am excited to try out the shiny new ERA5 anemoi dataset, but for my purposes I need this in daily format.

I could do this manually with zarr/dask etc, but I imagine it could be somewhat painful and I don't want to break any of the nice optimisations/encoding/chunking that is set up on the 6-hrly dataset. I would like to write the daily data to a new zarr dataset rather than do aggregation on the fly, as I think that will be big bottleneck for ML training.

I saw this: https://anemoi.readthedocs.io/projects/datasets/en/latest/howtos/using/01-interpolate-step-dataset-combination.html#interpolate-step which looks like it could be useful, but it's not clear how values are generated at the lower temporal resolution. Eg, precipitation is 6-hour accumulation, so we would want to sum the 4 values at hours 6/12/18/24, but wind speeds are instantaneous, so we would want to take the mean of the 4 values at hours 0/6/12/18. How is the aggregation done here?

Would you suggest using anemoi to generate the daily dataset? Are there any tips or suggestions for doing this? Or would I be better off just writing a script to do it all manually?

Many thanks

[anaprietonem]

Hello @lopsided thanks for the issue. Regarding the ERA5 dataset and the possibility to build a daily one out of this. Currently in anemoi-datasets it is not supported to compute accumulations when opening the dataset, so if you wanted to redo variables like tp to define the accumulations over 24hr that's not possible. For other instantaneous variables you could look at using the ecmwf/anemoi-datasets#448 rolling_average keyword to compute the daily means. The functionality for rolling_average is just available at reading time not when building. If you want daily means ready for creating the dataset, you'd need to preprocess the data ahead.

Depending on what's your exact use case or how you want to aggregate your date and compute the accumulations, you might prefer to rebuild the dataset at least for the accumulations. Support for accumuations and aggregations at both building and reading time is on active development and we hope to improve that in the future, but we can't give you any timelines.