Skip to content

Add filters to support creating tabular (observations) datasets #193

@aaron-hopkinson

Description

@aaron-hopkinson

This issue covers creating filters in anemoi-transform to support observations datasets within anemoi. As a first step, we will look at porting across existing AI-DOP routines that would be appropriate to be used as filters in the context of anemoi-datasets.

To-do:

  • Identify a list of processing functions which can be converted to filters.
  • Add unit tests for all existing processing functions.
  • Support the selection of "filters" via a registry (so that filters can be selected from either anemoi-transform or the existing code during the porting).
  • Create base class for new filters, matching anemoi-transform interface.
  • Convert filters to class based structure matching anemoi-transform filters.
  • Ensure the filters have a consistent interface, i.e. accept a dictionary as a config argument only.
  • Reorganise the file structure of the filters so they can be moved across to anemoi-transform in bulk.
    • (Still need to break out dependent functions so filters can be cleanly moved)
  • Create integration tests with small (complete) configs.
  • Make keyword arguments/names (i.e. yaml interface) consistent across filters.
  • Update docstrings/documentation for new filters.
  • Improve type hints/add schema documentation to all filters (basically "flavours" of dataframe).
  • Move code across to anemoi-transform.
  • Delete filters in existing repo.
  • Throughout: ensure existing configs are updated to work with changes.

Metadata

Metadata

Type

No type

Projects

Status

Now In Progress

Relationships

None yet

Development

No branches or pull requests

Issue actions