-
Notifications
You must be signed in to change notification settings - Fork 18
Open
Description
This issue covers creating filters in anemoi-transform to support observations datasets within anemoi. As a first step, we will look at porting across existing AI-DOP routines that would be appropriate to be used as filters in the context of anemoi-datasets.
To-do:
- Identify a list of processing functions which can be converted to filters.
- Add unit tests for all existing processing functions.
- Support the selection of "filters" via a registry (so that filters can be selected from either
anemoi-transformor the existing code during the porting). - Create base class for new filters, matching
anemoi-transforminterface. - Convert filters to class based structure matching
anemoi-transformfilters. - Ensure the filters have a consistent interface, i.e. accept a dictionary as a config argument only.
- Reorganise the file structure of the filters so they can be moved across to
anemoi-transformin bulk.- (Still need to break out dependent functions so filters can be cleanly moved)
- Create integration tests with small (complete) configs.
- @pinnstorm / @pwlean to add more configs?
- Make keyword arguments/names (i.e. yaml interface) consistent across filters.
- Update docstrings/documentation for new filters.
-
Improve type hints/add schema documentation to all filters (basically "flavours" of dataframe).- Not a blocking feature, so created Consider techniques for documenting/validating inputs/outputs of filters #195 to cover this work.
- Move code across to
anemoi-transform. - Delete filters in existing repo.
- Throughout: ensure existing configs are updated to work with changes.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status
Now In Progress