Different pre-processors for for different input sources in cutout

[bdvllrs]

Is your feature request related to a problem? Please describe.

Currently, the preprocessors are created by the runner and they are used indifferently on all input sources.

This is problematic in the case of a cutout where we might want different pre-processors for the different sources (e.g. use remove_nans for the local source and nothing for the global source).

I would happily provide a PR with the chosen method.

Describe the solution you'd like

Because pre-processors are applied by inputs, maybe pre-processors should rather be a parameter of inputs?

For example for a cutout input with two grib inputs:

input:
  cutout:
    lam_0:
        grib:
          path: /path/to/local.grib
          pre_processors:
            - forward_transform_filter: remove_nans
    global:
        grib:
          path: /path/to/global.grib

This would also make it explicit that preprocessing are handled by inputs.

Describe alternatives you've considered

An alternative would be to have preprocessors prefixed by a source:

pre_processors:
    lam_0:
    - forward_transform_filter: remove_nans

It could be defined in the create_pre_processors of a cutout runner.

(I have seen that there is a current Cutout runner, but I haven't yet understood its purpose in the context of a cutout inference).

Additional context

No response

Organisation

Météo-France

[gmertes]

Great idea! I suggest we go with the first option (processors can be attached to an input and output) but we also keep the top-level definition. So you can do this:

pre_processors:
  - preproc1

input:
  cutout:
    lam_0:
        grib:
          path: /path/to/local.grib
          pre_processors:
            - preproc2
    global:
        grib:
          path: /path/to/global.grib

In the above case, we apply preproc1 to both inputs, and preproc2 only to the lam_0 input.

We should do the same thing for the post-processors. We have the tee output that can write to multiple outputs, so there it would also be useful to do something like:

post_processors:
  - postproc1

output:
  tee:
  - netcdf: 
      path: file.nc
      post_processors:
        - postproc2
  - grib: /path/to/grib/file.grib
       path: file.grib

Same idea here: postproc1 to both outputs, postproc2 only to the netcdf output.

On the question of what the order should be: top-level processors are applied to the model input/output, inner-level processors are applied to the data input/output.

Using the above as an example:

• pre_processors: read input from disk -> preproc2 -> preproc1 -> model input
• post_processors: model output -> postproc1 -> postproc2 -> write output to disk

I would be happy to receive your PR. Let me know if anything is unclear or if you want to discuss implementation. One thing to note is that all of this should be independent of the runner, it should work with any runner not just the cutout runner.