Handle inconsistent encoder & processor inputs

[mo-mburnand]

Is your feature request related to a problem? Please describe.

A recent use case of mine, I have wanted to take a fine-tuned model but adapt it to be used with a different analyses/dataset, which has only a subset of the variables that the original model was trained on. In my case it's deemed important to use the full level set as well.

My plan would be to freeze the processor and train a new encoder/decoder that can hopefully learn to map from a datasets with more levels & reduced variables to the latent representation that is expected of the processor.

To my understanding however, if transfer_learning = True, the compare_variables function catches a mismatch in request & model variables.

But if transfer_learning = False, it loads the model from checkpoint and ultimately fails on an index error when it instantiates a normaliser for the processor because the mapper is longer than the data statistics loaded.

I believe they would both also fail at trainer. fit() because of a tensor size mismatch.

Describe the solution you'd like

I assume it's important in most use cases to enforce checks, that the variables requested are consistent with the loaded model. Therefore would a config flag to turn off these checks make sense?

Any input from others appreciated.

Describe alternatives you've considered

Another options could be to train an entirely new model on reduced variables & more levels but that feels like wasted compute.

Additional context

No response

Organisation

Met Office

[anaprietonem]

@dietervdb-meteo - related to the normalizer, I remember you raised something similar a while back? About how we handle the pre/post-processors states when transferring checkpoints?

@JesperDramsch feel free to chime in, any ideas how we could make the callback for checking the variables more flexible?

[dietervdb-meteo]

Making transfer learning both more flexible and more controlled/configurable I think is on the wish list of many people. Issue #373 was an early overview but quite outdated, a more recent proposals with a similar flavor is #783 , I think also the option to transfer learn with different variables (or even datasets given the new multi-dataset functionality) would be very nice. I think we need to first collect what features and use cases are desired and then somehow see how to implement them.