[Feature] Gracefully handle saving OmegaConf configs containing non-primitives

[wizeng23]

Feature request

Our OmegaConfigs have fields for kwargs that can contain arbitrary values. When we try to save these as yaml files for logging purposes here:

oumi/src/oumi/train.py

Line 232 in a59c9d7

config.to_yaml(str(telemetry_dir / "training_config.yaml"))

They will trigger an error:

UnsupportedValueType: Value 'function' is not a supported primitive type
    full_key: trainer_kwargs.preprocess_logits_for_metrics
    reference_type=Dict[str, Any]
    object_type=dict

We should gracefully handle this. For example, we can parse through the omegaconf and find non-primitive fields, warn when trying to log these fields, and remove them. However, there is a greater question on how to ensure reproducibility of our jobs if the saved YAML configs of jobs don't contain all the required information to reproduce them.

Motivation / references

See #1623 for context

Your contribution

Happy to advise

[abhiramvad]

Raised a pull request with suggested changes to this issue #1684