[Ensemble Integration] Improve catalog-generator behavior for multiple realizations

[CostanzaD]

catalog-generator: possible improvements

The current implementation of catalog-generator technically works as intended:
it rewrites the source each time and allows specifying the number of realizations.

To make this work in the workflow, it requires explicitly setting regen_cat: true, check with Aina.

• add possibility to change the 'default' realization

However, this implementation could be improved in the following ways:

• In the longer term, we might consider avoiding full source rewrites and instead supporting incremental generation of extra realizations.
• This could be important if catalog generation becomes a bottleneck during ensemble runs.

Tasks

• Confirm current behavior is needed for all workflows
• Discuss possible implications of avoiding full rewrite
• Investigate support for extra realizations

[silviacaprioli]

I checked with @ainagaya : in the workflow, the regen_cat_key flag must be set to True for the catalog to be regenerated (its default is False).
Since catgen rewrites the entire catalog entry when called, we need to trigger it whenever a new realization is produced.
From the workflow perspective, this just means ensuring that regen_cat_key is set to True when handling ensemble, no further changes are needed.

Regarding the default_realization option, I already opened PR #2058 , but I'm starting to think it might not be a great idea to include it. I suppose it's quite a rare case for someone to want to start e.g. from realization 2 instead of 1 simply because the first hasn't been run yet.
Also, we’d need to ask the workflow team to support it explicitly in their config files, which may not be worth the added complexity...
@mnurisso @oloapinivad what do you think?

[mnurisso]

About the default realization I don't think I agree, what we see in this cycle is that the number of members can shrink or change based on the resources availability. For this reason they may not run always in the same experiment. Being able to set a new default if a new one is run in a new expid I think it is a useful feature. I think in general, if well documented, an extra feature is always better to an hardcoded solution