Update push analysis for ensembles

[jhardenberg]

PR description:

Extends push_analysis and related tools to deal with ensembles.
For compatibility for now the functionality is implemented through a -e flag which will be removed some day when not necessary any more.

Tasks

• Update make-content.py to support the new folder layout
• Modify push_analysis.py to account for realization in folder structure and config
• Modify pdf_to_png.sh
• Extend support to the script that runs multiple push analysis (submit-aqua-web)

Issues closed by this pull request:

Close #2048

---

• Documentation is included if a new feature is included.
• Docstrings are updated if needed.
• Changelog is updated.

[jhardenberg]

"include realization in the generated experiment.yaml" is not needed since realization can be derived from the directory.

[jhardenberg]

I still have to test comprehensively the whole machinery but the main infrastructure is in place. I tried to maintain it still compatible with the old structure (if nothing else is specified the tools keep working exactly as before). The new options are:

• push_analysis.sh: now accepts -e/--ensemble to assume the new 4-level directory structure and support ensembles
• make_contents.py (called by the former one): in case of ensembles it is called with --ensemble telling it to expect an ensemble--style new directory structure. It will create an experiments.yaml file which has a structure like {catalog: model: exp: [realization]} instead of {catalog: model: [exp]}. I made some effort for make_contents.py to be forgiving and compatible: if it realizes that we are trying to add an esnsemble experiment to an experiments.yaml file still with the old structure it will convert it to the new one assuming realization r1 for the single ones.
• submit-aqua-web.py: now accepts --ensemble and --realization <realization> if single exp is specified. Consequently it will expect a fourth coulumn in the experiment list. It calls aqua-analysis.py specifying --realization
• pdf_to_png.sh(called by push_analysis.sh): now can deal with a 4-level directory structure

Due to compatibility there is some extra code to accomodate both old and new cases. When in the next months all our experiment output will be stored with the new structure we can make a PR to simplify

[jhardenberg]

I did an integrated test, I guess it is time that somebody gives a look

[jhardenberg]

I will merge the main to implement #2074 and test again

[mnurisso]

ok, I'm going to merge ecmean and then I'll check this PR for merge

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.43%. Comparing base (b018534) to head (a7bbada).
Report is 2 commits behind head on main.

✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2070      +/-   ##
==========================================
+ Coverage   77.41%   77.43%   +0.02%     
==========================================
  Files          77       77              
  Lines        7761     7774      +13     
  Branches     1467     1467              
==========================================
+ Hits         6008     6020      +12     
- Misses       1175     1176       +1     
  Partials      578      578