Analysis updates#11

Merged

ilaflott merged 72 commits intoNOAA-GFDL:mainfrom

ceblanton:147.catalogs-before-analysis

Feb 3, 2025

Commits on Jul 12, 2024

Remove old and unused bin/ scripts

Chris Blanton
authored and
Chris Blanton
committed
cylc gui can be run anywhere now

Chris Blanton
authored and
Chris Blanton
committed
Removed pp.yaml updates that are now in fre-cli

Chris Blanton
authored and
Chris Blanton
committed
Remove unused and broken mdtf task. mdtf can be run as an analysis script anyway

Chris Blanton
authored and
Chris Blanton
committed
#147 Insist the data-catalog task succeed

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jul 16, 2024

#147 build the catalog after saving chunk-b timeseries

Chris Blanton
authored and
Chris Blanton
committed
#147 Generate data catalog before running analysis scripts

Chris Blanton
authored and
Chris Blanton
committed
#147 Build analysis script environments, cylc graph and task definitions

Chris Blanton
authored and
Chris Blanton
committed
#147 fix; add build-analysis- tasks for the every-interval and defined-interval analysis

Chris Blanton
authored and
Chris Blanton
committed
#147 fix; add the data-catalog graph for the cumulative and every-interval analysis

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jul 17, 2024

#147 .csh for csh, .fish for fish, but nothing for bourne

Chris Blanton
authored and
Chris Blanton
committed
#147 add analysis publish tasks for cumulative analysis

Chris Blanton
authored and
Chris Blanton
committed

Commits on Aug 19, 2024

Merge remote-tracking branch 'gh/main' into 147.catalogs-before-analysis

Chris Blanton
authored and
Chris Blanton
committed

Commits on Aug 22, 2024

#147 Added initial cycle point dependence with ^ syntax

Chris Blanton
authored and
Chris Blanton
committed
#147 dummy publish-analysis environment

Chris Blanton
authored and
Chris Blanton
committed

Commits on Aug 30, 2024

#147 Put the build tasks in R1 and put the build dependence in the analysis intervals with [^] syntax

Chris Blanton
authored and
Chris Blanton
committed
Turn off epmt collection while epmt service is down

Chris Blanton
authored and
Chris Blanton
committed

Commits on Sep 4, 2024

#147 Plausible updates to graph and define build and publish tasks

Chris Blanton
authored and
Chris Blanton
committed
#147 Remove the draft analysis-install script from flow.cylc

Chris Blanton
authored and
Chris Blanton
committed
#147 Add install ray script

Chris Blanton
authored and
Chris Blanton
committed

Commits on Sep 23, 2024

Merge branch 'main' into 147.catalogs-before-analysis
ceblanton
authored
#147 For clarity print to stderr whether the build and publish steps are used

Chris Blanton
authored and
Chris Blanton
committed
Update the soon-to-be-retired app/analysis/rose-app.conf analysis collection

Chris Blanton
authored and
Chris Blanton
committed
#147 Use the correct Cylc job env variable

Chris Blanton
authored and
Chris Blanton
committed
#147 add dummy publish script

Chris Blanton
authored and
Chris Blanton
committed
Rename ray-analysis to cloud

Chris Blanton
authored and
Chris Blanton
committed
#137 analysis-clouds to use the cylc-installed python environment

Chris Blanton
authored and
Chris Blanton
committed
#147 add plausible new land analysis script from Anthony

Chris Blanton
authored and
Chris Blanton
committed

Commits on Sep 30, 2024

Set the stall timeout to 1 week; the default of one hour is way too short

Chris Blanton
authored and
Chris Blanton
committed
#147 include the full path to the publish script

Chris Blanton
authored and
Chris Blanton
committed

Commits on Oct 23, 2024

#147 Added rewritten analysis jinja methods

Chris Blanton
authored and
Chris Blanton
committed

Commits on Oct 29, 2024

#147 Update flow.cylc to use new get_analysis_info interface

Chris Blanton
authored and
Chris Blanton
committed

Commits on Oct 30, 2024

#147 updated interface to get_analysis_info

Chris Blanton
authored and
Chris Blanton
committed
#147 various fixes for rewritten get_analysis_info.py; passes cylc validation now

Chris Blanton
authored and
Chris Blanton
committed

Commits on Dec 17, 2024

Merge remote-tracking branch 'origin/main' into 147.catalogs-before-analysis

Chris Blanton
authored and
Chris Blanton
committed
Remove analysis scripts from repository

Chris Blanton
authored and
Chris Blanton
committed
Retrieve the combined yaml as <experiment>.yaml

Chris Blanton
authored and
Chris Blanton
committed

Commits on Dec 21, 2024

Remove analysis script configuartions from fre-workflows

Chris Blanton
authored and
Chris Blanton
committed
keep $FRE_ANALYSIS_HOME available to analysis scripts in a hard-coded way for now

Chris Blanton
authored and
Chris Blanton
committed
Parse time consistently, and some wip changes

Chris Blanton
authored and
Chris Blanton
committed
added install graph and definitions for one case

Chris Blanton
authored and
Chris Blanton
committed
Task graph for new-analysis cases every-chunk every-chunk-cumulative

Chris Blanton
authored and
Chris Blanton
committed

Commits on Dec 23, 2024

New analysis graphs should be working now

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 6, 2025

Graph updates for the 3 types of new-analysis scripts,
Chris Blanton
committed

Commits on Jan 7, 2025

Various fixes for legacy analysis scripts mostly

Chris Blanton
authored and
Chris Blanton
committed
Thanks to Ray's get_analysis_info updates, we can reduce these

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 8, 2025

Set inherited scripting for analysis-build tasks, and hard-code FRE-VERSION for now

Chris Blanton
authored and
Chris Blanton
committed
Fix typo on line continuation in system call

Chris Blanton
authored and
Chris Blanton
committed
Replace "canopy" with {{ FRE_VERSION }}

Chris Blanton
authored and
Chris Blanton
committed
Retry 'fre analysis build' tasks twice as it fails sometimes for unknown reasons but succeeds on retry

Chris Blanton
authored and
Chris Blanton
committed
plausible task definitions for new and legacy analysis scripts

Chris Blanton
authored and
Chris Blanton
committed
identation fixes

Chris Blanton
authored and
Chris Blanton
committed
ready for runtime testing

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 13, 2025

Remove default rose scripting for tasks that do not set script or env-script.

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 15, 2025

analysis tasks should load fre

Chris Blanton
authored and
Chris Blanton
committed
Updates for the defined-interval scripting

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 21, 2025

Various fixes

Chris Blanton
authored and
Chris Blanton
committed
Various fixes for analysis scripts

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 24, 2025

More legacy analysis adjustments

Chris Blanton
authored and
Chris Blanton
committed
a few more legacy analysis changes

Chris Blanton
authored and
Chris Blanton
committed
Add check for invalid analysis timeaverage requests.

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 25, 2025

Hard fail if analysis script requests non-available component, and add Bronx frequency mappings for AM5 scripts

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 27, 2025

Some analysis scripts expect a trailing slash for $in_data_dir

Chris Blanton
authored and
Chris Blanton
committed
Add databegyr/dataendyr legacy analysis template vars.

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 29, 2025

The analysis script jinja call-out must have access to both pp chunks,

Chris Blanton
authored and
Chris Blanton
committed
Update get-analysis-info callout to choose between the available pp chunks

Chris Blanton
authored and
Chris Blanton
committed
Workaround for data catalog race condition

Chris Blanton
authored and
Chris Blanton
committed

Commits on Jan 31, 2025

Retry new analysis scripts 10 times until we solve the intake-esm problem

Chris Blanton
authored and
Chris Blanton
committed
Merge remote-tracking branch 'origin/main' into 147.catalogs-before-analysis

Chris Blanton
authored and
Chris Blanton
committed
Remove unintended changes to regrid-xy and remap-pp-component rose-app.conf

Chris Blanton
authored and
Chris Blanton
committed
Undo unintended change to rose-suite.conf template

Chris Blanton
authored and
Chris Blanton
committed
Merge remote-tracking branch 'me/147.catalogs-before-analysis' into 147.catalogs-before-analysis

Chris Blanton
authored and
Chris Blanton
committed