Skip to content

Conversation

@DavidNew-NOAA
Copy link
Contributor

@DavidNew-NOAA DavidNew-NOAA commented Dec 9, 2025

Description

This PR makes changes in two areas of GW code for JEDI jobs.

First, it makes changes to ush/python/pygfs/jedi/jedi.py related to observations handling that does the following:

  1. It creates a stage_observations() method for the Jedi class that stages observations for analysis jobs, rather than relying on the task config YAML in GDASApp for staging. This change is justified by the fact that obs staging is essentially the same across all analysis tasks.
  2. Methods for staging, extracting, taring bias corrections, and saving obs diags and radiative bias corrections are moved from the the Analysis class into the Jedi class for the following reason:
  3. The paths, file prefixes, and file suffixes used in these methods for obs, obs diags, and bias corrections are taken from the JCB config dictionary in the Jedi class. This ensures that the file structure and naming for obs and their statistics are consistent between how they are staged in GW and how they are stages and saved by JEDI applications. Thus there can never be naming conflicts.
  4. The JCB config dictionary for a Jedi object is created by the class constructor rather than the initialize class. This way, task_config doesn't need to be passed to both the class constructor and the initialize class. One benefit of this is that it cuts down on the number of times task_config is dumped by the logger.
  5. Other minor changes are made to the Jedi class code such as hardening and more descriptive method/variable naming.

Second, ush/python/pygfs/task/analysis_stats.py is refactored in the following ways:

  1. AnalysisStats class now inherits from Analysis rather than Task. This allows it to inherit parameters like APREFIX, GPREFIX, etc.
  2. Changes are made so that task_config is never modified after the class constructor is run, consistent now with all other tasks.
  3. The "base config" and "JEDI config" YAMLs are consolidated into a single master config YAML (like all other task now), and any staging/saving that was carried on in the original Python code is now invoked by the FileHandler with that master YAML (data_in and data_out keys).
  4. The run directory is reorganized by analysis type with subdirectories for the inputs and outputs.
  5. Input and output paths for staging/saving are taken from the JCB config dictionary of the relevant Jedi object, to ensure that file paths and naming are consistent between the GW code and JEDI application configuration YAMLs.

Resolves #4224
Resolves #4228

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this change expected to change outputs (e.g. value changes to existing outputs, new files stored in COM, files removed from COM, filename changes, additions/subtractions to archives)? YES/NO (If YES, please indicate to which system(s))
    • GFS
    • GEFS
    • SFS
    • GCAFS
  • Is this a breaking change (a change in existing functionality)? YES
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

Clone, build, and full CI suite on Hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

COMOUT_AERO_ANLMON:COM_CHEM_ANLMON_TMPL \
COMOUT_SNOW_ANLMON:COM_SNOW_ANLMON_TMPL

mkdir -m 755 -p "${COMOUT_ATMOS_ANALYSIS}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should only mkdir these if these components are active in the experiment

"correction_increment.yaml",
"ensemble_recenter.yaml"] %}
{% else %}
{% set da_stat_files = ["stat.atm.tar"]%}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are probably the old GSI files and not IODA, right?

{% endfor %}

{% if DO_JEDISNOWDA %}
- "{{ COMIN_SNOW_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx.ensmean.tar"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I vaguely remember @aerorahul mentioning we should drop the gzipping

ARCDIR ~ "/pgbanl." ~ RUN ~ "." ~ cycle_YMDH ~ ".grib2"]) %}

{% if DO_JEDIATMVAR == True %}
{% do det_anl_files.append([COMIN_ATMOS_ANALYSIS ~ "/" ~ head ~ "stat.atm.tar",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as before, this might be GSI not JEDI/IODA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create stage_obs method for Jedi class anlstat job hardening and refactoring

2 participants