Skip to content

Revamp obs staging and analysis stats job #4306

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
DavidNew-NOAA wants to merge 31 commits into
base: develop
Choose a base branch
from
Draft

Revamp obs staging and analysis stats job #4306

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
78a0279
Initial commit
DavidNew-NOAA Nov 5, 2025
8fdd557
Update gdas hash
DavidNew-NOAA Nov 5, 2025
d5c4519
Save changes
DavidNew-NOAA Nov 6, 2025
f32e97f
Debug
DavidNew-NOAA Nov 6, 2025
0b62b7e
mkdir obs, diags, and bc in stage_obs()
DavidNew-NOAA Nov 6, 2025
a8fc75a
Attempt to fix weird pynorms
DavidNew-NOAA Nov 7, 2025
b061def
pynorms?
DavidNew-NOAA Nov 7, 2025
a771101
Tinkering
DavidNew-NOAA Nov 7, 2025
f623e6d
Update
DavidNew-NOAA Nov 13, 2025
ce8a749
Clean up
DavidNew-NOAA Nov 13, 2025
2524219
Slight name change
DavidNew-NOAA Nov 13, 2025
a690382
Debug
DavidNew-NOAA Nov 14, 2025
0185dc6
Debug
DavidNew-NOAA Nov 21, 2025
df6a972
Merge branch 'develop' into feature/stage-obs
DavidNew-NOAA Nov 21, 2025
9c83b2f
pynorms
DavidNew-NOAA Nov 21, 2025
a425b44
Update
DavidNew-NOAA Nov 21, 2025
619bb93
Update
DavidNew-NOAA Dec 1, 2025
68f4390
debug
DavidNew-NOAA Dec 1, 2025
cb40af6
Merge remote-tracking branch 'origin/develop' into feature/stage-obs
DavidNew-NOAA Dec 1, 2025
b19b8d3
Debug and new features
DavidNew-NOAA Dec 3, 2025
e62a3f9
Missed files in last commit
DavidNew-NOAA Dec 3, 2025
39c4e46
Move diag files and bias corrections to come with JEDI class methods
DavidNew-NOAA Dec 4, 2025
f305ccc
Some logs, etc
DavidNew-NOAA Dec 4, 2025
37e8d0e
Minor change
DavidNew-NOAA Dec 4, 2025
e46c347
Debug
DavidNew-NOAA Dec 8, 2025
f8486f9
Missing files
DavidNew-NOAA Dec 8, 2025
cc8f48e
Merge branch 'develop' into feature/stage-obs
DavidNew-NOAA Dec 8, 2025
8f42352
Revamp somethings
DavidNew-NOAA Dec 8, 2025
6289e2b
Merge branch 'develop' into feature/stage-obs
DavidNew-NOAA Dec 8, 2025
35b2088
debug
DavidNew-NOAA Dec 9, 2025
cd8e70e
Merge branch 'develop' into feature/stage-obs
DavidNew-NOAA Dec 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions dev/jobs/JGLOBAL_ANALYSIS_STATS
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,15 @@ YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COMIN_OBS:COM_OBS_TMPL \
COMOUT_AERO_ANLMON:COM_CHEM_ANLMON_TMPL \
COMOUT_SNOW_ANLMON:COM_SNOW_ANLMON_TMPL

mkdir -m 755 -p "${COMOUT_ATMOS_ANALYSIS}"
Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should only mkdir these if these components are active in the experiment

mkdir -m 755 -p "${COMOUT_OCEAN_ANALYSIS}"
mkdir -m 755 -p "${COMOUT_AERO_ANALYSIS}"
mkdir -m 755 -p "${COMOUT_SNOW_ANALYSIS}"
mkdir -m 755 -p "${COMOUT_ATMOS_ANLMON}"
mkdir -m 755 -p "${COMOUT_OCEAN_ANLMON}"
mkdir -m 755 -p "${COMOUT_AERO_ANLMON}"
mkdir -m 755 -p "${COMOUT_SNOW_ANLMON}"

###############################################################
# Run relevant script

Expand Down
4 changes: 2 additions & 2 deletions dev/parm/config/gcafs/config.anlstat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ echo "BEGIN: config.anlstat"
# Get task specific resources
source "${EXPDIR}/config.resources" anlstat

export JEDI_CONFIG_YAML="${PARMgfs}/gdas/anlstat/anlstat_jedi_config.yaml.j2"
export BASE_CONFIG_YAML="${PARMgfs}/gdas/anlstat/anlstat_base_config.yaml.j2"
export OBS_LIST_YAML="${PARMgfs}/gdas/anlstat/anlstat_obs_list.yaml.j2"
export TASK_CONFIG_YAML="${PARMgfs}/gdas/anlstat/anlstat_config.yaml.j2"

echo "END: config.anlstat"
4 changes: 2 additions & 2 deletions dev/parm/config/gfs/config.anlstat → dev/parm/config/gfs/config.anlstat.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ echo "BEGIN: config.anlstat"
# Get task specific resources
source "${EXPDIR}/config.resources" anlstat

export JEDI_CONFIG_YAML="${PARMgfs}/gdas/anlstat/anlstat_jedi_config.yaml.j2"
export BASE_CONFIG_YAML="${PARMgfs}/gdas/anlstat/anlstat_base_config.yaml.j2"
export OBS_LIST_YAML="${PARMgfs}/gdas/anlstat/anlstat_obs_list.yaml.j2"
export TASK_CONFIG_YAML="${PARMgfs}/gdas/anlstat/anlstat_config.yaml.j2"

echo "END: config.anlstat"
27 changes: 14 additions & 13 deletions dev/scripts/exglobal_analysis_stats.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,23 +20,24 @@
# Take configuration from environment and cast it as python dictionary
config = cast_strdict_as_dtypedict(os.environ)

# Instantiate the atm analysis task
AnlStats = AnalysisStats(config)

# Create list based on DA components
AnlStats.task_config['STAT_ANALYSES'] = []
if AnlStats.task_config.DO_AERO_ANL:
AnlStats.task_config['STAT_ANALYSES'].append('aero')
if AnlStats.task_config.DO_JEDISNOWDA:
AnlStats.task_config['STAT_ANALYSES'].append('snow')
if AnlStats.task_config.DO_JEDIATMVAR:
AnlStats.task_config['STAT_ANALYSES'].append('atmos')
config.STAT_ANALYSES = []
if config.DO_AERO_ANL:
config.STAT_ANALYSES.append('aero')
if config.DO_JEDISNOWDA:
config.STAT_ANALYSES.append('snow')
if config.DO_JEDIATMVAR:
config.STAT_ANALYSES.append('atmos')
else:
AnlStats.task_config['STAT_ANALYSES'].append('atmos_gsi')
AnlStats.convert_gsi_diags()
config.STAT_ANALYSES.append('atmos_gsi')

# Instantiate the analysis stats task
AnlStats = AnalysisStats(config)

# Initialize JEDI variational analysis
if not config.DO_JEDIATMVAR:
AnlStats.convert_gsi_diags()
AnlStats.initialize()
for anl in AnlStats.task_config.STAT_ANALYSES:
for anl in config.STAT_ANALYSES:
AnlStats.execute(anl)
AnlStats.finalize(anl)
4 changes: 2 additions & 2 deletions parm/archive/enkf.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ enkf:
"correction_increment.yaml",
"ensemble_recenter.yaml"] %}
{% else %}
{% set da_stat_files = ["stat.atm.tar"]%}
Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these are probably the old GSI files and not IODA, right?

{% set da_stat_files = ["atm_analysis.ioda_hofx.tar.gz"]%}
{% set da_conf_files = ["atmensanlletkf.yaml",
"atmensanlfv3inc.yaml",
"correction_increment.yaml",
Expand All @@ -98,7 +98,7 @@ enkf:
{% endfor %}

{% if DO_JEDISNOWDA %}
- "{{ COMIN_SNOW_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx.ensmean.tar"
Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I vaguely remember @aerorahul mentioning we should drop the gzipping

- "{{ COMIN_SNOW_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx.ensmean.tar.gz"
{% for itile in range(1,7) %}
# Snow analysis is 3dvar
- "{{ COMIN_SNOW_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ cycle_YMD }}.{{ cycle_HH }}0000.snow_increment.sfc_data.tile{{ itile }}.nc"
Expand Down
2 changes: 1 addition & 1 deletion parm/archive/gcafs_arcdir.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@
{% set det_anl_files = [] %}

{% if DO_AERO_ANL %}
{% do det_anl_files.append([COMIN_CHEM_ANALYSIS ~ "/" ~ head ~ "aerostat.tgz",
{% do det_anl_files.append([COMIN_CHEM_ANALYSIS ~ "/" ~ head ~ "aero_analysis.ioda_hofx.tar.gz",
ARCDIR ~ "/aerostat." ~ RUN ~ "." ~ cycle_YMDH ~ ".tgz"]) %}
{% endif %}

Expand Down
6 changes: 3 additions & 3 deletions parm/archive/gdas.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,7 @@ gdas:
{% if DO_JEDIATMVAR %}
- "{{ COMIN_CONF | relpath(ROTDIR) }}/{{ head }}anlvar.atm.yaml"
- "{{ COMIN_CONF | relpath(ROTDIR) }}/{{ head }}anlvar.fv3.atm.yaml"
- "{{ COMIN_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ head }}stat.atm.tar"
- "{{ COMIN_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ head }}atm_analysis.ioda_hofx.tar.gz"
- "{{ COMIN_ATMOS_ANLMON | relpath(ROTDIR) }}/{{ head }}atmos_analysis.ioda_hofx_stats.tar.gz"
{% else %}
- "{{ COMIN_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ head }}gsistat.txt"
Expand All @@ -96,7 +96,7 @@ gdas:
- "{{ COMIN_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ head }}abias_int.txt"
{% endif %}
{% if DO_AERO_ANL %}
- "{{ COMIN_CHEM_ANALYSIS | relpath(ROTDIR) }}/{{ head }}aerostat.tgz"
- "{{ COMIN_CHEM_ANALYSIS | relpath(ROTDIR) }}/{{ head }}aero_analysis.ioda_hofx.tar.gz"
- "{{ COMIN_CONF | relpath(ROTDIR) }}/{{ head }}aerovar.yaml"
{% for itile in range(1,7) %}
- "{{ COMIN_CHEM_ANALYSIS | relpath(ROTDIR) }}/aeroinc.{{ cycle_YMD }}.{{ cycle_HH }}0000.fv_tracer.res.tile{{ itile }}.nc"
Expand All @@ -112,7 +112,7 @@ gdas:
{% for itile in range(1,7) %}
- "{{ COMIN_SNOW_ANALYSIS | relpath(ROTDIR) }}/{{ current_cycle | to_fv3time }}.snow_increment.sfc_data.tile{{ itile }}.nc"
{% endfor %}
- "{{ COMIN_SNOW_ANALYSIS | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx.tar"
- "{{ COMIN_SNOW_ANALYSIS | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx.tar.gz"
- "{{ COMIN_CONF | relpath(ROTDIR) }}/{{ head }}snowanlvar.yaml"
- "{{ COMIN_SNOW_ANLMON | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx_stats.tar.gz"
{% endif %}
Expand Down
8 changes: 4 additions & 4 deletions parm/archive/gfs_arcdir.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,20 +38,20 @@
ARCDIR ~ "/pgbanl." ~ RUN ~ "." ~ cycle_YMDH ~ ".grib2"]) %}

{% if DO_JEDIATMVAR == True %}
{% do det_anl_files.append([COMIN_ATMOS_ANALYSIS ~ "/" ~ head ~ "stat.atm.tar",
Copy link
Contributor

@CoryMartin-NOAA CoryMartin-NOAA Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as before, this might be GSI not JEDI/IODA

{% do det_anl_files.append([COMIN_ATMOS_ANALYSIS ~ "/" ~ head ~ "atm_analysis.ioda_hofx.tar.gz",
ARCDIR ~ "/atmstat." ~ RUN ~ "." ~ cycle_YMDH ]) %}
{% else %}
{% do det_anl_files.append([COMIN_ATMOS_ANALYSIS ~ "/" ~ head ~ "gsistat.txt",
ARCDIR ~ "/gsistat." ~ RUN ~ "." ~ cycle_YMDH ]) %}
{% endif %}

{% if DO_JEDISNOWDA == True %}
{% do det_anl_files.append([COMIN_SNOW_ANALYSIS ~ "/" ~ head ~ "snow_analysis.ioda_hofx.tar",
{% do det_anl_files.append([COMIN_SNOW_ANALYSIS ~ "/" ~ head ~ "snow_analysis.ioda_hofx.tar.gz",
ARCDIR ~ "/snowstat." ~ RUN ~ "." ~ cycle_YMDH ~ ".tar"]) %}
{% endif %}

{% if DO_AERO_ANL %}
{% do det_anl_files.append([COMIN_CHEM_ANALYSIS ~ "/" ~ head ~ "aerostat.tgz",
{% do det_anl_files.append([COMIN_CHEM_ANALYSIS ~ "/" ~ head ~ "aero_analysis.ioda_hofx.tar.gz",
ARCDIR ~ "/aerostat." ~ RUN ~ "." ~ cycle_YMDH ~ ".tgz"]) %}
{% endif %}

Expand Down Expand Up @@ -135,7 +135,7 @@
# Ensemble analysis files
{% set enkf_files = [] %}
{% if DO_JEDIATMENS == True %}
{% do enkf_files.append([COMIN_ATMOS_ANALYSIS_ENSSTAT ~ "/" ~ head ~ "stat.atm.tar",
{% do enkf_files.append([COMIN_ATMOS_ANALYSIS_ENSSTAT ~ "/" ~ head ~ "atm_analysis.ioda_hofx.ens_mean.tar.gz",
ARCDIR ~ "/atmensstat." ~ RUN ~ "." ~ cycle_YMDH ]) %}
{% else %}
{% do enkf_files.append([COMIN_ATMOS_ANALYSIS_ENSSTAT ~ "/" ~ head ~ "enkfstat.txt",
Expand Down
6 changes: 3 additions & 3 deletions parm/archive/gfsa.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,12 +37,12 @@ gfsa:
{% if DO_JEDIATMVAR %}
- "{{ COMIN_CONF | relpath(ROTDIR) }}/{{ head }}anlvar.atm.yaml"
- "{{ COMIN_CONF | relpath(ROTDIR) }}/{{ head }}anlvar.fv3.atm.yaml"
- "{{ COMIN_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ head }}stat.atm.tar"
- "{{ COMIN_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ head }}atm_analysis.ioda_hofx.tar.gz"
{% else %}
- "{{ COMIN_ATMOS_ANALYSIS | relpath(ROTDIR) }}/{{ head }}gsistat.txt"
{% endif %}
{% if DO_AERO_ANL %}
- "{{ COMIN_CHEM_ANALYSIS | relpath(ROTDIR) }}/{{ head }}aerostat.tgz"
- "{{ COMIN_CHEM_ANALYSIS | relpath(ROTDIR) }}/{{ head }}aero_analysis.ioda_hofx.tar.gz"
{% endif %}
{% if DO_PREP_OBS_AERO %}
- "{{ COMIN_OBS | relpath(ROTDIR) }}/{{ head }}aeroobs"
Expand All @@ -54,7 +54,7 @@ gfsa:
{% for itile in range(1,7) %}
- "{{ COMIN_SNOW_ANALYSIS | relpath(ROTDIR) }}/{{ current_cycle | to_fv3time }}.snow_increment.sfc_data.tile{{ itile }}.nc"
{% endfor %}
- "{{ COMIN_SNOW_ANALYSIS | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx.tar"
- "{{ COMIN_SNOW_ANALYSIS | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx.tar.gz"
- "{{ COMIN_CONF | relpath(ROTDIR) }}/{{ head }}snowanlvar.yaml"
{% endif %}

Expand Down
2 changes: 1 addition & 1 deletion parm/archive/master_gcdas.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ datasets:
- "logs/{{ cycle_YMDH }}/{{ RUN }}_aeroanlinit.log"
- "logs/{{ cycle_YMDH }}/{{ RUN }}_aeroanlvar.log"
- "logs/{{ cycle_YMDH }}/{{ RUN }}_aeroanlfinal.log"
- "{{ COMIN_CHEM_ANALYSIS | relpath(ROTDIR) }}/{{ head }}aerostat.tgz"
- "{{ COMIN_CHEM_ANALYSIS | relpath(ROTDIR) }}/{{ head }}aero_analysis.ioda_hofx.tar.gz"
- "{{ COMIN_CONF | relpath(ROTDIR) }}/{{ head }}aerovar.yaml"
{% for itile in range(1,7) %}
- "{{ COMIN_CHEM_ANALYSIS | relpath(ROTDIR) }}/aeroinc.{{ cycle_YMD }}.{{ cycle_HH }}0000.fv_tracer.res.tile{{ itile }}.nc"
Expand Down
2 changes: 1 addition & 1 deletion sorc/gdas.cd
Open in desktop
Submodule gdas.cd updated 52 files
+1 −1 .github/pull_request_template.md
+1 −0 ci/ci_tests.sh
+10 −6 ci/driver.sh
+10 −0 ci/gaeac6.sh
+8 −1 ci/run_ci.sh
+1 −1 ci/stable_driver.sh
+3 −3 modulefiles/GDAS/noaacloud.intel.lua
+2 −2 parm/aero/aero_bmat_config.yaml.j2
+2 −10 parm/aero/aero_det_config.yaml.j2
+1 −1 parm/aero/aero_det_inc_vars.yaml
+0 −6 parm/aero/aero_stage_obs.yaml.j2
+0 −2 parm/aero/jcb-base.yaml.j2
+2 −2 parm/analcalc/analcalc_config.yaml.j2
+7 −9 parm/anlstat/aero/jcb-base.yaml.j2
+0 −60 parm/anlstat/anlstat_base_config.yaml.j2
+45 −0 parm/anlstat/anlstat_config.yaml.j2
+0 −28 parm/anlstat/anlstat_jedi_config.yaml.j2
+19 −0 parm/anlstat/anlstat_obs_list.yaml.j2
+7 −9 parm/anlstat/atmos/jcb-base.yaml.j2
+7 −9 parm/anlstat/atmos_gsi/jcb-base.yaml.j2
+7 −9 parm/anlstat/snow/jcb-base.yaml.j2
+2 −10 parm/atm/atm_det_config.yaml.j2
+2 −2 parm/atm/atm_ecen_config.yaml.j2
+3 −11 parm/atm/atm_ens_config.yaml.j2
+0 −6 parm/atm/atm_stage_obs.yaml.j2
+0 −2 parm/atm/jcb-base.yaml.j2
+1 −1 parm/jcb-algorithms
+1 −1 parm/jcb-gdas
+3 −3 parm/marine/marine_bmat_config.yaml.j2
+3 −10 parm/marine/marine_det_config.yaml.j2
+1 −1 parm/marine/marine_ecen_config.yaml.j2
+0 −7 parm/marine/marine_ens_config.yaml.j2
+0 −1 parm/snow/jcb-base.yaml.j2
+2 −3 parm/snow/prep/prep_ims.yaml.j2
+2 −5 parm/snow/snow_det_config.yaml.j2
+3 −6 parm/snow/snow_ens_config.yaml.j2
+1 −1 sorc/da-utils
+1 −1 sorc/ioda
+1 −1 sorc/oops
+1 −1 sorc/saber
+1 −1 sorc/ufo
+2 −2 test/atm/global-workflow/jjob_ens_final.sh
+2 −2 test/atm/global-workflow/jjob_ens_inc.sh
+2 −2 test/atm/global-workflow/jjob_ens_init.sh
+2 −2 test/atm/global-workflow/jjob_ens_init_split.sh
+3 −2 test/atm/global-workflow/jjob_ens_letkf.sh
+2 −2 test/atm/global-workflow/jjob_ens_obs.sh
+2 −2 test/atm/global-workflow/jjob_ens_sol.sh
+2 −2 test/atm/global-workflow/jjob_var_final.sh
+2 −2 test/atm/global-workflow/jjob_var_inc.sh
+2 −2 test/atm/global-workflow/jjob_var_init.sh
+2 −2 test/atm/global-workflow/jjob_var_run.sh
Loading
Loading