-
Notifications
You must be signed in to change notification settings - Fork 204
Revamp obs staging and analysis stats job #4306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
| COMOUT_AERO_ANLMON:COM_CHEM_ANLMON_TMPL \ | ||
| COMOUT_SNOW_ANLMON:COM_SNOW_ANLMON_TMPL | ||
|
|
||
| mkdir -m 755 -p "${COMOUT_ATMOS_ANALYSIS}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should only mkdir these if these components are active in the experiment
| "correction_increment.yaml", | ||
| "ensemble_recenter.yaml"] %} | ||
| {% else %} | ||
| {% set da_stat_files = ["stat.atm.tar"]%} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
these are probably the old GSI files and not IODA, right?
| {% endfor %} | ||
|
|
||
| {% if DO_JEDISNOWDA %} | ||
| - "{{ COMIN_SNOW_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}snow_analysis.ioda_hofx.ensmean.tar" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I vaguely remember @aerorahul mentioning we should drop the gzipping
| ARCDIR ~ "/pgbanl." ~ RUN ~ "." ~ cycle_YMDH ~ ".grib2"]) %} | ||
|
|
||
| {% if DO_JEDIATMVAR == True %} | ||
| {% do det_anl_files.append([COMIN_ATMOS_ANALYSIS ~ "/" ~ head ~ "stat.atm.tar", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same as before, this might be GSI not JEDI/IODA
Description
This PR makes changes in two areas of GW code for JEDI jobs.
First, it makes changes to
ush/python/pygfs/jedi/jedi.pyrelated to observations handling that does the following:stage_observations()method for theJediclass that stages observations for analysis jobs, rather than relying on the task config YAML in GDASApp for staging. This change is justified by the fact that obs staging is essentially the same across all analysis tasks.Analysisclass into theJediclass for the following reason:Jediclass. This ensures that the file structure and naming for obs and their statistics are consistent between how they are staged in GW and how they are stages and saved by JEDI applications. Thus there can never be naming conflicts.Jediobject is created by the class constructor rather than theinitializeclass. This way,task_configdoesn't need to be passed to both the class constructor and theinitializeclass. One benefit of this is that it cuts down on the number of timestask_configis dumped by the logger.Jediclass code such as hardening and more descriptive method/variable naming.Second,
ush/python/pygfs/task/analysis_stats.pyis refactored in the following ways:AnalysisStatsclass now inherits fromAnalysisrather thanTask. This allows it to inherit parameters likeAPREFIX,GPREFIX, etc.task_configis never modified after the class constructor is run, consistent now with all other tasks.data_inanddata_outkeys).Jediobject, to ensure that file paths and naming are consistent between the GW code and JEDI application configuration YAMLs.Resolves #4224
Resolves #4228
Type of change
Change characteristics
How has this been tested?
Clone, build, and full CI suite on Hera
Checklist