Refactor cleanup task and add checks for gempak files #4258

DavidHuber-NOAA · 2025-11-20T17:40:49Z

Description

This adds more specific checks for gempak files and makes an attempt at making the cleanup job less disk-intensive by running a single find -f command.

Resolves #4271
Resolves #4166

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

Will test on an extended test.

Checklist

scripts/exglobal_cleanup.sh

DavidHuber-NOAA · 2025-11-20T17:53:13Z

FYI @CatherineThomas-NOAA @JessicaMeixner-NOAA @aerorahul. I will test this on an extended case with spoofed older COM structures.

…/global-workflow into feature/save_gempak_1p00

scripts/exgfs_atmos_postsnd.sh

ush/gfs_sndp.sh

ush/gfs_bfr2gpk.sh

dev/parm/config/gfs/config.cleanup

emcbot · 2025-12-03T19:24:15Z

C96_atm3DVar FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C96_atm3DVar_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C96_atm3DVar_da19eb69-6066/logs/2021122100/gdas_arch_tar_gdas_restarta.log

View Error Logs: (gdas_arch_tar_gdas_restarta.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2025-12-03T19:42:31Z

C48_ATM FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C48_ATM_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48_ATM_da19eb69-6066/logs/2021032312/gfs_arch_tar_gfsb.log

View Error Logs: (gfs_arch_tar_gfsb.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2025-12-03T19:43:03Z

C96C48_hybatmsnowDA FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C96C48_hybatmsnowDA_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C96C48_hybatmsnowDA_da19eb69-6066/logs/2021122018/enkfgdas_earc_tars_01.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C96C48_hybatmsnowDA_da19eb69-6066/logs/2021122018/gdas_arch_tar_gdas_restartb.log

View Error Logs: (enkfgdas_earc_tars_01.log) (gdas_arch_tar_gdas_restartb.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2025-12-03T19:43:07Z

C96C48mx500_S2SW_cyc_gfs FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C96C48mx500_S2SW_cyc_gfs_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C96C48mx500_S2SW_cyc_gfs_da19eb69-6066/logs/2021122018/enkfgfs_earc_tars_01.log

View Error Logs: (enkfgfs_earc_tars_01.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2025-12-03T19:49:12Z

C48_S2SW FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C48_S2SW_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48_S2SW_da19eb69-6066/logs/2021032312/gfs_arch_tar_gfsa.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48_S2SW_da19eb69-6066/logs/2021032312/gfs_arch_tar_gfsb.log

View Error Logs: (gfs_arch_tar_gfsa.log) (gfs_arch_tar_gfsb.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2025-12-03T20:07:39Z

C96C48_hybatmsoilDA FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C96C48_hybatmsoilDA_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C96C48_hybatmsoilDA_da19eb69-6066/logs/2022051512/enkfgdas_earc_tars_01.log

View Error Logs: (enkfgdas_earc_tars_01.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2025-12-03T20:19:47Z

C48_S2SWA_gefs FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C48_S2SWA_gefs_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48_S2SWA_gefs_da19eb69-6066/logs/2021032312/gefs_arch_tars.log

View Error Logs: (gefs_arch_tars.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2025-12-03T20:32:07Z

C96C48_hybatmDA FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C96C48_hybatmDA_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C96C48_hybatmDA_da19eb69-6066/logs/2021122100/gfs_arch_tar_gfs_netcdfb.log

View Error Logs: (gfs_arch_tar_gfs_netcdfb.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

emcbot · 2025-12-03T20:38:56Z

C48mx500_3DVarAOWCDA FAILED on Gaeac6 (pipeline ID: 6066)

In directory: /gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/EXPDIR/C48mx500_3DVarAOWCDA_da19eb69-6066

Error Log Files:


/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_da19eb69-6066/logs/2021032500/gfs_arch_tar_gfs_flux_1p00.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_da19eb69-6066/logs/2021032500/gfs_arch_tar_gfs_netcdfb.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_da19eb69-6066/logs/2021032500/gfs_arch_tar_gfsa.log
/gpfs/f6/drsa-precip3/world-shared/global/CI/GITLAB/pr_cases_4258_da19eb69_6066/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_da19eb69-6066/logs/2021032500/gfs_arch_tar_ocean_grib2.log

View Error Logs: (gfs_arch_tar_gfs_flux_1p00.log) (gfs_arch_tar_gfs_netcdfb.log) (gfs_arch_tar_gfsa.log) (gfs_arch_tar_ocean_grib2.log)

This failure was detected automatically by global-workflow's CI/CD Pipeline

This test is covered by C48_S2SW_extended

DavidHuber-NOAA · 2025-12-09T18:33:55Z

@CatherineThomas-NOAA @aerorahul Could you please review this PR?

dev/ci/cases/pr/C48_S2SW.yaml

parm/archive/gfs_downstream.yaml.j2

ush/gfs_sndp.sh

ChristopherHill-NOAA

The latest test of the workflow - specifically modifying GEMPAK data file cleanup through case C96_atm3DVar_extended_gempak3 - provided the desired reduction in the retention of past cycle GEMPAK data files while retaining those GEMPAK files needed by JGFS_ATMOS_GEMPAK_META, thereby providing resolution to #4166.

With DO_GEMPAK=YES, DO_BUFRSND=YES, and DO_ARCHCOM=YES each set in config.base with the test case, and with no errors resulting after changes to gfs_bfr2gpk.sh and gfs_sndp.sh, there appears to be resolution to #4271.

aerorahul

thanks for addressing the comments. lgtm.

DavidHuber-NOAA · 2025-12-09T19:08:08Z

Launching CI on WCOSS2.

dev/parm/config/gfs/config.cleanup

CatherineThomas-NOAA

Looks great. Thanks @DavidHuber-NOAA!

DavidHuber-NOAA added 3 commits November 20, 2025 17:34

Enable set -o pipefail to catch errors in piped commands

3d0f427

Refactor cleanup task and add checks for gempak files

4ad6542

Merge remote-tracking branch 'emc/develop' into feature/save_gempak_1p00

00372f3

github-advanced-security bot found potential problems Nov 20, 2025

View reviewed changes

scripts/exglobal_cleanup.sh Fixed Show fixed Hide fixed

scripts/exglobal_cleanup.sh Fixed Show fixed Hide fixed

Shellcheck fixes

8e4322b

github-advanced-security bot found potential problems Nov 20, 2025

View reviewed changes

scripts/exglobal_cleanup.sh Fixed Show fixed Hide fixed

scripts/exglobal_cleanup.sh Fixed Show fixed Hide fixed

DavidHuber-NOAA added 6 commits November 20, 2025 20:56

Ignore 2312

c6a23cf

Remove unused variable

484235c

Disable pipefail exits for now

749f917

Shellcheck issues

ceeae5e

Merge branch 'feature/save_gempak_1p00' of github.com:davidhuber-noaa…

cea9f0e

…/global-workflow into feature/save_gempak_1p00

Fix bufrsnd tarball creation

7006ec5

DavidHuber-NOAA commented Nov 21, 2025

View reviewed changes

scripts/exgfs_atmos_postsnd.sh Outdated Show resolved Hide resolved

DavidHuber-NOAA added 3 commits November 21, 2025 19:18

Correct archiving to actually pull the data used

4efde5a

Correct positional arguments

214eb50

Tar all bufr files

ebb0e33

BoCui-NOAA reviewed Nov 21, 2025

View reviewed changes

ush/gfs_sndp.sh Outdated Show resolved Hide resolved

DavidHuber-NOAA added 4 commits November 21, 2025 22:31

Use a find command instead

cdf5a8f

Merge remote-tracking branch 'emc/develop' into feature/save_gempak_1p00

ef21b22

Do not load gh anymore

3303cc4

Enable archiving on WCOSS2 by default

473c7b5

DavidHuber-NOAA mentioned this pull request Nov 25, 2025

gfs_arch_tar_gfs_downstream failure #4271

Open

Make the bufr soundings and surface files EE2-compliant

7cda242

DavidHuber-NOAA commented Nov 25, 2025

View reviewed changes

ush/gfs_bfr2gpk.sh Show resolved Hide resolved

DavidHuber-NOAA added 2 commits November 26, 2025 14:51

Update soundings tarball name to be EE2-compliant

41effff

Correct atmos product dependencies

c8bc24c

JessicaMeixner-NOAA reviewed Dec 1, 2025

View reviewed changes

dev/parm/config/gfs/config.cleanup Outdated Show resolved Hide resolved

DavidHuber-NOAA added 2 commits December 1, 2025 18:30

Replace tabs with spaces

30395ab

Increase memory for gempak jobs (needed for meta task)

a88f475

DavidHuber-NOAA removed the CI-Gaeac6-Failed **Bot use only** CI testing on Gaea C6 for this PR has failed label Dec 5, 2025

JessicaMeixner-NOAA mentioned this pull request Dec 5, 2025

gdas_marinebmat job failure NOAA-EMC/GDASApp#1996

Open

DavidHuber-NOAA added 5 commits December 8, 2025 12:59

Merge remote-tracking branch 'emc/develop' into feature/save_gempak_1p00

76bc130

Merge remote-tracking branch 'emc/develop' into feature/save_gempak_1p00

f677e4f

Re-disable HPSS archiving on WCOSS2 by default

bb7a31a

Disable C48_S2SW on WCOSS2

4a8bb34

This test is covered by C48_S2SW_extended

Turn on HPSS archiving for extended tests

8382dea

DavidHuber-NOAA requested a review from TerrenceMcGuinness-NOAA as a code owner December 9, 2025 18:31

aerorahul reviewed Dec 9, 2025

View reviewed changes

dev/ci/cases/pr/C48_S2SW.yaml Show resolved Hide resolved

parm/archive/gfs_downstream.yaml.j2 Show resolved Hide resolved

ush/gfs_sndp.sh Outdated Show resolved Hide resolved

ChristopherHill-NOAA reviewed Dec 9, 2025

View reviewed changes

DavidHuber-NOAA added 2 commits December 9, 2025 19:00

Revert the use of 'bufr_target'

fb0f744

Remove now-unused variable

246487c

aerorahul previously approved these changes Dec 9, 2025

View reviewed changes

DavidHuber-NOAA added the CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress label Dec 9, 2025

CatherineThomas-NOAA reviewed Dec 9, 2025

View reviewed changes

dev/parm/config/gfs/config.cleanup Outdated Show resolved Hide resolved

Correct GEMPAK_CLEANUP check

865a53d

DavidHuber-NOAA dismissed aerorahul’s stale review via 865a53d December 9, 2025 19:21

CatherineThomas-NOAA approved these changes Dec 9, 2025

View reviewed changes

ChristopherHill-NOAA mentioned this pull request Dec 9, 2025

gfs_gempakmeta occasionally runs out of memory #4087

Open

Refactor cleanup task and add checks for gempak files #4258

Are you sure you want to change the base?

Refactor cleanup task and add checks for gempak files #4258

Uh oh!

Conversation

DavidHuber-NOAA commented Nov 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA commented Nov 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

emcbot commented Dec 3, 2025

Uh oh!

DavidHuber-NOAA commented Dec 9, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChristopherHill-NOAA left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA commented Dec 9, 2025

Uh oh!

Uh oh!

CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

DavidHuber-NOAA commented Nov 20, 2025 •

edited

Loading

ChristopherHill-NOAA left a comment •

edited

Loading