Alternative way to populate g-w fix/gsi

[RussTreadon-NOAA]

Description
An alternative approach to populating g-w fix/gsi is proposed with the goal of improving consistency between stand-alone GSI development and subsequent cycled tests. This issue is a follow on to discussion found in g-w issue #1550.

Tasks

• Discuss needs with GSI and g-w developers
• Consider alternative approach suggested in this issue
• Adopt, modify, reject, or propose alternative ways to manage GSI fix.
• Make related workflow/component updates.

[RussTreadon-NOAA]

Work for this issue will be done in RussTreadon-NOAA:feature/gsifix.

f4b9985 commits changes to two files:

• sorc/checkout.sh - turn on recursive clone of NOAA-EMC/GSI. This populates gsi_enkf.fd/fix with ASCII GSI fix files from NOAA-EMC/GSI-fix using the GSI-fix hash associated with the cloned NOAA-EMC/GSI hash. When sorc/build_all.sh is executed the gsi_enkf build populates gsi_enkf.fd/fix with binary GSI fix files from a EIB staged location. The path to the EIB location is specified by variable GSI_BINARY_SOURCE_DIR in platform specific modulefiles in gsi_enkf.fd/modulefiles.
• sorc/link_workflow.sh - create g-w fix/gsi as a link pointing at sorc/gsi_enkf.fd/fix

[RussTreadon-NOAA]

One drawback of the above is that gsi_enkf.fd/ush/build.sh currently copies binary GSI fix files from GSI_BINARY_SOURCE_DIR to gsi_enkf.fd/fix. To speed the build and reduce disk usage the copy of binary files could be replaced with links.

If the above approach is adopted, ASCII GSI fix files currently in GSI_BINARY_SOURCE_DIR should be removed. Then GSI_BINARY_SOURCE_DIR only contains binary GSI fix files as the variable name suggests.

[aerorahul]

@RussTreadon-NOAA
If these ASCII GSI-fix files are updated often, then they are not really "fix" files, are they?

[RussTreadon-NOAA]

You are right. ASCII GSI-fix files are not permanently fixed. The same is true of the binary fix files. For example, both Rcov* and Big_Endian/global_berror* files have changed since their initial creation. ASCII files related to data usage (e.g., global_convinfo.txt) change the most frequently. Some ASCII fix files (e.g., global_scaninfo.txt) change much less frequently.

[aerorahul]

Yes, most binaries are updated during their lifetime. The question here is the frequency at which they get updated. The global_berror* files have not been updated in over 5 years (new resolutions have been added).

The more frequent update ascii files e.g. global_convinfo.txt fall under the parlance of parm/ files rather than "fix" files.

I am sure there is a happy medium where we can get to where the frequent development as well as longer maintenance can be achieved.

Is there a way to separate out the largely stagnated, from the frequently evolving?

[RussTreadon-NOAA]

Moving part or all of NOAA-EMC/GSI-fix into NOAA-EMC/GSI as parm/ is an option. Probably need a meeting with, at a minimum, developers representing all operational realizations of the GSI and figure out a common path forward.

[RussTreadon-NOAA]

Thanks @aerorahul for converting this into a g-w discussion. Hopefully others join the conversation so this moves forward.

[RussTreadon-NOAA]

Since this is now a discussion and not an issue. I'll delete RussTreadon-NOAA:feature/gsifix since we don't know when or how this conversation will end.

I don't want to keep resynching my fork with the head of g-w develop for an extended period. My fork already has a conflict with the current head (e55f090) of develop.

[RussTreadon-NOAA]

RussTreadon-NOAA:feature/gsifix at f4b9985 has been deleted.

[aerorahul]

@RussTreadon-NOAA
Is this discussion still active or should it be archived?

[RussTreadon-NOAA]

I'm not keen on the way things are currently done but since no comments have been made in this discussion since 5/10/2023 I will close it.