[1Pt] PR: Add back multi-processesing to inundate_gms

[GregoryPetrochenkov-NOAA]

Quick inclusion to allow threading or multiprocessing in inundate_gms.py processing.

Changes

• tools/inundate_gms.py: Add multi_process argument, when set to True use processes instead of threads.
• tools/inundate_mosaic_wrapper.py: Add gms_multi_process argument to dictate use of threads or processes in inundate_gms.py.
• tools/inundation.py: Remove prange from numba operation.
• tools/run_test_case.py: Run inundate_gms with processes for run_test_cases and add threading.
• tools/synthesize_test_cases.py: Add a threads argument for branches.
• tools/inundation_nation.py: Run inundate_gms with processes and include threads argument.

Testing

Ran tests with both gms_multi_process set to True and False. The following performance was achieved.

Set to False:

Set to True:

Last test was on my EC2, inundate_nation.py for 10 HUCS. The old version was ran as follows:

python /foss_fim/tools/inundate_nation.py -r /outputs/Inundation_Nation_Test -m 100_0 -f /data/inputs/rating_curve/nwm_recur_flows/nwm3_17C_recurr_10_0_cms.csv -s -j 15 -o /outputs/Inundation_Nation_Test

Runtime: 18 minutes 22 seconds

This branch with the following command:

python /foss_fim/tools/inundate_nation.py -r /outputs/Inundation_Nation_Test -m 100_0 -f /data/inputs/rating_curve/nwm_recur_flows/nwm3_17C_recurr_10_0_cms.csv -s -j 15 -t 8 -o /outputs/Inundation_Nation_Test

Runtime: 4 minutes 37 seconds

I also tested synthesize_test_cases.py, with the following command:

python /foss_fim/tools/synthesize_test_cases.py -c DEV -v mini-test -b ble -jh 7 -jb 2 -m /outputs/mini-test/gms_synth_metrics.csv -vg -o

Runtime: 27 minutes and 48 seconds

python /foss_fim/tools/synthesize_test_cases.py -c DEV -v mini-test -b ble -jh 7 -jb 2 -tb 8 -m /outputs/mini-test/gms_synth_metrics.csv -vg -o

Runtime: 26 minutes 51 seconds

---- UPDATE ----

I ran inundation nation on UAT minus 7 non-CONUS HUCs with the following command:

python /foss_fim/tools/inundate_nation.py -r /outputs/lofi_uat_test -m 100_0 -f /data/inputs/rating_curve/nwm_recur_flows/nwm3_17C_recurr_10_0_cms.csv -s -j 63 -t 64 -o /outputs/lofi_uat_test

Runtime: 32 minutes and 6 seconds

Deployment Plan (For developer use)

How does the changes affect the product?

• Code only?
• If applicable, has a deployment plan be created with the deployment person/team?
• Require new or adjusted data inputs? Does it have start, end and duration code (in UTC)?
• If new or updated data sets, has the FIM code been updated and tested with the new/adjusted data (subset is fine, but must be a subset of the new data)?
• Require new pre-clip set?
• Has new or updated python packages?

Issuer Checklist (For developer use)

You may update this checklist before and/or after creating the PR. If you're unsure about any of them, please ask, we're here to help! These items are what we are going to look for before merging your code.

• Informative and human-readable title, using the format: [_pt] PR: <description>
• Links are provided if this PR resolves an issue, or depends on another other PR
• If submitting a PR to the dev branch (the default branch), you have a descriptive Feature Branch name using the format: dev-<description-of-change> (e.g. dev-revise-levee-masking)
• Changes are limited to a single goal (no scope creep)
• The feature branch you're submitting as a PR is up to date (merged) with the latest dev branch
• pre-commit hooks were run locally
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• CHANGELOG updated with template version number, e.g. 4.x.x.x
• Add yourself as an assignee in the PR as well as the FIM Technical Lead

Merge Checklist (For Technical Lead use only)

• Update CHANGELOG with latest version number and merge date
• Update the Citation.cff file to reflect the latest version number in the CHANGELOG
• If applicable, update README with major alterations

[ZahraGhahremani]

Ran successfully for 10 HUCs. Please update the changelog.

[RobHanna-NOAA]

Ran successfully for 10 HUCs. Please update the changelog.

Can I recommend you try a sample bigger than just 10? And play with how job number affect it? :)

[GregoryPetrochenkov-NOAA]

I ran inundation nation on UAT minus 7 non-CONUS HUCs with the following command:

python /foss_fim/tools/inundate_nation.py -r /outputs/lofi_uat_test -m 100_0 -f /data/inputs/rating_curve/nwm_recur_flows/nwm3_17C_recurr_10_0_cms.csv -s -j 63 -t 64 -o /outputs/lofi_uat_test

Runtime: 32 minutes and 6 seconds