Distribution fixes #788

hkershaw-brown · 2024-12-26T20:57:16Z

Description:

Fixes for distributions that have been hanging out on https://github.com/NCAR/DART/tree/beta_distribution_fix

beta distribution only supporting standard beta (previously had some generalization that was only partially done causing tests to fail bug: beta distribution not correct (failing tests) #717)
gamma distribution only supporting standard gamma (previously had some generalization only partially done)
normal distribution, setting all distribution_params_type values (not causing known problems, but maybe future problems)

Forcing the gamma and beta qceff options (bounds) in qceff table to be:

GAMMA_DISTRIBUTION (lower bound at 0)
BETA_DISTRIBUTION (bound between 0 and 1)

test in developer_tests/qceff/test_force_bounds

I can split this into 3 pull requests if that is easier to review.

Fixes issue

fixes #717 only supporting standard beta
fixes #786 only supporting standard gamma - note see issue, still have upper and lower bound
fixes #787 see notes - initialize distribution_param_type to UNSET?

There was an E_MSG about failing to converge, switched this to E_ALLMSG since the failure could be on any task.

I think the documentation for the qceff available distributions probably needs updating to clarify that the GAMMA & BETA bounds are fixed (QCEFF table values are ignored) Edit: went ahead and changed the documentation.

DART/guide/qceff_probit.rst

Lines 126 to 135 in bddda57

    
           Available distributions 
        
           ------------------------ 
        
             * NORMAL_DISTRIBUTION (default) 
        
             * BOUNDED_NORMAL_RH_DISTRIBUTION 
        
             * GAMMA_DISTRIBUTION 
        
             * BETA_DISTRIBUTION 
        
             * LOG_NORMAL_DISTRIBUTION 
        
             * UNIFORM_DISTRIBUTION 
        
             * KDE_DISTRIBUTION

Maybe this is sufficient, I dunno:

 * GAMMA_DISTRIBUTION (lower bound at 0)
 * BETA_DISTRIBUTION (bound between 0 and 1)

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Tests

Please describe any tests you ran to verify your changes.
developer_tests/*_dist
developer_tests/qceff/test_force_bounds

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

Functions inv_beta_cdf_params and beta_cdf_params now include an error check to make sure that the lower and upper bounds in the distribution_params_type have been set to 0 and 1 as other values are not supported. HK note it might be better to remove this and have pure functions with no side effects. Subroutine set_beta_params_from_ens has changed the distribution_params_type to an intent out argument and defines all six parameters correctly. It also set the parameter type to BETA_DISTRIBUTION. fixes #717

…mma distributions. jla commit message: A comment now makes it clear that this module only supports the standard gamma distribution that is bounded below by 0 and unbounded above. Subroutines gamma_cdf, inverse_gamma_cdf, set_gamma_params_from_ens, and inv_gamma_first_guess no longer have bounded_below, bounded_above, lower_bound, and upper_bound as arguments. inv_gamma_cdf now sets the bounded_below, bounded_above, lower_bound and upper_bound parameters to the correct values for the gamma. Functions inv_gamma_cdf_params and gamma_cdf_params now include an error check to make sure that the lower and upper bounds in the distribution_params_type have been set to 0 and missing_r8 as other values are not supported. Subroutine set_gamma_params_from_ens has changed the distribution_params_type to an intent out argument and defines all six parameters correctly. It also sets the parameter type to GAMMA_DISTRIBUTION.

jla commit message: Functions inv_std_normal_cdf and set_normal_params_from_ens now set the appropriate values for the bounded_below, bounded_above, lower_bound and upper_bound components of the distribution_params_type. The distribution_params_type was changed to intent out in set_normal_params_from_ens. The magic number definition of the maximum delta in the inv_cdf root searching routine was changed to be a parameter but the value of 1e-8 was unchanged. A comment notes that changing this parameter to 1e-9 allows all of Ian Groom’s KDE distribution tests to pass. HK note, assuming this is if you use inv_cdf from normal_distribution_mod rather than rootfinding mode with KDE. fixes #787

Only occurances on task 0 were printing #787 (comment)

assimilation_code/modules/assimilation/normal_distribution_mod.f90

qceff table bounds are ignored for these

issues #717 #786 Added developer test to check forced values are set correctly Required changes to run_all.sh for qceff tests to deal with input.nml

assimilation_code/modules/assimilation/probit_transform_mod.f90

hkershaw-brown · 2025-01-21T17:36:04Z

@mjs2369 can you take a look at this when you get a chance.

mjs2369

Looking good Helen. My only questions/comments are posted

mjs2369 · 2025-01-21T22:15:51Z

developer_tests/qceff/work/runall.sh

@@ -41,6 +41,9 @@ else
 fi
 }

+cp input.nml input.nml.bak
+cp input.nml.no_alg input.nml


What is the motivation here for using this input.nml file without an &algorithm_info_nml?

the other tests switch in an &algorithm_info_nml with various qceff_tables by appending &algorithm_info_nml to the end of the input.nml
The namelist read in fortran will take the first occurrence of the namelist in the file and ignore any subsequent ones. So the input.nml.no_alg is to start with a input.nml that doesn't have a &algorithm_info_nml in it.
This reason it is in this pull request is test_force_bounds needs an input.nml, so now have input.nml and input.nml.no_alg

guide/qceff_probit.rst

developer_tests/qceff/test_force_bounds.f90

Co-authored-by: Marlena Smith <[email protected]>

https://github.com/NCAR/DART/pull/788/files/be7f568f4faad4cafefb12a21160609d3f84cc54..b9e67b4e214162de8c5b47db418f5112cbe30833#r1904307694

mjs2369

Good stuff @hkershaw-brown

hkershaw-brown added 4 commits December 26, 2024 13:03

fix: any task with a failure to converge prints a message

86c7266

Only occurances on task 0 were printing #787 (comment)

hkershaw-brown added the QCEFF quantile conserving filters label Dec 31, 2024

have UNSET as the default initalization for distribution_params_type

55b7042

hkershaw-brown commented Jan 6, 2025

View reviewed changes

assimilation_code/modules/assimilation/normal_distribution_mod.f90 Outdated Show resolved Hide resolved

hkershaw-brown commented Jan 6, 2025

View reviewed changes

assimilation_code/modules/assimilation/normal_distribution_mod.f90 Outdated Show resolved Hide resolved

doc: note on standard bounds only for GAMMA and BETA

538d238

qceff table bounds are ignored for these

hkershaw-brown force-pushed the distribution-fixes branch from 397a971 to 538d238 Compare January 7, 2025 18:54

feat: force standard gamma and beta in distribution in qceff table read

e0fb360

issues #717 #786 Added developer test to check forced values are set correctly Required changes to run_all.sh for qceff tests to deal with input.nml

hkershaw-brown force-pushed the distribution-fixes branch from 1a9dc77 to e0fb360 Compare January 7, 2025 20:05

hkershaw-brown commented Jan 7, 2025

View reviewed changes

assimilation_code/modules/assimilation/probit_transform_mod.f90 Outdated Show resolved Hide resolved

hkershaw-brown commented Jan 7, 2025

View reviewed changes

assimilation_code/modules/assimilation/probit_transform_mod.f90 Outdated Show resolved Hide resolved

hkershaw-brown marked this pull request as ready for review January 7, 2025 20:17

hkershaw-brown requested review from jlaucar and mjs2369 January 7, 2025 20:17

hkershaw-brown and others added 3 commits January 9, 2025 13:29

Merge branch 'main' into distribution-fixes

79545d6

set the distribution type for normal distribution

7930bff

remove unused bound arguments for to_probit_gamma

bcbeefe

hkershaw-brown requested review from jlaucar and mjs2369 and removed request for mjs2369 and jlaucar January 9, 2025 19:03

Merge branch 'main' into distribution-fixes

e0db378

mjs2369 reviewed Jan 21, 2025

View reviewed changes

hkershaw-brown and others added 3 commits January 21, 2025 19:51

doc: readability change

5d441ba

Co-authored-by: Marlena Smith <[email protected]>

fix: state number of tests for TAP

c68fe38

Co-authored-by: Marlena Smith <[email protected]>

comment on test_normal failing if r8=r4

8583817

https://github.com/NCAR/DART/pull/788/files/be7f568f4faad4cafefb12a21160609d3f84cc54..b9e67b4e214162de8c5b47db418f5112cbe30833#r1904307694

hkershaw-brown added the release! bundle with next release label Jan 22, 2025

hkershaw-brown and others added 2 commits January 22, 2025 08:47

Changelog and sync conf.py for release

d0f0351

Merge branch 'main' into distribution-fixes

4d6ebc8

mjs2369 approved these changes Jan 22, 2025

View reviewed changes

hkershaw-brown merged commit eafcb5b into main Jan 22, 2025
4 checks passed

hkershaw-brown deleted the distribution-fixes branch January 22, 2025 19:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Distribution fixes #788

Distribution fixes #788

Uh oh!

hkershaw-brown commented Dec 26, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hkershaw-brown commented Jan 21, 2025

Uh oh!

mjs2369 left a comment

Uh oh!

mjs2369 Jan 21, 2025

Uh oh!

hkershaw-brown Jan 22, 2025

Uh oh!

Uh oh!

Uh oh!

mjs2369 left a comment

Uh oh!

Uh oh!

Uh oh!

	Available distributions
	------------------------

	* NORMAL_DISTRIBUTION (default)
	* BOUNDED_NORMAL_RH_DISTRIBUTION
	* GAMMA_DISTRIBUTION
	* BETA_DISTRIBUTION
	* LOG_NORMAL_DISTRIBUTION
	* UNIFORM_DISTRIBUTION
	* KDE_DISTRIBUTION

Distribution fixes #788

Distribution fixes #788

Uh oh!

Conversation

hkershaw-brown commented Dec 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description:

Fixes issue

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Checklist for release

Testing Datasets

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hkershaw-brown commented Jan 21, 2025

Uh oh!

mjs2369 left a comment

Choose a reason for hiding this comment

Uh oh!

mjs2369 Jan 21, 2025

Choose a reason for hiding this comment

Uh oh!

hkershaw-brown Jan 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mjs2369 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hkershaw-brown commented Dec 26, 2024 •

edited

Loading