remove copy of state into mpi window by hkershaw-brown · Pull Request #735 · NCAR/DART

hkershaw-brown · 2024-09-10T15:59:07Z

Description:

Puts the %copies array into the mpi window for get_state (forward operator)
Removes the copy of the %copies(1:real_ens_members, :) to the window.
This reduces the per-code memory usage (removes a %copies(1:real_ens_members, 1:my_num_vars)
~1/2 for the state per memory core usage.

The offset is calculated from %num_copies, the get_state is only the real ensemble members

I think this is a good time to remove the cray window module.

Fixes issue

fixes #718

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Tests

There is a developer test included. Note this needs fortran-testanything code to run.
Bitwase lorenz_96 mpi, mpif08, nompi

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

using external fortran-testanything model_mod.f90 is in the work directory. This is a lazy move to avoid fiddling with EXTRA in quickbuild.sh for issue #718

mpi version with all of %copies in the window mpif08 version with all of %copies in the window null_mpi_utilities_mod.f90 version with matchin arguments to the mpi version (does nothing) fixes #718

The cray window mod was originally added because not all compilers/ mpi implemenations (mpi 2.0) supported mpi windows without using cray pointers. The cray_win_mod has not been compiled into dart for several years, and support for mpi windows has improved (mpi 3.0)

hkershaw-brown · 2024-09-30T13:41:30Z

@hkershaw-brown check PMO

mjs2369

Remove these identical comments -

DART/assimilation_code/modules/utilities/no_cray_win_mod.f90

Lines 32 to 33 in 8c54973

    
           !>@todo should this be in the window_mod?  you will have to change in both cray  
        
           !> and non cray versions

DART/assimilation_code/modules/utilities/no_cray_winf08_mod.f90

Lines 32 to 33 in 8c54973

    
           !>@todo should this be in the window_mod?  you will have to change in both cray  
        
           !> and non cray versions

assimilation_code/modules/utilities/no_cray_winf08_mod.f90

mjs2369 · 2024-10-30T19:43:44Z

Do we want to rename the window mods to not mention cray?

no_cray_win_mod.f90 -> win_mod.f90
no_cray_winf08_mod.f90 -> winf08_mod.f90

mjs2369 · 2024-10-30T20:03:51Z

developer_tests/window/work/threed_model_mod.f90

+! These routines are passed through from default_model_mod.
+! To write model specific versions of these routines
+! remove the routine from this use statement and add your code to
+! this the file.


remove this comment?

This is the template model_mod, it it just in there to compile the tests.

I can change the quickbuild.sh to get the template model mod rather than a copying (lazy see commit 9d0f897)

Doesn't matter if we aren't going to put the tests in the repo, but if we do I think we should use the template

used template, added fortran-test anything to developer tests.

mjs2369 · 2024-10-30T21:02:11Z

In buildfunctions.sh and buildconvfunctions.sh, you can remove all mentions of windowsrc. It's not needed anymore.

I just pushed up my commits from when I removed the cray window a while back if you want to take a look at these or cherry pick and commits (on branch remove_craywin). It has windowsrc already taken out of the code. It also has the window files already renamed and updates buildfunctions.sh and buildconvfunctions.sh accordingly

hkershaw-brown · 2024-10-31T15:01:40Z

In buildfunctions.sh and buildconvfunctions.sh, you can remove all mentions of windowsrc. It's not needed anymore.

I just pushed up my commits from when I removed the cray window a while back if you want to take a look at these or cherry pick and commits (on branch remove_craywin). It has windowsrc already taken out of the code. It also has the window files already renamed and updates buildfunctions.sh and buildconvfunctions.sh accordingly

Good point. I shall grab your changes from remove_craywin.
Since we only have null/mpi/mpif08 windows we don't have to have a separate window file (cray vs not cray). I quite like the window being in a separate file rather than adding more stuff to {null|mpi}_utilties_mod.f90

…indowsrc variable

hkershaw-brown · 2024-11-01T14:21:27Z

In buildfunctions.sh and buildconvfunctions.sh, you can remove all mentions of windowsrc. It's not needed anymore.

I just pushed up my commits from when I removed the cray window a while back if you want to take a look at these or cherry pick and commits (on branch remove_craywin). It has windowsrc already taken out of the code. It also has the window files already renamed and updates buildfunctions.sh and buildconvfunctions.sh accordingly

cherry-picked your commits onto this branch

from dennisdjensen https://github.com/dennisdjensen/fortran-testanything license included in this commit. renamed test.f08 to test.f90 because *.f90 is what we have mkmf looking for updated buildfunctions to include the fortran-testanything code for developer tests only removed the local threed_model_mod.f90 (using the template model path like other developer tests)

re #735 (review)

re #735 (comment)

hkershaw-brown · 2024-11-01T16:26:10Z

Remove these identical comments -

DART/assimilation_code/modules/utilities/no_cray_win_mod.f90

Lines 32 to 33 in 8c54973

!>@todo should this be in the window_mod? you will have to change in both cray

!> and non cray versions

DART/assimilation_code/modules/utilities/no_cray_winf08_mod.f90

Lines 32 to 33 in 8c54973

!>@todo should this be in the window_mod? you will have to change in both cray

!> and non cray versions

done

mjs2369

Everything looks good, developer test working well.

adding missing v to tags in changelog attribution to dennisdjensen for fortran-testanything

hkershaw-brown added 3 commits September 10, 2024 10:51

test harness for window code

9d0f897

using external fortran-testanything model_mod.f90 is in the work directory. This is a lazy move to avoid fiddling with EXTRA in quickbuild.sh for issue #718

feat: window without a copy of the data

7d46eff

mpi version with all of %copies in the window mpif08 version with all of %copies in the window null_mpi_utilities_mod.f90 version with matchin arguments to the mpi version (does nothing) fixes #718

explicit about length of array since the length is known

06a27ec

hkershaw-brown requested review from braczka and johnsonbk September 11, 2024 13:27

hkershaw-brown and others added 2 commits September 11, 2024 10:00

remove cray_win_mod

519cda8

The cray window mod was originally added because not all compilers/ mpi implemenations (mpi 2.0) supported mpi windows without using cray pointers. The cray_win_mod has not been compiled into dart for several years, and support for mpi windows has improved (mpi 3.0)

Merge branch 'main' into mpi-win

2496b53

hkershaw-brown added 2 commits September 30, 2024 09:41

Merge branch 'main' into mpi-win

5a45023

Merge branch 'main' into mpi-win

8c54973

mjs2369 reviewed Oct 30, 2024

View reviewed changes

assimilation_code/modules/utilities/no_cray_winf08_mod.f90 Outdated Show resolved Hide resolved

mjs2369 reviewed Oct 30, 2024

View reviewed changes

assimilation_code/modules/utilities/no_cray_winf08_mod.f90 Show resolved Hide resolved

mjs2369 reviewed Oct 30, 2024

View reviewed changes

mjs2369 added 2 commits November 1, 2024 10:04

renaming the window mods as they no longer need to mention cray

f2d9824

updating buildfunctions and buildconvfunctions to no longer use the w…

6c73b04

…indowsrc variable

hkershaw-brown added 3 commits November 1, 2024 12:00

remove old comments

4ee33d4

re #735 (review)

comment fix to be specific about which ensemble_handle

fcfd721

re #735 (comment)

mjs2369 approved these changes Nov 4, 2024

View reviewed changes

hkershaw-brown added the release! bundle with next release label Nov 6, 2024

hkershaw-brown force-pushed the mpi-win branch from 419d96c to b97bc33 Compare November 7, 2024 20:27

bump conf.py and changelog for release

d56aa4a

adding missing v to tags in changelog attribution to dennisdjensen for fortran-testanything

hkershaw-brown force-pushed the mpi-win branch from b97bc33 to d56aa4a Compare November 7, 2024 20:30

hkershaw-brown merged commit 9eb01c7 into main Nov 7, 2024

hkershaw-brown deleted the mpi-win branch November 7, 2024 20:33

	!>@todo should this be in the window_mod? you will have to change in both cray
	!> and non cray versions

Conversation

hkershaw-brown commented Sep 10, 2024

Description:

Fixes issue

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Checklist for release

Testing Datasets

Uh oh!

hkershaw-brown commented Sep 30, 2024

Uh oh!

mjs2369 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mjs2369 commented Oct 30, 2024

Uh oh!

mjs2369 Oct 30, 2024

Choose a reason for hiding this comment

Uh oh!

hkershaw-brown Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

hkershaw-brown Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

mjs2369 Oct 31, 2024

Choose a reason for hiding this comment

Uh oh!

hkershaw-brown Nov 1, 2024

Choose a reason for hiding this comment

Uh oh!

mjs2369 commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hkershaw-brown commented Oct 31, 2024

Uh oh!

hkershaw-brown commented Nov 1, 2024

Uh oh!

hkershaw-brown commented Nov 1, 2024

Uh oh!

mjs2369 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

mjs2369 commented Oct 30, 2024 •

edited

Loading