Quickbuild tests #593

hkershaw-brown · 2023-12-11T14:12:52Z

Description:

building on Ann's previous pull request #575
improved build_everything, submits a job on Derecho for each compiler, each job runs every quickbuild.sh

time:
ccc ~12minutes
gfortran ~2:30 minutes
nvhpc ~4:40 minutes
ifort ~4:40 minutes

Note several of the converters require external libraries or code to be added before compiling, ignoring these for this pull request (rttov, hdfeos, wrf code, ncep prepbuf code):

intel RESULT: 1 /glade/derecho/scratch/hkershaw/build_everything/intel/DART/developer_tests/forward_operators/work/ FAILED
intel RESULT: 10 /glade/derecho/scratch/hkershaw/build_everything/intel/DART/observations/obs_converters/GOES/work/ FAILED
intel RESULT: 11 /glade/derecho/scratch/hkershaw/build_everything/intel/DART/observations/obs_converters/gps/work/ FAILED
intel RESULT: 12 /glade/derecho/scratch/hkershaw/build_everything/intel/DART/observations/obs_converters/NSIDC/work/ FAILED
intel RESULT: 28 /glade/derecho/scratch/hkershaw/build_everything/intel/DART/observations/obs_converters/var/work/ FAILED
intel RESULT: 29 /glade/derecho/scratch/hkershaw/build_everything/intel/DART/observations/obs_converters/AIRS/work/ FAILED
intel RESULT: 31 /glade/derecho/scratch/hkershaw/build_everything/intel/DART/observations/obs_converters/quikscat/work/ FAILED
intel RESULT: 42 /glade/derecho/scratch/hkershaw/build_everything/intel/DART/observations/obs_converters/GMI/work/ FAILED

Other failures: #592 #352, #594

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Documentation update

Documentation changes needed?

My change requires a change to the documentation.
- I have updated the documentation accordingly.

Tests

Derecho: build everything (that has a quickbuild.sh) in DART for cce, intel, gcc, nvhpc

Checklist for merging

Updated changelog entry
Documentation updated
Update conf.py

Checklist for release

Merge into main
Create release from the main branch with appropriate tag
Delete feature-branch

Testing Datasets

Dataset needed for testing available upon request
Dataset download instructions included
No dataset needed

Add module load for all compiler, mpif08 argument for gnu compiler, testing submission script for Derecho

…tter the convert_airs_L2 converter was printing 1 more obs converted than was actually done. also, the settings for the convert_airs_L2 namelist thinned out all the obs. change the defaults to include all possible obs.

nvhpc also needs mpif08

…build.sh

not rm -rf existing runs, instead moving compiled runs to run.$(date)

fixes #592

defaults to main

developer_tests/build_everything/run_all_quickbuilds.sh

mjs2369

In order to add ifx to the list of compilers to build with, we need to update fixsystem to include ifx

developer_tests/build_everything/README

Co-authored-by: Marlena Smith <[email protected]>

mjs2369 · 2023-12-11T20:37:06Z

The builds that use specific libraries such as rttov will be addressed in a future pull request

hkershaw-brown · 2023-12-11T20:37:11Z

In order to add ifx to the list of compilers to build with, we need to update fixsystem to include ifx

will do, thanks Marlee!

…he files that don't actually call nc_check

ifx mkmf.template

hkershaw-brown · 2023-12-11T22:01:25Z

I've added ifx. To run, you'll need to run with the quickbuild_tests branch because main does not have the mkmf.template.ifx.linux

./submit_jobs quickbuild_tests

mjs2369 · 2023-12-11T23:39:18Z

To run, you'll need to run with the quickbuild_tests branch because main does not have the mkmf.template.ifx.linux
./submit_jobs quickbuild_tests

^^^ This should be added to the README

And a follow up question @hkershaw-brown - if the submit_jobs.sh script submits a job for each compiler, are we expecting the ifx build to still run and just error out on all branches other than quickbuild_tests?

mjs2369

Should we add some code for teardown? Currently, consecutive runs will fail with
Directory exists: /glade/derecho/scratch/masmith/build_everything/nvhpc

hkershaw-brown · 2023-12-12T00:55:46Z

To run, you'll need to run with the quickbuild_tests branch because main does not have the mkmf.template.ifx.linux
./submit_jobs quickbuild_tests
^^^ This should be added to the README

Nope don't add to the README. When this pull request is merged into main, mkmf.template.ifx.linux will exist on main. This 'To run' note was just for you for this pull request.

And a follow up question @hkershaw-brown - if the submit_jobs.sh script submits a job for each compiler, are we expecting the ifx build to still run and just error out on all branches other than quickbuild_tests?

Yes because mkmf.template.ifx.linux only exists on the quickbuild_tests branch.
Once quickbuild_tests is merged into main mkmf.template.ifx.linux will exist on main.

hkershaw-brown · 2023-12-12T01:01:52Z

Should we add some code for teardown? Currently, consecutive runs will fail with Directory exists: /glade/derecho/scratch/masmith/build_everything/nvhpc

This is the teardown:

DART/developer_tests/build_everything/run_all_quickbuilds.sh

Line 143 in a20971a

mv $test_dir $test_dir.$(date +"%FT%H%M")

If you don't have a compiler.dateTime directory then the job did not finish.
So I think at that point it is worth manually investigating why the job has failed, rather than rm -rf directories.

mjs2369 · 2023-12-12T16:49:13Z

To run, you'll need to run with the quickbuild_tests branch because main does not have the mkmf.template.ifx.linux
./submit_jobs quickbuild_tests
^^^ This should be added to the README
Nope don't add to the README. When this pull request is merged into main, mkmf.template.ifx.linux will exist on main. This 'To run' note was just for you for this pull request.

And a follow up question @hkershaw-brown - if the submit_jobs.sh script submits a job for each compiler, are we expecting the ifx build to still run and just error out on all branches other than quickbuild_tests?

Yes because mkmf.template.ifx.linux only exists on the quickbuild_tests branch. Once quickbuild_tests is merged into main mkmf.template.ifx.linux will exist on main.

That makes much more sense and is obvious in hindsight. For some reason, I thought you meant that mkmf.template.ifx.linux was not going to be added to main with this PR. Ignore this.

mjs2369 · 2023-12-12T20:57:51Z

Sometimes Derecho is unable to make a connection to the DART remote repository, causing some of the jobs to fail with this message:

Cloning into 'DART'...
fatal: unable to access 'https://github.com/NCAR/DART.git/': Failed to connect to github.com port 443 after 1 ms: Couldn't connect to server
./run_all_quickbuilds.sh: line 51: cd: DART: No such file or directory
fatal: not a git repository (or any parent up to mount point /glade/derecho)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /glade/derecho)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
unknown branch

I have submitted a support request with CISL Help to address this issue.

mjs2369 · 2023-12-12T21:04:17Z

Additional failures with cce:

Just making a note of these on this PR, as other failures are noted in the body, but we can create issues for these as well

masmith@derecho1:~/DART/developer_tests/build_everything> grep -a FAILED test-2/build-everything-cce.o2646766 
cce RESULT: 0 /glade/derecho/scratch/masmith/build_everything/cce/DART/models/clm/work/ FAILED
cce RESULT: 3 /glade/derecho/scratch/masmith/build_everything/cce/DART/models/noah/work/ FAILED
cce RESULT: 6 /glade/derecho/scratch/masmith/build_everything/cce/DART/models/wrf_hydro/work/ FAILED
cce RESULT: 12 /glade/derecho/scratch/masmith/build_everything/cce/DART/models/wrf/work/ FAILED

CLM:

ftn -O2  -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cce/15.0.1/cuko/include  -c	/glade/u/home/masmith/DART/models/clm/dart_to_clm.f90


ftn-1569 ftn: WARNING UPDATE_SNOW, File = ../../../../../../u/home/masmith/DART/models/clm/dart_to_clm.f90, Line = 612, Column = 19 
  A DO loop variable or expression of type default real or double precision real is a deleted feature of the Fortran standard.


ftn-1569 ftn: WARNING UPDATE_SNOW, File = ../../../../../../u/home/masmith/DART/models/clm/dart_to_clm.f90, Line = 628, Column = 19 
  A DO loop variable or expression of type default real or double precision real is a deleted feature of the Fortran standard.


ftn-319 ftn: ERROR UPDATE_SNOW, File = ../../../../../../u/home/masmith/DART/models/clm/dart_to_clm.f90, Line = 752, Column = 76 
  A subscript must be a scalar integer expression.

Cray Fortran : Version 15.0.1 (20230120205242_66f7391d6a03cf932f321b9f6b1d8612ef5f362c)

Line in question -

DART/models/clm/dart_to_clm.f90

Line 752 in 74b4221

    
           snowdp_po(icolumn) = snowdp_pr(icolumn) + sum(gain_dzsno(nlevsno+1+snlsno(icolumn):nlevsno,icolumn))

snlsno(ncolumn), which is the subscript for gain_dzsno, is a real(r8) - changed to integer (line 418) and it compiles

DART/models/clm/dart_to_clm.f90

Line 418 in 74b4221

real(r8) :: clm_SNLSNO(ncolumn)

NOAH/WRF_HYDRO: 

ftn -O2  -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cce/15.0.1/cuko/include  -c	/glade/u/home/masmith/DART/models/wrf_hydro/noah_hydro_mod.f90
   Error message      ::  _expr_type: Invalid table type
   Error detected     ::  File '/home/jenkins/crayftn/pdgcs/v_expr_utl.c', line 7360
   Initiated from     ::  Line 1280 (v_main.c)
   Optimizer built    ::  2023-01-20 (production)

   File               ::  /glade/u/home/masmith/DART/models/wrf_hydro/noah_hydro_mod.f90
   Function           ::  getchannelgridcoords
   at or near line    ::  660

   Compiler hash      ::  66f7391d6a03cf932f321b9f6b1d8612ef5f362c
   Target             ::  x86-milan

ftn-7991 ftn: INTERNAL GETCHANNELGRIDCOORDS, File = ../../../../../../u/home/masmith/DART/models/wrf_hydro/noah_hydro_mod.f90, Line = 660 
  INTERNAL COMPILER ERROR:  "_expr_type: Invalid table type" (/home/jenkins/crayftn/pdgcs/v_expr_utl.c, line 7360, version 66f7391d6a03cf932f321b9f6b1d8612ef5f362c)

Line in question:

DART/models/wrf_hydro/noah_hydro_mod.f90

Line 660 in 74b4221

n_link = sum(CH_NETRT*0+1, mask = CH_NETRT >= 0)

WRF:

Building  WRF_DART_utilities/add_pert_where_high_refl  build  18  of  29
............................................................................................... Makefile is ready.
ftn -O2  -I/glade/u/apps/derecho/23.06/spack/opt/spack/netcdf/4.9.2/cce/15.0.1/cuko/include  -c	/glade/u/home/masmith/DART/models/wrf/WRF_DART_utilities/add_pert_where_high_refl.f90


ftn-292 ftn: ERROR ADD_PERT_WHERE_HIGH_REFL, File = ../../../../../../u/home/masmith/DART/models/wrf/WRF_DART_utilities/add_pert_where_high_refl.f90, Line = 37, Column = 8 
  "F2KCLI" is specified as the module name on a USE statement, but the compiler cannot find it.

Cray Fortran : Version 15.0.1 (20230120205242_66f7391d6a03cf932f321b9f6b1d8612ef5f362c)

Solution is to remove the following line:
use f2kcli

DART/models/wrf/WRF_DART_utilities/add_pert_where_high_refl.f90

Line 37 in 74b4221

use f2kcli

Compiles after this change

hkershaw-brown · 2023-12-12T21:06:06Z

@mjs2369
#599
#598

mjs2369

Ready to merge. The quickbuild failures are being accurately reported

Resolution to Issue #336 - updated nc_check

fix print of number of obs converted, and make input.nml defaults better

mom6 model_mod .eqv. for logicals comparison

ann-norcio and others added 18 commits November 8, 2023 14:53

Adding quickbuild_tests to developer_tests

2c5622e

Resolution to Issue #336 - updated nc_check

4371d60

Merge branch 'main' into nc_check_subroutine

c8b8d49

Removed nc_check from utilities_mod.f90

8cce38e

Modify all_quickbuilds.sh

e3eb511

Add module load for all compiler, mpif08 argument for gnu compiler, testing submission script for Derecho

Merge branch 'main' into quickbuild_tests

389e434

make script executable

89fc244

remove multiple compilers from single script

58217a7

nvhpc also needs mpif08

chore: file shuffling, Ann's all_quickdbuild.sh replace run_all_quick…

4a985d3

…build.sh

submittion script for Derecho

ef9209a

renamed FC to compiler since FC is an enviroment variable

86849ca

not rm -rf existing runs, instead moving compiled runs to run.$(date)

actually running quickbuild.sh

1e23a24

submit all jobs for all compilers

f6148ac

obs converters, do not use mpif08 option

5f47089

skip template model since this is not supposed to compile

3cc10f5

cce takes 12 minutes

abef575

readme: how to look for failures in results

0921b58

hkershaw-brown assigned mjs2369 and ann-norcio Dec 11, 2023

hkershaw-brown added 3 commits December 11, 2023 07:15

update usage and simpler message print

4d92949

eqv for logicals comparison

73457ac

fixes #592

add option to build a different branch

42b33e6

defaults to main

hkershaw-brown unassigned mjs2369 and ann-norcio Dec 11, 2023

hkershaw-brown requested review from mjs2369 and ann-norcio December 11, 2023 18:03

hkershaw-brown commented Dec 11, 2023

View reviewed changes

developer_tests/build_everything/run_all_quickbuilds.sh Outdated Show resolved Hide resolved

mjs2369 reviewed Dec 11, 2023

View reviewed changes

developer_tests/build_everything/README Show resolved Hide resolved

add branch_name to README usage

2851c07

Co-authored-by: Marlena Smith <[email protected]>

mjs2369 and others added 2 commits December 11, 2023 14:13

Removing the statement to use nc_check from netcdf_utilities_mod in t…

a1e063c

…he files that don't actually call nc_check

added ifx as an option

a20971a

ifx mkmf.template

hkershaw-brown mentioned this pull request Dec 11, 2023

Resolution to Issue #336 - updated nc_check #587

Merged

15 tasks

mjs2369 reviewed Dec 12, 2023

View reviewed changes

speed test, using /tmp vs filesystem

c95a630

mjs2369 approved these changes Dec 12, 2023

View reviewed changes

hkershaw-brown added the release! bundle with next release label Dec 12, 2023

hkershaw-brown and others added 4 commits December 13, 2023 10:38

Merge pull request #587 from c-merchant/nc_check_subroutine

707fe9a

Resolution to Issue #336 - updated nc_check

Merge pull request #597 from NCAR/airs_fix

6774b60

fix print of number of obs converted, and make input.nml defaults better

Merge pull request #600 from NCAR/mom6-eqv-test

99c6e58

mom6 model_mod .eqv. for logicals comparison

CHANGELOG and bump version for release

54a374d

hkershaw-brown merged commit 99ebec5 into main Dec 13, 2023
4 checks passed

hkershaw-brown deleted the quickbuild_tests branch December 13, 2023 17:28

This was referenced Jun 17, 2024

on-demand GitHub action which runs every quickbuild.sh in DART #690

Closed

on-demand GitHub action which runs every quickbuild.sh in DART #691

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Quickbuild tests #593

Quickbuild tests #593

hkershaw-brown commented Dec 11, 2023 •

edited

Loading

mjs2369 left a comment

mjs2369 commented Dec 11, 2023

hkershaw-brown commented Dec 11, 2023

hkershaw-brown commented Dec 11, 2023

mjs2369 commented Dec 11, 2023 •

edited

Loading

mjs2369 left a comment

hkershaw-brown commented Dec 12, 2023 •

edited

Loading

hkershaw-brown commented Dec 12, 2023

mjs2369 commented Dec 12, 2023

mjs2369 commented Dec 12, 2023

mjs2369 commented Dec 12, 2023 •

edited

Loading

hkershaw-brown commented Dec 12, 2023

mjs2369 left a comment

Quickbuild tests #593

Quickbuild tests #593

Conversation

hkershaw-brown commented Dec 11, 2023 • edited Loading

Description:

Types of changes

Documentation changes needed?

Tests

Checklist for merging

Checklist for release

Testing Datasets

mjs2369 left a comment

Choose a reason for hiding this comment

mjs2369 commented Dec 11, 2023

hkershaw-brown commented Dec 11, 2023

hkershaw-brown commented Dec 11, 2023

mjs2369 commented Dec 11, 2023 • edited Loading

mjs2369 left a comment

Choose a reason for hiding this comment

hkershaw-brown commented Dec 12, 2023 • edited Loading

hkershaw-brown commented Dec 12, 2023

mjs2369 commented Dec 12, 2023

mjs2369 commented Dec 12, 2023

mjs2369 commented Dec 12, 2023 • edited Loading

hkershaw-brown commented Dec 12, 2023

mjs2369 left a comment

Choose a reason for hiding this comment

hkershaw-brown commented Dec 11, 2023 •

edited

Loading

mjs2369 commented Dec 11, 2023 •

edited

Loading

hkershaw-brown commented Dec 12, 2023 •

edited

Loading

mjs2369 commented Dec 12, 2023 •

edited

Loading