Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix hanging Hindcast tests #311

Merged
merged 29 commits into from
Apr 27, 2024
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
2ccc286
Merge pull request #267 from MHKiT-Software/develop
ssolson Aug 31, 2023
ae6b715
assert to exception
ssolson Nov 21, 2023
7614867
Revert "Ensure interpolation values are within range when sampling co…
ssolson Nov 21, 2023
d09e606
Merge branch 'master' of https://github.com/MHKiT-Software/MHKiT-Pyth…
ssolson Nov 29, 2023
35a917c
Merge branch 'develop' of https://github.com/ssolson/MHKiT-Python int…
ssolson Nov 29, 2023
afd3171
Merge branch 'develop' of https://github.com/MHKiT-Software/MHKiT-Pyt…
ssolson Dec 6, 2023
667a416
Merge branch 'develop' of https://github.com/ssolson/MHKiT-Python int…
ssolson Dec 6, 2023
76cf500
Merge branch 'develop' of https://github.com/MHKiT-Software/MHKiT-Pyt…
ssolson Feb 5, 2024
e267aa0
Merge branch 'develop' of https://github.com/MHKiT-Software/MHKiT-Pyt…
ssolson Mar 12, 2024
05916f5
Merge branch 'develop' of https://github.com/ssolson/MHKiT-Python int…
ssolson Mar 12, 2024
d906411
Merge branch 'develop' of https://github.com/MHKiT-Software/MHKiT-Pyt…
ssolson Mar 19, 2024
915d99d
Merge branch 'develop' of https://github.com/MHKiT-Software/MHKiT-Pyt…
ssolson Apr 23, 2024
ecbc2fc
Merge branch 'develop' of https://github.com/MHKiT-Software/MHKiT-Pyt…
ssolson Apr 25, 2024
f1f8a24
checkout v4
ssolson Apr 25, 2024
9bf60f8
path filters v3
ssolson Apr 25, 2024
7c334e6
checkout v4
ssolson Apr 25, 2024
93d0b4d
hindcast wave and wind tests as 2 steps
ssolson Apr 25, 2024
f1299cf
correct name to wind
ssolson Apr 25, 2024
60aa4a8
small change to force hindcast tests to run
ssolson Apr 25, 2024
0924d85
fix hindcast change path
ssolson Apr 25, 2024
b46b720
create 3 seperate cache jobs to adhere to 6 hour job limit
ssolson Apr 26, 2024
bd14353
upload and consolidate wave and wind data
ssolson Apr 26, 2024
da821c9
add install dependencies
ssolson Apr 26, 2024
5850af0
non hindcast data does not need check-change job
ssolson Apr 26, 2024
cf4289c
use hindcast folder inside cache for consolidation
ssolson Apr 26, 2024
b0a5295
run on changes to the test as well
ssolson Apr 26, 2024
936233b
artifact v4
ssolson Apr 26, 2024
411a385
miniconda v3
ssolson Apr 26, 2024
bb1bcea
actions/setup-python@v5
ssolson Apr 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 116 additions & 22 deletions .github/workflows/main.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,15 +30,15 @@ jobs:
should-run-hindcast: ${{ steps.hindcast-logic.outputs.should-run-hindcast }}
steps:
- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Check for changes in wave/io/hindcast
id: changes
uses: dorny/paths-filter@v2
uses: dorny/paths-filter@v3
with:
filters: |
wave_io_hindcast:
- 'mhkit/tests/wave/io/hindcast/**'
- 'mhkit/wave/io/hindcast/**'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should probably check for changes in both places: hindcast tests and the hindcast module itself

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks I have been too lazy to investigate if I could just add the test line to the filters. Looks like I can.

I'll push the commit up after the current actions pass or fail.


- id: hindcast-logic
run: |
Expand All @@ -48,14 +48,13 @@ jobs:
echo "should-run-hindcast=false" >> "$GITHUB_OUTPUT"
fi

prepare-cache:
needs: [check-changes]
prepare-nonhindcast-cache:
runs-on: ubuntu-latest
env:
PYTHON_VER: 3.9
steps:
- name: Checkout code
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Setup Conda
uses: s-weigand/setup-conda@v1
Expand All @@ -68,7 +67,7 @@ jobs:
run: |
conda create --name TEST python=${{ env.PYTHON_VER }} numpy cython pip hdf5 libnetcdf cftime netcdf4 --strict-channel-priority
source activate TEST
export PATH="${CONDA_PREFIX}/bin:${CONDA_PREFIX}/Library/bin:$PATH" # so setup.py finds nc-config
export PATH="${CONDA_PREFIX}/bin:${CONDA_PREFIX}/Library/bin:$PATH"
pip install -e . --no-deps --force-reinstall

- name: Install dependencies
Expand All @@ -78,30 +77,106 @@ jobs:
python -m pip install --upgrade pip wheel
pip install coverage pytest coveralls .

- name: Prepare data
- name: Prepare non-hindcast API data
run: |
source activate TEST
pytest mhkit/tests/river/test_io_usgs.py
pytest mhkit/tests/tidal/test_io.py
pytest mhkit/tests/wave/io/test_cdip.py

- name: Prepare Hindcast data
if: (needs.check-changes.outputs.should-run-hindcast == 'true')
- name: Upload data as artifact
uses: actions/upload-artifact@v3
with:
name: data
path: ~/.cache/mhkit

prepare-wave-hindcast-cache:
needs: [check-changes]
runs-on: ubuntu-latest
env:
PYTHON_VER: 3.9
if: (needs.check-changes.outputs.should-run-hindcast == 'true')
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup Conda
uses: s-weigand/setup-conda@v1
with:
activate-conda: false
conda-channels: conda-forge

- name: Python setup
shell: bash -l {0}
run: |
conda create --name TEST python=${{ env.PYTHON_VER }} numpy cython pip hdf5 libnetcdf cftime netcdf4 --strict-channel-priority
source activate TEST
export PATH="${CONDA_PREFIX}/bin:${CONDA_PREFIX}/Library/bin:$PATH"
pip install -e . --no-deps --force-reinstall

- name: Install dependencies
shell: bash -l {0}
run: |
source activate TEST
python -m pip install --upgrade pip wheel
pip install coverage pytest coveralls .

- name: Prepare Wave Hindcast data
run: |
# pytest tests/test_specific_file.py::TestClass::test_function
source activate TEST
pytest mhkit/tests/wave/io/hindcast/test_hindcast.py

- name: Upload Wave Hindcast data as artifact
uses: actions/upload-artifact@v3
with:
name: wave-hindcast-data
path: ~/.cache/mhkit

prepare-wind-hindcast-cache:
needs: [check-changes]
runs-on: ubuntu-latest
env:
PYTHON_VER: 3.9
if: (needs.check-changes.outputs.should-run-hindcast == 'true')
steps:
- name: Checkout code
uses: actions/checkout@v4

- name: Setup Conda
uses: s-weigand/setup-conda@v1
with:
activate-conda: false
conda-channels: conda-forge

- name: Python setup
shell: bash -l {0}
run: |
conda create --name TEST python=${{ env.PYTHON_VER }} numpy cython pip hdf5 libnetcdf cftime netcdf4 --strict-channel-priority
source activate TEST
export PATH="${CONDA_PREFIX}/bin:${CONDA_PREFIX}/Library/bin:$PATH"
pip install -e . --no-deps --force-reinstall

- name: Install dependencies
shell: bash -l {0}
run: |
source activate TEST
python -m pip install --upgrade pip wheel
pip install coverage pytest coveralls .

- name: Prepare Wind Hindcast data
run: |
source activate TEST
pytest mhkit/tests/wave/io/hindcast/test_wind_toolkit.py

- name: Upload data as artifact
- name: Upload Wind Hindcast data as artifact
uses: actions/upload-artifact@v3
with:
name: data
name: wind-hindcast-data
path: ~/.cache/mhkit

conda-build:
name: conda-${{ matrix.os }}/${{ matrix.python-version }}
needs: [set-os, prepare-cache]
needs: [set-os, prepare-nonhindcast-cache]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
Expand All @@ -112,7 +187,7 @@ jobs:
PYTHON_VER: ${{ matrix.python-version }}

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Setup Conda
uses: s-weigand/setup-conda@v1
Expand Down Expand Up @@ -159,7 +234,7 @@ jobs:

pip-build:
name: pip-${{ matrix.os }}/${{ matrix.python-version }}
needs: [set-os, prepare-cache]
needs: [set-os, prepare-nonhindcast-cache]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
Expand All @@ -176,7 +251,7 @@ jobs:
python-version: ${{ matrix.python-version }}

- name: Set up Git repository
uses: actions/checkout@v3
uses: actions/checkout@v4

- name: Download data from artifact
uses: actions/download-artifact@v3
Expand All @@ -202,7 +277,13 @@ jobs:

hindcast-calls:
name: hindcast-${{ matrix.os }}/${{ matrix.python-version }}
needs: [check-changes, prepare-cache, set-os]
needs:
[
check-changes,
prepare-wave-hindcast-cache,
prepare-wind-hindcast-cache,
set-os,
]
if: (needs.check-changes.outputs.should-run-hindcast == 'true')

runs-on: ${{ matrix.os }}
Expand All @@ -214,7 +295,7 @@ jobs:
python-version: ['3.8', '3.9', '3.10', '3.11']

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4

- name: Setup Conda
uses: s-weigand/setup-conda@v1
Expand All @@ -230,11 +311,24 @@ jobs:
export PATH="${CONDA_PREFIX}/bin:${CONDA_PREFIX}/Library/bin:$PATH" # so setup.py finds nc-config
pip install -e . --no-deps --force-reinstall

- name: Download data from artifact
- name: Download Wave Hindcast data from artifact
uses: actions/download-artifact@v3
with:
name: data
path: ~/.cache/mhkit
name: wave-hindcast-data
path: ~/.cache/mhkit/wave-hindcast

- name: Download Wind Hindcast data from artifact
uses: actions/download-artifact@v3
with:
name: wind-hindcast-data
path: ~/.cache/mhkit/wind-hindcast

- name: Consolidate hindcast data
run: |
mkdir -p ~/.cache/mhkit/hindcast
mv ~/.cache/mhkit/wave-hindcast/hindcast/* ~/.cache/mhkit/hindcast/
mv ~/.cache/mhkit/wind-hindcast/hindcast/* ~/.cache/mhkit/hindcast/
shell: bash

- name: Install MHKiT and run pytest
shell: bash -l {0}
Expand Down
2 changes: 1 addition & 1 deletion mhkit/wave/io/hindcast/hindcast.py
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ def request_wpto_point_data(

Note: To access the WPTO hindcast data, you will need to configure
h5pyd for data access on HSDS. Please see the WPTO_hindcast_example
notebook for more information.
notebook for setup instructions.

Parameters
----------
Expand Down
Loading