Run hindcast tests only on change to hindcast module #270

ssolson · 2023-10-12T17:15:30Z

This Pull Request introduces a mechanism to selectively trigger the execution of expensive hindcast tests based on the context of the changes made. Specifically, the hindcast tests will only run under two scenarios:

If there are changes made to the modules within the mhkit/wave/io/hindcast directory.
If the merge request is targeted at the master branch.

To achieve this, a new job named check-changes has been added to the workflow, which precedes the prepare-cache job.

Here's how the check-changes job operates:

Detecting Module Changes:
- Check if any files within the mhkit/wave/io/hindcast directory have been altered.
  - If yes, set wave_io_hindcast_changed to true.
  - If no, set wave_io_hindcast_changed to false.
Determining Hindcast Test Execution:
- Check if the PR is against the master branch or if wave_io_hindcast_changed is true.
- If either condition is met, set should-run-hindcast to true.
  - Otherwise, set should-run-hindcast to false.

Some additional minor modifications have been made to ensure the seamless integration of Coveralls reporting, regardless of whether the hindcast tests are executed or skipped.

ssolson · 2023-10-12T19:19:58Z

@akeeste can I request a quick review on this PR to help us move PRs through review quicker due to less time spent on hindcast calls?

akeeste

Hi Sterling, thanks for doing this. A couple minor comments included below

akeeste · 2023-10-12T21:21:20Z

.github/workflows/main.yml

@@ -211,6 +244,7 @@ jobs:
  coveralls:
    name: Indicate completion to coveralls.io
    needs: [conda-build, pip-build, hindcast-calls]
+    if: always() && (needs.hindcast-calls.result == 'success' || needs.hindcast-calls.result == 'skipped')


Is this indicating that if the hindcast tests are run but fail, coveralls reporting is skipped? Will coveralls still consider the hindcast module to be covered if the hindcast tests don't need to run?

If hindcast fails coveralls is skipped. You can see that as the current behavior (I grabbed the following screenshot from PR #268 which currently fails):

In my screenshot for this PR description I show the used logic. You can see that hindcast is skipped and that coveralls is submitted.

What coveralls does is consider lines of code ran vs lines of code tested and aggregates that into a single number.

When we do not run hindcast the lines in the hindcast module do not count against coverage (in our case we will catch any degradation of coverage on hindcast in commits to master to when the hindcast module is changed).

Coverage is always a tenuous metric at best.

.github/workflows/main.yml

# MHKiT v0.8.0 We're excited to announce the release of MHKiT v0.8.0, which brings a host of new features, enhancements, and bug fixes across various modules, ensuring compatibility with Python 3.10 and 3.11, and introducing full xarray support for more flexible data handling. Significant updates in the Wave and DOLfYN modules improve functionality and extend capabilities. ## Python 3.10 & 3.11 Support MHKiT now supports python 3.10 and 3.11. Support for 3.12 will follow in the next minor update. - #240 ## Wave Module ### Enhancements: **Automatic Threshold Calculation for Peaks-Over-Threshold**: We've introduced a new feature that automatically calculates the "best" threshold for identifying significant wave events. This method, originally developed by Neary, V. S., et al. in their 2020 study, has now been translated from Matlab to Python, enhancing our existing peaks-over-threshold functionality. **Wave Heights Analysis**: A new function, `wave_heights`, has been added to extract the heights of individual waves from a time series. This function uses zero up-crossing analysis to accurately measure wave heights, improving upon our previous methods which only provided the maximum value between up-crossings. **Enhanced Zero Crossing Analysis**: Building on the above, the zero crossing code previously embedded in `global_peaks` has been isolated into a helper function. This modular approach not only refines the codebase but also supports new functionalities such as calculating wave heights, zero crossing periods, and identifying crests. ### Bug Fixes: **Contour Sampling Error in Wave Contours**: A bug identified in `mhkit.wave.contours.samples_contour` has been resolved. The issue occurred when period samples defined using the maximum period resulted in values outside the interpolation range of the contour data. This was corrected by ensuring that all sampling points are within the interpolation range and adjusting the contour data selection process accordingly. - #268 - #252 - #278 ## Xarray Support MHKiT functions now fully support the use of xarray for passing and returning data. - #279 - #282 - #285 - #302 - #310 ## DOLfYN Thanks to the many user contributions and users of MHKiT the DOLFYN module include a significant number of enhancements and bug fixes. ### Enhancements: **Altimeter Support**: Enhanced the Nortek Signature Reader to add capability for reading ADCP dual profile configurations. **Data Handling Improvements**: Introduced logic to skip messy header data that can accumulate during measurements collected via Nortek software on PCs and Macs. **Instrument Noise Subtraction**: Added a function to subtract instrument noise from turbulence intensity estimation using RMS calculations, providing results that differ by approximately 1% from the existing standard deviation-based "I" property. **Improved File Handling**: Updates for RDI files to handle changing "number of cells" and variable "cell sizes," which are now bin-averaged into the largest cell size. ### Bug Fixes: **Power Spectra Calculation**: Fixed a bug where a given noise value was not being subtracted from the power spectra, and noise was inadvertently added as an input to dissipation rate calculations. **Improved Header Handling**: Allowed RDI reader to skip junk headers effectively. **Nortek Reader C Types Update**: Adjusted C types in the Nortek reader to handle below-zero water temperatures and to allow pitch and roll values to go negative. - #280 - #289 - #290 - #292 - #293 - #294 - #299 ## River & Tidal: D3D Added limits to `variable_interpolation` and added 3 array input capability to `create_points` - #271 ## Developer Experience ### Black formatting Black formatting is now enforced on all MHKiT files. This ensures consistent formatting across the MHKiT package. - #281 ### Linting & Type Hints MHKiT is in the process of enforcing pylint and adding type hints to all functions. Currently this has been achieved and is enforced in the Loads and Power modules. - #288 - #296 ### CI/CD This release introduces significant reduction in testing time for development. This is achieved by reducing the number of tests for pulls against the develop branch and only running hindcast test when changes are made to it. A bug in the hindcast CI was fixed which only ran on changes to the hindcast tests instead of the hindcast module. Additionally the wave and wind hindcast needed to be separated in 2 jobs due to the excessive time taken to run a wind cache. This created a number of follow on PRs around solidifying the logic of these job. A special case for Python 3.8, pip, and Mac OS was added to use homebrew to install NetCDF and HDF5 to get tests to pass. - #241 - #270 - #306 - #311 - #317 - #318 - #319 - #320 - #324 ### Clean Up MHKiT fixed an implementation error where functions used assert instead of built in errors for type and value checking. Additionally these PRs removed unused files, fixed typos, and created an argument which allows users to run CDIP API calls silently. - #276 - #272 - #273

ssolson added 4 commits October 12, 2023 11:13

run hindcast only on change

5e23262

always run hindcast on changes to master

8c28e1a

indicate completion to coveralls on hindcast skip

10a0f7b

move hindcast run logic into check changes job

24991f1

ssolson self-assigned this Oct 12, 2023

ssolson requested a review from akeeste October 12, 2023 18:36

ssolson added the enhancement New feature or request label Oct 12, 2023

ssolson marked this pull request as ready for review October 12, 2023 19:18

akeeste reviewed Oct 12, 2023

View reviewed changes

ssolson added 5 commits October 12, 2023 15:34

use should-run-hindcast variable

b81f61f

require conda & pip success to run

93b4675

log job

d82a760

add always

b7a1b91

remove log job

4719d75

ssolson merged commit a82bc3f into MHKiT-Software:develop Oct 16, 2023

ssolson deleted the tests branch April 16, 2024 14:55

ssolson mentioned this pull request Apr 24, 2024

MHKiT v0.8.0 #307

Closed

ssolson mentioned this pull request May 6, 2024

MHKiT v0.8.0 #321

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Run hindcast tests only on change to hindcast module #270

Run hindcast tests only on change to hindcast module #270

Uh oh!

ssolson commented Oct 12, 2023 •

edited

Loading

Uh oh!

ssolson commented Oct 12, 2023

Uh oh!

akeeste left a comment •

edited

Loading

Uh oh!

akeeste Oct 12, 2023

Uh oh!

ssolson Oct 12, 2023 •

edited

Loading

Uh oh!

ssolson Oct 12, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Run hindcast tests only on change to hindcast module #270

Run hindcast tests only on change to hindcast module #270

Uh oh!

Conversation

ssolson commented Oct 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ssolson commented Oct 12, 2023

Uh oh!

akeeste left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

akeeste Oct 12, 2023

Choose a reason for hiding this comment

Uh oh!

ssolson Oct 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ssolson Oct 12, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ssolson commented Oct 12, 2023 •

edited

Loading

akeeste left a comment •

edited

Loading

ssolson Oct 12, 2023 •

edited

Loading

ssolson Oct 12, 2023 •

edited

Loading