Release 2.1.0

Merge from develop to main for release 2.1.0

Author: tennlee

.gitignore

@@ -110,3 +110,7 @@ dmypy.json
 # Cython debug symbols
 cython_debug/
 
+# pixi environments
+.pixi
+pixi.lock
+*.egg-info

.zenodo.json

@@ -74,6 +74,11 @@
             "orcid": "https://orcid.org/0009-0006-7361-163X",
             "affiliation": "Independent Contributor, Australia",
             "name": "Bluett, Liam"
+        },
+        {
+            "orcid": "https://orcid.org/0000-0003-3271-6874",
+            "affiliation": "Australian National University, Australia",
+            "name": "Squire, Dougal T."
         }
     ],    
     "license": "Apache-2.0",

README.md

@@ -20,10 +20,10 @@ Below is a **curated selection** of the metrics, tools and statistical tests inc
 |-----------------------	|-----------------	|--------------	|
 | **[Continuous](https://scores.readthedocs.io/en/stable/included.html#continuous)**        	|Scores for evaluating single-valued continuous forecasts.                  	|MAE, MSE, RMSE, Additive Bias, Multiplicative Bias, Percent Bias, Pearson's Correlation Coefficient, Kling-Gupta Efficiency, Flip-Flop Index, Quantile Loss, Quantile Interval Score, Interval Score, Murphy Score, and threshold weighted scores for expectiles, quantiles and Huber Loss.             	|
 | **[Probability](https://scores.readthedocs.io/en/stable/included.html#probability)**        |Scores for evaluating forecasts that are expressed as predictive distributions, ensembles, and probabilities of binary events.                   |Brier Score, Continuous Ranked Probability Score (CRPS) for Cumulative Density Functions (CDF) and ensembles (including threshold weighted versions), Receiver Operating Characteristic (ROC), Isotonic Regression (reliability diagrams).               |
-| **[Categorical](https://scores.readthedocs.io/en/stable/included.html#categorical)**       	|Scores for evaluating forecasts of categories.                	|18 binary contingency table (confusion matrix) metrics and the FIxed Risk Multicategorical (FIRM) Score.               	|
+| **[Categorical](https://scores.readthedocs.io/en/stable/included.html#categorical)**       	|Scores for evaluating forecasts of categories.                	|18 binary contingency table (confusion matrix) metrics, the FIxed Risk Multicategorical (FIRM) Score, and the SEEPS score.               	|
 | **[Spatial](https://scores.readthedocs.io/en/stable/included.html#spatial)** 	|Scores that take into account spatial structure.                 	|Fractions Skill Score.              	|
 | **[Statistical Tests](https://scores.readthedocs.io/en/stable/included.html#statistical-tests)** 	|Tools to conduct statistical tests and generate confidence intervals.                 	|Diebold Mariano.              	|
-| **[Processing Tools](https://scores.readthedocs.io/en/stable/included.html#processing-tools-for-preparing-data)**        	|Tools to pre-process data.                 	|Data matching, Discretisation, Cumulative Density Function Manipulation.              	|
+| **[Processing Tools](https://scores.readthedocs.io/en/stable/included.html#processing-tools-for-preparing-data)**        	|Tools to pre-process data.                 	|Data matching, Discretisation, Block Bootstrapping, and Cumulative Density Function Manipulation.              	|
 | **[Emerging](https://scores.readthedocs.io/en/stable/included.html#emerging)**        	|Emerging scores that are still undergoing mathematical peer review. They may change in line with the peer review process.                 	|Risk Matrix Score.             	|
 
 `scores` not only includes common scores (e.g., MAE, RMSE), it also includes novel scores not commonly found elsewhere (e.g., FIRM, Flip-Flop Index), complex scores (e.g., threshold weighted CRPS), and statistical tests (e.g., the Diebold Mariano test). Additionally, it provides pre-processing tools for preparing data for scores in a variety of formats including cumulative distribution functions (CDF). `scores` provides its own implementations where relevant to avoid extensive dependencies.

docs/api.md

@@ -24,6 +24,7 @@
 .. autofunction:: scores.continuous.multiplicative_bias
 .. autofunction:: scores.continuous.pbias
 .. autofunction:: scores.continuous.kge
+.. autofunction:: scores.continuous.nse
 .. autofunction:: scores.continuous.isotonic_fit
 .. autofunction:: scores.continuous.consistent_expectile_score
 .. autofunction:: scores.continuous.consistent_quantile_score
@@ -68,6 +69,7 @@
     :members:
 .. autoclass:: scores.categorical.EventOperator
     :members:
+.. autofunction:: scores.categorical.seeps
 ```
 
 ## scores.spatial
@@ -87,6 +89,7 @@
 
 ## scores.processing
 ```{eval-rst}
+.. autofunction:: scores.processing.block_bootstrap
 .. autofunction:: scores.processing.isotonic_fit
 .. autofunction:: scores.processing.broadcast_and_match_nan
 .. autofunction:: scores.processing.comparative_discretise

docs/conf.py

@@ -9,7 +9,7 @@
 
 project = "scores"
 copyright = "Licensed under Apache 2.0 - https://www.apache.org/licenses/LICENSE-2.0"
-release = "2.0.0"
+release = "2.1.0"
 
 version = __version__
 

docs/data.md

@@ -20,7 +20,7 @@ To use `scores` with [GRIB](https://codes.wmo.int/grib2) data, install [cfgrib](
 
 ### Working with NetCDF Data
 
-To use `scores` with [NetCDF](https://doi.org/10.5065/D6H70CW6) or [HDF5](https://github.com/HDFGroup/hdf5) data, install [h5netcdf](https://github.com/h5netcdf/h5netcdf). The h5netcdf library is included in the `scores` ["all"](installation.md#all-dependencies-excludes-some-maintainer-only-packages) and ["tutorial"](installation.md#tutorial-dependencies) installation options. Opening NetCDF data is demonstrated in [this tutorial](project:./tutorials/First_Data_Fetching.md). 
+To use `scores` with [NetCDF](https://doi.org/10.5065/D6H70CW6) or [HDF5](https://github.com/HDFGroup/hdf5) data, install [h5netcdf](https://github.com/h5netcdf/h5netcdf). The h5netcdf library is included in the `scores` ["all"](installation.md#all-dependencies-excludes-some-maintainer-only-packages) and ["tutorial"](installation.md#tutorial-dependencies) installation options. Opening NetCDF data is demonstrated in [this tutorial](project:./tutorials/First_Data_Fetching.md).
 
 ## Weather and Climate Data
 
@@ -35,13 +35,17 @@ Global numerical weather prediction (NWP) models are used to generate medium ran
 Archived datasets are available for:
 
 - Bureau of Meteorology's Australian Parallel Suite version 3 (APS3) Australian Community Climate and Earth-System Simulator (ACCESS), see [https://doi.org/10.25914/608a993391647](https://doi.org/10.25914/608a993391647).
+- [WeatherBench 2](https://weatherbench2.readthedocs.io/en/latest/data-guide.html) contains forecasts with data-driven (AI) and physical NWP models on a common grid. The [twCRPS for ensemble forecasts tutorial](project:./tutorials/Threshold_Weighted_CRPS_for_Ensembles.md) shows how to use this data with `scores`.
 - National Oceanic and Atmospheric Administration (NOAA) Global Forecast System (GFS), see [https://www.ncei.noaa.gov/products/weather-climate-models/global-forecast](https://www.ncei.noaa.gov/products/weather-climate-models/global-forecast).
+- An archive of AI weather models going back to October 2020 is hosted at [https://noaa-oar-mlwp-data.s3.amazonaws.com/index.html](https://noaa-oar-mlwp-data.s3.amazonaws.com/index.html) as part of the Open Data Dissemination program. It contains, FourCastNet v2-small, Pangu-Weather, and GraphCast Operational data. It is updated twice a day. You can read more about it in their [paper](https://doi.org/10.1175/BAMS-D-24-0057.1).
 
 #### Point-Based Data
 
 Point-based observations (e.g. from weather stations or buoys) are shared routinely between countries for the purposes of weather modelling.
 
-The NOAA Integrated Surface Database (ISD) provides hourly point-based (*in-situ*) weather station data globally. It is a good starting point for understanding how to work with point-based data. For more information about the NOAA ISD see [https://www.ncei.noaa.gov/products/land-based-station/integrated-surface-database](https://www.ncei.noaa.gov/products/land-based-station/integrated-surface-database).
+- The NOAA Integrated Surface Database (ISD) provides hourly point-based (*in-situ*) weather station data globally. It is a good starting point for understanding how to work with point-based data. For more information about the NOAA ISD see [https://www.ncei.noaa.gov/products/land-based-station/integrated-surface-database](https://www.ncei.noaa.gov/products/land-based-station/integrated-surface-database).
+- [WeatherReal](https://github.com/microsoft/WeatherReal-Benchmark) contains quality controlled weather station data that uses the ISD. You can read more about WeatherReal in the [pre-print](https://doi.org/10.48550/arXiv.2409.09371).
+- The [Iowa Environmental Mesonet](https://mesonet.agron.iastate.edu/) contains a rich variety of datasets. One particularly useful dataset is the [1-minute Automated Surface Observing Network (ASOS) data](https://mesonet.agron.iastate.edu/request/asos/1min.phtml).
 
 #### Gridded Model Reanalysis Data
 

docs/included.md

@@ -101,6 +101,10 @@
     [Tutorial](project:./tutorials/Murphy_Diagrams.md)
   - 
     [Ehm et al. (2016) - Corollary 2 (p.521)](https://doi.org/10.1111/rssb.12154); [Taggart (2022) - Corollary 5.6](https://doi.org/10.1214/21-ejs1957)
+* - Nash-Sutcliffe Model Efficiency Coefficient (NSE)
+  - [API](api.md#scores.continuous.nse)
+  - [Tutorial](project:./tutorials/NSE.md)
+  - [Nash and Sutcliffe (1970)](https://doi.org/10.1016/0022-1694%2870%2990255-6)
 * - Pearson's Correlation Coefficient
   - [API](api.md#scores.continuous.correlation.pearsonr)
   - [Tutorial](project:./tutorials/Pearsons_Correlation.md)
@@ -559,6 +563,10 @@
   - [API](api.md#scores.categorical.probability_of_false_detection)
   - [Tutorial](project:./tutorials/ROC.md)
   - [Probability of false detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#POFD)
+* - Stable Equitable Error in Probability Space (SEEPS)
+  - [API](api.md#scores.categorical.seeps)
+  - [Tutorial](project:./tutorials/SEEPS.md)
+  - [Rodwell et al. (2010)](https://doi.org/10.1002/qj.656)
 * - Threshold Event Operator
   - [API](api.md#scores.categorical.ThresholdEventOperator)
   - [Tutorial](project:./tutorials/Binary_Contingency_Scores.md)
@@ -649,6 +657,9 @@
 * - Binary Discretise Proportion
   - [API](api.md#scores.processing.binary_discretise_proportion)    
   - Flip-Flop Index
+* - Block Bootstrap
+  - [API](api.md#scores.processing.block_bootstrap)
+  - Confidence intervals. See [tutorial](project:./tutorials/Block_Bootstrapping.md)
 * - Broadcast and Match Not-a-Number (NaN)
   - [API](api.md#scores.processing.broadcast_and_match_nan)   
   - Murphy Score (Mean Elementary Score)
@@ -764,15 +775,18 @@
     - Risk Matrix Score
   - [API](api.md#scores.emerging.risk_matrix_score)
   - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
-  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+  - [Taggart and Wilke (2025)](https://doi.org/10.48550/arXiv.2502.08891)
+
 * -  
     - Risk Matrix Score - Matrix Weights to Array
   - [API](api.md#scores.emerging.matrix_weights_to_array)
   - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
-  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+  - [Taggart and Wilke (2025)](https://doi.org/10.48550/arXiv.2502.08891)
+
 * -  
     - Risk Matrix Score - Warning Scaling to Weight Array
   - [API](api.md#scores.emerging.weights_from_warning_scaling)
   - [Tutorial](project:./tutorials/Risk_Matrix_Score.md)
-  - Taggart, R. J., & Wilke, D. J. (2024). Warnings based on risk matrices: a coherent framework with consistent evaluation. In preparation.
+  - [Taggart and Wilke (2025)](https://doi.org/10.48550/arXiv.2502.08891)
+
 ```

docs/installation.md

@@ -126,3 +126,49 @@ A sample command to register a new kernel is:
 
 [https://jupyter-tutorial.readthedocs.io/en/24.1.0/kernels/install.html](https://jupyter-tutorial.readthedocs.io/en/24.1.0/kernels/install.html) provides additional technical details regarding the registration of kernels.
 
+## Using `pixi` for Environment Management (Optional)
+
+An optional, alternative, approach that `scores` supports for installing environments is [`pixi`](https://pixi.sh).
+`pixi` is a powerful environment management tool.
+
+It uses a combination of PyPI and Conda channels. `pixi` is configured in `pyproject.toml` in the
+root directory of the `scores` GitHub repository. It is configured with some default tasks that a
+user can run in ephemeral environments specific for those tasks (see examples below).
+
+`pixi` handles creation, swapping, stacking and cleanup of environments automatically, depending on
+the task being run.
+
+```{note}
+`scores` currently does not save `pixi.lock` files in its GitHub repository. While `pixi` is
+supported in `scores`, it is *not* part of the recommended development toolchain.
+
+`pixi.lock` is intentionally filtered out in `.gitignore`, in order to prevent accidental commits of
+the lock file. This may change in the future if there is sufficient adoption.
+
+`pixi` is mainly there for users who *already* are familiar with it, and those who prefer not to
+manually deal with python environments.
+```
+
+### Installation
+
+`pixi` supports multiple platforms. Its installation process is straightforward and can be found
+here: <https://pixi.sh/latest/#installation>.
+
+### Examples
+
+- **As a developer** I want to run some tests.
+   - Command: `pixi run pytest-src`
+   - Description: this will test the source code in the `dev` environment.
+- **As a researcher** I want to launch JupyterLab.
+  - Command: `pixi run jupyterlab`
+  - Description: this will launch a local JupyterLab server in the `tutorial` environment.
+- **As a maintainer** I want to render the docs as html.
+  - Command:  `pixi run make-docs`
+  - Description: this will render the docs locally to "htmldocs" (similar to what the GitHub
+    pipeline currently does).
+- **As any user** I want to run a specified command in a particular environment.
+  - Command: `pixi run -e <env> <cmd>`, where `<env> = dev | tutorial | maintainer | all` - see
+    section on [installation options](#installation-options) above.
+  - Description: this will run the command in the specified environment, and return you back to the
+    original shell once it has been executed.
+

docs/release_notes.md

@@ -1,5 +1,47 @@
 # Release Notes (What's New)
 
+## Version 2.1.0 (April 30, 2025)
+
+For a list of all changes in this release, see the [full changelog](https://github.com/nci/scores/compare/2.0.0...2.1.0). Below are the changes we think users may wish to be aware of.
+
+### Features
+
+- Added a new fuction:
+	- Block bootstrap: `scores.processing.block_bootstrap`. See [PR #418](https://github.com/nci/scores/pull/418).
+- Added two new metrics:
+	- Stable equitable error in probability space (SEEPS): `scores.categorical.seeps`. See [PR #809](https://github.com/nci/scores/pull/809) and [PR #833](https://github.com/nci/scores/pull/833).
+	- Nash-Sutcliffe model efficiency coefficient (NSE): `scores.continuous.nse`. See [PR #815](https://github.com/nci/scores/pull/815).
+
+### Documentation
+
+- Added "Block Bootstrapping" tutorial. See [PR #418](https://github.com/nci/scores/pull/418). 
+- Added "Stable Equitable Error in Probability Space (SEEPS)" tutorial. See [PR #809](https://github.com/nci/scores/pull/809).
+- Added "Nash-Sutcliffe Efficiency (NSE)" tutorial. See [PR #815](https://github.com/nci/scores/pull/815).
+- Updated the "Continuous Ranked Probability Score (CRPS) for Ensembles" tutorial:
+	- Labelled dimensions in fcst/obs data.
+	- Updated description of the plot to say the area squared corresponds to the CRPS.
+	- Added an example with multiple coordinates along a dimension.  
+	See [PR #805](https://github.com/nci/scores/pull/805).
+- Updated "Data Sources":
+	- Added links to two additional datasets for gridded global numerical weather prediction.
+	- Added links to several additional datasets for point-based data.  
+	See [PR #823](https://github.com/nci/scores/pull/823) and [PR #831](https://github.com/nci/scores/pull/831). 
+- Updated references in several sections of the documentation, following the publication of a [preprint](https://doi.org/10.48550/arXiv.2502.08891) for the risk matrix score. See [PR #827](https://github.com/nci/scores/pull/827).
+
+### Internal Changes
+
+- Tested and added compatibility for recent Xarray versions (2025 and onwards) and adjusted dependency specification so new year "major version" rollovers will be permitted by default in future. See [commit #f109f2f](https://github.com/nci/scores/commit/f109f2f434ac684b3d54f447c330466d33703279) and [commit #8428d64](https://github.com/nci/scores/commit/8428d64dcf2a5f5480c61b266284260d4b5078d2).
+- In `scores.emerging.weights_from_warning_scaling`, changed the name of the argument `assessment_weights` to  `evaluation_weights`. See [PR #806](https://github.com/nci/scores/issues/806).  
+***Note:** This is technically a breaking change, but does not trigger a major release as it is contained within the "emerging" section of the API. This area of the API is designated for metrics which are still undergoing peer review and as such are expected to undergo change. Once peer review is concluded, the implementation will be finalised and moved.*
+- Add support for developers of `scores` who choose to use the `pixi` tool for environment management. See [PR #835](https://github.com/nci/scores/pull/835), [PR #839](https://github.com/nci/scores/pull/839) and [PR #840](https://github.com/nci/scores/pull/840).
+
+### Contributors to this Release
+
+Dougal T. Squire* ([@dougiesquire](https://github.com/dougiesquire)), Mohammad Mahadi Hasan* ([@engrmahadi](https://github.com/engrmahadi)), Mohammadreza Khanarmuei ([@reza-armuei](https://github.com/reza-armuei)), Nikeeth Ramanathan ([@nikeethr](https://github.com/nikeethr)) Tennessee Leeuwenburg ([@tennlee](https://github.com/tennlee)), Nicholas Loveday ([@nicholasloveday](https://github.com/nicholasloveday)), 
+Robert J. Taggart ([@rob-taggart](https://github.com/rob-taggart)), Durga Shrestha ([@durgals](https://github.com/durgals)) and Stephanie Chong ([@Steph-Chong](https://github.com/Steph-Chong)).
+
+\* indicates that this release contains their first contribution to `scores`.
+
 ## Version 2.0.0 (December 7, 2024)
 
 For a list of all changes in this release, see the [full changelog](https://github.com/nci/scores/compare/1.3.0...2.0.0). Below are the changes we think users may wish to be aware of.