Merge pull request #498 from nci/minor_release_8_5

Minor release 0.8.5

.binder/requirements.txt

@@ -333,7 +333,7 @@ tinycss2==1.2.1
     # via nbconvert
 tomli==2.0.1
     # via jupyterlab
-tornado==6.4
+tornado==6.4.1
     # via
     #   ipykernel
     #   jupyter-client

README.md

@@ -17,9 +17,10 @@ Here is a **curated selection** of the metrics, tools and statistical tests incl
 
 |                       	| **Description** 	| **Selection of Included Functions** 	|
 |-----------------------	|-----------------	|--------------	|
-| **[Continuous](https://scores.readthedocs.io/en/latest/included.html#continuous)**        	|Scores for evaluating single-valued continuous forecasts.                  	|Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Additive Bias, Multiplicative Bias, Pearson's Correlation Coefficient, Flip-Flop Index, Quantile loss, Murphy score.              	|
+| **[Continuous](https://scores.readthedocs.io/en/latest/included.html#continuous)**        	|Scores for evaluating single-valued continuous forecasts.                  	|Mean Absolute Error (MAE), Mean Squared Error (MSE), Root Mean Squared Error (RMSE), Additive Bias, Multiplicative Bias, Pearson's Correlation Coefficient, Flip-Flop Index, Quantile Loss, Murphy Score.              	|
 | **[Probability](https://scores.readthedocs.io/en/latest/included.html#probability)**       	|Scores for evaluating forecasts that are expressed as predictive distributions, ensembles, and probabilities of binary events.                 	|Brier Score, Continuous Ranked Probability Score (CRPS) for Cumulative Density Function (CDF), Threshold weighted CRPS for CDF, CRPS for ensembles, Receiver Operating Characteristic (ROC), Isotonic Regression (reliability diagrams).              	|
-| **[Categorical](https://scores.readthedocs.io/en/latest/included.html#categorical)**       	|Scores for evaluating forecasts based on categories.                	|Probability of Detection (POD), False Alarm Ratio (FAR), Probability of False Detection (POFD), Success Ratio, Accuracy, Peirce's Skill Score, Critical Success Index (CSI), Gilbert Skill Score, Heidke Skill Score, Odds Ratio, Odds Ratio Skill Score, F1 score, FIxed Risk Multicategorical (FIRM) Score.               	|
+| **[Categorical](https://scores.readthedocs.io/en/latest/included.html#categorical)**       	|Scores for evaluating forecasts based on categories.                	|Probability of Detection (POD), False Alarm Ratio (FAR), Probability of False Detection (POFD), Success Ratio, Accuracy, Peirce's Skill Score, Critical Success Index (CSI), Gilbert Skill Score, Heidke Skill Score, Odds Ratio, Odds Ratio Skill Score, F1 score, Symmetric Extremal Dependence Index, FIxed Risk Multicategorical (FIRM) Score.               	|
+| **[Spatial](https://scores.readthedocs.io/en/latest/included.html#spatial)** 	|Scores that take into account spatial structure.                 	|Fractions Skill Score.              	|
 | **[Statistical Tests](https://scores.readthedocs.io/en/latest/included.html#statistical-tests)** 	|Tools to conduct statistical tests and generate confidence intervals.                 	|Diebold Mariano.              	|
 | **[Processing Tools](https://scores.readthedocs.io/en/latest/included.html#processing-tools-for-preparing-data)**        	|Tools to pre-process data.                 	|Data matching, Discretisation, Cumulative Density Function Manipulation.              	|
 

docs/SECURITY.md

@@ -1,4 +1,4 @@
-## Handling Security Concerns
+# Handling Security Concerns
 
 The scores package is not intended to function as a security boundary, checker or gatekeeper on untrusted or malicious data inputs. It should not be passed user-generated data or be used directly in web applications. 
 

docs/api.md

@@ -7,7 +7,6 @@
    :backlinks: none
 ```
 
-
 ## scores.continuous
 ```{eval-rst}
 .. automodule:: scores.continuous
@@ -54,6 +53,18 @@
     :members:
 ```
 
+## scores.spatial
+```{eval-rst}
+.. autofunction:: scores.spatial.fss_2d
+.. autofunction:: scores.spatial.fss_2d_binary
+.. autofunction:: scores.spatial.fss_2d_single_field
+```
+
+## scores.stats
+```{eval-rst}
+.. autofunction:: scores.stats.statistical_tests.diebold_mariano
+```
+
 ## scores.processing
 ```{eval-rst}
 .. autofunction:: scores.processing.isotonic_fit
@@ -72,14 +83,10 @@
 .. autofunction:: scores.processing.cdf.cdf_envelope
 ```
 
-## scores.stats
-```{eval-rst}
-.. autofunction:: scores.stats.statistical_tests.diebold_mariano
-```
-
 ## scores.pandas
 ```{eval-rst}
 .. autofunction:: scores.pandas.continuous.mse
 .. autofunction:: scores.pandas.continuous.rmse
 .. autofunction:: scores.pandas.continuous.mae
-```
+```
+

docs/conf.py

@@ -9,7 +9,7 @@
 
 project = "scores"
 copyright = "Licensed under Apache 2.0 - https://www.apache.org/licenses/LICENSE-2.0"
-release = "0.8.4"
+release = "0.8.5"
 
 version = __version__
 
@@ -38,8 +38,11 @@
 html_theme = "sphinx_book_theme"
 html_theme_options = {
     "repository_url": "https://github.com/nci/scores",
+    "canonical_url": "https://scores.readthedocs.io/en/latest/",
     "use_repository_button": True,
+    "show_toc_level": 3,
 }
+html_baseurl = "https://scores.readthedocs.io/en/latest/"
 
 # -- nbsphinx ---------------------------------------------------------------
 # This is processed by Jinja2 and inserted after each notebook

docs/included.md

@@ -26,7 +26,7 @@
   - 
     [Griffiths et al. (2019)](https://doi.org/10.1002/met.1732); [Griffiths et al. (2021)](https://doi.org/10.1071/ES21010)
 * -  
-    - Flip-Flop Index - proportion exceeding
+    - Flip-Flop Index - Proportion Exceeding
   -
     [API](https://scores.readthedocs.io/en/latest/api.html#scores.continuous.flip_flop_index_proportion_exceeding)
   - 
@@ -72,15 +72,15 @@
   - 
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Murphy_Diagrams.html)
   - 
-    [Ehm et al. (2016) - Theorem 1](https://www.jstor.org/stable/24775351); [Taggart (2022) - Theorem 5.3](https://doi.org/10.1214/21-ejs1957)
+    [Ehm et al. (2016) - Theorem 1](https://doi.org/10.1111/rssb.12154); [Taggart (2022) - Theorem 5.3](https://doi.org/10.1214/21-ejs1957)
 * -  
-    - Murphy Score (Mean Elementary Score) - theta values
+    - Murphy Score (Mean Elementary Score) - Theta Values
   -
     [API](https://scores.readthedocs.io/en/latest/api.html#scores.continuous.murphy_thetas)
   - 
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Murphy_Diagrams.html)
   - 
-    [Ehm et al. (2016) - Corollary 2 (p.521)](https://www.jstor.org/stable/24775351); [Taggart (2022) - Corollary 5.6](https://doi.org/10.1214/21-ejs1957)
+    [Ehm et al. (2016) - Corollary 2 (p.521)](https://doi.org/10.1111/rssb.12154); [Taggart (2022) - Corollary 5.6](https://doi.org/10.1214/21-ejs1957)
 * - Pearson's Correlation Coefficient
   - [API](https://scores.readthedocs.io/en/latest/api.html#scores.continuous.correlation)
   - [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Pearsons_Correlation.html)
@@ -175,15 +175,15 @@
   - 
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Murphy_Diagrams.html)
   - 
-    [Ehm et al. (2016) - Theorem 1](https://www.jstor.org/stable/24775351); [Taggart (2022) - Theorem 5.3](https://doi.org/10.1214/21-ejs1957)
+    [Ehm et al. (2016) - Theorem 1](https://doi.org/10.1111/rssb.12154); [Taggart (2022) - Theorem 5.3](https://doi.org/10.1214/21-ejs1957)
 * -  
-    - Murphy Score (Mean Elementary Score) - theta values
+    - Murphy Score (Mean Elementary Score) - Theta Values
   -
     [API](https://scores.readthedocs.io/en/latest/api.html#scores.probability.murphy_thetas)
   - 
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Murphy_Diagrams.html)
   - 
-    [Ehm et al. (2016) - Corollary 2 (p.521)](https://www.jstor.org/stable/24775351); [Taggart (2022) - Corollary 5.6](https://doi.org/10.1214/21-ejs1957)
+    [Ehm et al. (2016) - Corollary 2 (p.521)](https://doi.org/10.1111/rssb.12154); [Taggart (2022) - Corollary 5.6](https://doi.org/10.1214/21-ejs1957)
 * - Receiver (Relative) Operating Characteristic (ROC)
   - [API](https://scores.readthedocs.io/en/latest/api.html#scores.probability.roc_curve_data)   
   - [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/ROC.html)
@@ -215,6 +215,14 @@
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
   - 
     [Accuracy (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#ACC)
+* -  
+    - Base Rate
+  -
+    [API](https://scores.readthedocs.io/en/latest/api.html#scores.categorical.BasicContingencyManager.base_rate)
+  - 
+    [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
+  - 
+    [Hogan and Mason (2011)](https://doi.org/10.1002/9781119960003.ch3)
 * -  
     - Bias Score (Frequency Bias)
   -
@@ -246,7 +254,7 @@
   - 
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
   - 
-     [Equitable Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#ETS)
+    [Hogan et al. (2010)](https://doi.org/10.1175/2009WAF2222350.1); [Equitable Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#ETS)
 * -  
     - F1 Score
   -
@@ -271,6 +279,14 @@
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
   - 
     [False alarm ratio (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#FAR)
+* -  
+    - Forecast Rate
+  -
+    [API](https://scores.readthedocs.io/en/latest/api.html#scores.categorical.BasicContingencyManager.forecast_rate)
+  - 
+    [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
+  - 
+    [Hogan and Mason (2011)](https://doi.org/10.1002/9781119960003.ch3)
 * -  
     - Fraction Correct (Accuracy)
   -
@@ -294,7 +310,7 @@
   - 
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
   - 
-     [Equitable Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#ETS) 
+    [Hogan et al. (2010)](https://doi.org/10.1175/2009WAF2222350.1); [Equitable Threat score (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#ETS) 
 * -  
     - Hanssen and Kuipers' Discriminant (Peirce's Skill Score, True Skill Statistic)
   -
@@ -342,7 +358,7 @@
   - 
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
   - 
-    [Hanssen and Kuipers discriminant (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#HK)
+    [Peirce (1884)](https://doi.org/10.1126/science.ns-4.93.453.b); [Hanssen and Kuipers discriminant (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#HK)
 * -  
     - Precision (Success Ratio)
   -
@@ -375,6 +391,14 @@
     [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
   - 
     [Wikipedia](https://en.wikipedia.org/wiki/Precision_and_recall); [Probability of detection (WWRP/WGNE Joint Working Group on Forecast Verification Research)](https://www.cawcr.gov.au/projects/verification/#POD)
+* -  
+    - Symmetric Extremal Dependence Index (SEDI)
+  -
+    [API](https://scores.readthedocs.io/en/latest/api.html#scores.categorical.BasicContingencyManager.symmetric_extremal_dependence_index)
+  - 
+    [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Binary_Contingency_Scores.html)
+  - 
+    [Ferro and Stephenson (2011)](https://doi.org/10.1175/WAF-D-10-05030.1)
 * -  
     - Sensitivity (Hit Rate, Probability of Detection (POD), True Positive Rate, Recall)
   -
@@ -473,6 +497,45 @@
     —    
 ```
 
+## Spatial
+
+```{list-table}
+:header-rows: 1
+
+* - Name (Alphabetical order)
+  - API
+  - Tutorial
+  - Reference(s)
+* - Fractions Skill Score (FSS)
+  - 
+  - 
+  - 
+* -  
+    - FSS - 2D
+  -
+    [API](https://scores.readthedocs.io/en/latest/api.html#scores.spatial.fss_2d)
+  - 
+    [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Fractions_Skill_Score.html)
+  - 
+    [Roberts and Lean (2008)](https://doi.org/10.1175/2007mwr2123.1); [Mittermaier (2021)](https://doi.org/10.1175/mwr-d-18-0106.1)
+* -  
+    - FSS - 2D Binary
+  -
+    [API](https://scores.readthedocs.io/en/latest/api.html#scores.spatial.fss_2d_binary)
+  - 
+    [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Fractions_Skill_Score.html)
+  - 
+    —
+* -  
+    - FSS - 2D Single Field
+  -
+    [API](https://scores.readthedocs.io/en/latest/api.html#scores.spatial.fss_2d_single_field)
+  - 
+    [Tutorial](https://scores.readthedocs.io/en/latest/tutorials/Fractions_Skill_Score.html)
+  - 
+    [Roberts and Lean (2008)](https://doi.org/10.1175/2007mwr2123.1); [Faggian et al. (2015)](https://doi.org/10.54302/mausam.v66i3.555)
+```
+
 ## Statistical Tests
 
 ```{list-table}

docs/related_works.md

@@ -3,21 +3,31 @@
 Here are some related packages that may be of interest.
 
 - **`xskillscore`**
-	- Open source Python package that provides verification metrics of deterministic (and probabilistic from properscoring) forecasts with xarray.
-	- [Software Repository](https://github.com/xarray-contrib/xskillscore); [Documentation](https://xskillscore.readthedocs.io/en/latest/); [Reference](https://doi.org/10.5281/zenodo.5173153)
+	- Open source Python package that provides verification metrics of deterministic (and probabilistic from [`properscoring`](https://github.com/properscoring/properscoring)) forecasts with xarray.
+	- [Software Repository](https://github.com/xarray-contrib/xskillscore); [Documentation](https://xskillscore.readthedocs.io/en/latest/)
+	- Reference: [Bell et al. (2021)](https://doi.org/10.5281/zenodo.5173153) 	
 
 - **`climpred`**
 	- Open source Python package for verification of weather and climate forecasts.
-	- [Software Repository](https://github.com/pangeo-data/climpred); [Documentation](https://climpred.readthedocs.io/en/stable/); [Reference](https://doi.org/10.21105/joss.02781)
+	- [Software Repository](https://github.com/pangeo-data/climpred); [Documentation](https://climpred.readthedocs.io/en/stable/) 
+	- Reference: [Brady and Spring (2021)](https://doi.org/10.21105/joss.02781)
 
 - **`Verif`**  
 	- Open source command-line tool for forecast verification. Generates verification plots. Can evaluate deterministic and probabilistic predictions.
-	- [Software Repository](https://github.com/WFRT/verif); [Wiki](https://github.com/WFRT/verif/wiki); [Reference](https://doi.org/10.1175/bams-d-22-0253.1)
+	- [Software Repository](https://github.com/WFRT/verif); [Wiki](https://github.com/WFRT/verif/wiki) 
+	- Reference: [Nipen et al. (2023)](https://doi.org/10.1175/bams-d-22-0253.1)
 
 - **`METplus`**
-	- `METplus` includes a database and visualisation system with python wrappers to utilise the `MET` package. Verification scores in `MET` are implemented in C++. 
-	- 	[Website](https://dtcenter.org/community-code/metplus); [Software Repository](https://github.com/dtcenter/METplus); [Documentation](https://metplus.readthedocs.io/en/latest/); [Reference](https://doi.org/10.1175/bams-d-19-0093.1) 
-
-
-
-
+	- `METplus` includes a database and visualisation system, with Python and shell script wrappers to utilise the [`MET`](https://github.com/dtcenter/MET) package. Verification scores in [`MET`](https://github.com/dtcenter/MET) are implemented in C++. 
+	- 	[Software Repository](https://github.com/dtcenter/METplus); [Documentation](https://metplus.readthedocs.io/en/latest/); [Website](https://dtcenter.org/community-code/metplus)
+	- Reference: [Brown et al. (2021)](https://doi.org/10.1175/bams-d-19-0093.1) 
+
+- **`Pysteps`** 
+	- Open source Python library for short-term ensemble prediction systems, focusing on probabilistic nowcasting of radar precipitation fields. Includes a significant verification submodule.
+	- [Software Repository](https://github.com/pySTEPS/pysteps); [Documentation](https://pysteps.readthedocs.io/en/stable/); [Website](https://pysteps.github.io/)
+	- References: [Pulkkinen et al. (2019)](https://doi.org/10.5194/gmd-12-4185-2019); [Imhoff et al. (2023)](https://doi.org/10.1002/qj.4461)
+
+- **`PyForecastTools`**
+	- Open source Python module for model validation, forecast verification and plot generation. Uses dmarray data structures from [`SpacePy`](https://github.com/spacepy/spacepy).
+	- [Software Repository](https://github.com/drsteve/PyForecastTools); [Documentation](https://drsteve.github.io/PyForecastTools/)
+	- Reference: [Morley and Burrell (2020)](https://doi.org/10.5281/zenodo.3764117)

src/scores/__init__.py

@@ -13,7 +13,7 @@
 import scores.sample_data
 import scores.stats.statistical_tests  # noqa: F401
 
-__version__ = "0.8.4"
+__version__ = "0.8.5"
 
 __all__ = [
     "scores.categorical",