Weighted variance computation for sparse and dense arrays #6205
Conversation
markotoplak
added
the
dask
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #6205 +/- ##
=======================================
Coverage 86.65% 86.66%
=======================================
Files 316 316
Lines 67995 68021 +26
=======================================
+ Hits 58924 58950 +26
Misses 9071 9071 |
markotoplak
force-pushed
the
stats-nan-mean-variance
branch
2 times, most recently
from
e133a23 to
1cadd90
Compare
markotoplak
changed the title
Orange/statistics/util.py
Outdated
| @@ -476,6 +476,45 @@ def nanmean(x, axis=None, weights=None): | |||
| return means | |||
|
|
|||
|
|
|||
| def nan_mean_variance_axis(x, axis=None, weights=None): | |||
There was a problem hiding this comment.
Is there any particular reason why we wouldn't support axis=None in this case? I know sklearn's implementation doesn't support it, but why wouldn't we? Then we could call this function nan_mean_variance, and it would behave more similarly to our other functions here.
There was a problem hiding this comment.
No, no particular reason, it just additional work and should also work differently. With weights=None the input weights should be a matrix. Otherwise I do not think the problem is clearly defined. On a related note, I do not think nanmean works properly with weights when axis is None.
Therefore I avoided it in this PR. What do you think?
There was a problem hiding this comment.
with weights=None the input weights should be a matrix. [...] I do not think nanmean works properly with weights when axis is None.
I think you're right. It looks like the weights are just ignored. I would still prefer this function to be nan_mean_variance, so that if anyone, at some points, decides they need this computation with axis=None, we won't have to go through a renaming cycle, or end up with a second function with the practically same name. I'd be totally fine if this function raises a NotImplementedError if axis is None.
There was a problem hiding this comment.
Done in the last push.
markotoplak
force-pushed
the
stats-nan-mean-variance
branch
from
1cadd90 to
5a3ced8
Compare
markotoplak
changed the title
Implement computation of means and variance for dense or sparse arrays. Both can include NaNs. The computation is weighted and produces the same results as we get from Distributions.
markotoplak
force-pushed
the
stats-nan-mean-variance
branch
from
5a3ced8 to
f0ee0d6
Compare
Issue
Implements weighted variance (and mean) computation that works with sparse, dense. Also handles NaNs.
Needed in #6202.
Merge after #6204.
Includes