Handle problem with dff auto-percentile calculation #1288
Conversation
caiman/utils/stats.py
Outdated
| @@ -169,6 +169,8 @@ def df_percentile(inputData, axis=None): | |||
| Extracting the percentile of the data where the mode occurs and its value. | |||
| Used to determine the filtering level for DF/F extraction. Note that | |||
| computation can be inaccurate for short traces. | |||
|
|
|||
| If errors occur, will just return fallback value of median (50% percentile) | |||
There was a problem hiding this comment.
If errors occur, returns fallback value ...
caiman/utils/stats.py
Outdated
| while err: | ||
| err_count = 0 | ||
| max_err_count = 10 | ||
| while err==True and err_count < max_err_count: |
There was a problem hiding this comment.
while err and err_count...
Testing boolean values for ==True is an antipattern
caiman/utils/stats.py
Outdated
| @@ -241,12 +254,13 @@ def kde(data, N=None, MIN=None, MAX=None): | |||
| SqDCTData = (DCTData[1:] / 2)**2 | |||
|
|
|||
| # The fixed point calculation finds the bandwidth = t_star | |||
| # fixed_point is function defined below | |||
There was a problem hiding this comment.
I edited it a little. Will be nice to come back at some point and document better. The matlab code that it's taken from is slightly better documented: it's a useful method for finding the optimal width of a gaussian kernel to smooth data so you can get a kernel density estimate (estimate of a probability density). Original Matlab code:
https://www.mathworks.com/matlabcentral/fileexchange/14034-kernel-density-estimator
There was a problem hiding this comment.
Oh: it was leftover debugging note to myself I removed. fixed_point was argument to function, and I was wondering where it came from
Description
When
flag_autowas set toTruein dff calculation, there was a significant chance that things could end up in an infinite while loop.In the kernel density estimator, an exception which was rarely thrown was exposed recently with some scipy deprecations, so this has started happening a lot lately. It was handled previously with printing 'oops' and returning
None. This PR fixes that, stops the while loop from recurring indefinitely. The basic logic wasn't terrible it's fine to usetry/exceptblocks as control elements in Python, it's just a bad idea to let them go on indefintely. I patched that up, found the explicit exception, and tried to make the logic a little more clear.Fixes # (issue)
#1262 , #1274 , #1283
Type of change
Has your PR been tested?
caimanmanager test ran fine (demotest is currently running will report back if any problems).
I pushed pretty hard against the
detrend_df_f()function in the two main demo notebooks with different permutations of the kwargs. It behaved as expected with the logger outputting reasonable warnings (e.g., when using short time windows for percentiles).