Brittle Handling of failed utils/stats.kde(...) calls in utils/stats.df_percentile

[CorbanSwain]

CaImAn/caiman/utils/stats.py

Lines 183 to 192 in c873b03

	err = True
	while err:
	try:
	bandwidth, mesh, density, cdf = kde(inputData)
	err = False
	except:
	logging.warning('Percentile computation failed. Duplicating ' + 'and trying again.')
	if not isinstance(inputData, list):
	inputData = inputData.tolist()
	inputData += inputData

Given the broad catching of all exceptions; unexpected errors in kde(...) (i.e. any error not resolved by lengthening the input list) can lead to an infinite loop and exponential memory usage. I believe this was hanging Python when I encountered issue #1281 .

Example of my hotfix (but not necessarily to be duplicated) ... I think that the best solution would be to add the explicit exceptions intended to be caught when kde() fails after the except keyword.

def df_percentils(...):
   # ...
   if axis is not None:
      # ...
   else:
        RETRY_LIMIT = 16
        retry_count = 0
        err = True
        while err:
            try:
                bandwidth, mesh, density, cdf = kde(inputData)
                err = False
            except AttributeError:
                raise
            except:
                retry_count += 1
                if retry_count > RETRY_LIMIT:
                    raise

                logging.warning('Percentile computation failed. Duplicating ' + 'and trying again.')
                if not isinstance(inputData, list):
                    inputData = inputData.tolist()
                inputData += inputData

[pgunn]

This will take some thought; your approach seems reasonable, but I wonder if we might be able to detect how things are going wrong without retrying. If not, we might end up adopting roughly this solution.

[kushalkolar]

This issue is also discussed here: #1262

[EricThomson]

Yes duplicate. I'm working on this should fix this week.

Edit: thanks for bringing it up, indeed it is a terrible design!

[pgunn]

Closing this as a dup