Sorting analyzer sometimes does not work

[wei-xu-edinburgh]

Hi. My spikes are sorted with kilosort4 on a neuropixel 1 probe. Sorting analyzer works for some sessions but not for others. I get the error:

estimate_sparsity (workers: 10 processes): 100%|██████████| 1830/1830 [06:19<00:00, 4.83it/s]
compute_waveforms (workers: 10 processes): 100%|██████████| 1830/1830 [07:11<00:00, 4.24it/s]
noise_level (workers: 10 processes): 100%|██████████| 20/20 [00:00<00:00, 34.29it/s]
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\spawn.py", line 132, in _main
self = reduction.pickle.load(from_parent)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: pickle data was truncated
Traceback (most recent call last):
File "", line 1, in
File "C:\Program Files\JetBrains\PyCharm 2024.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_umd.py", line 197, in runfile
pydev_imports.execfile(filename, global_vars, local_vars) # execute the script
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\JetBrains\PyCharm 2024.3.2\plugins\python-ce\helpers\pydev_pydev_imps_pydev_execfile.py", line 18, in execfile
exec(compile(contents+"\n", file, 'exec'), glob, loc)
File "Z:\M1_thalamus\tasksignals\sorting_analyzer.py", line 39, in
main()
File "Z:\M1_thalamus\tasksignals\sorting_analyzer.py", line 34, in main
sorting_analyzer.compute(['random_spikes', 'templates', 'spike_amplitudes', 'spike_locations', 'waveforms', 'noise_levels',
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\sortinganalyzer.py", line 1461, in compute
self.compute_several_extensions(extensions=extensions, save=save, verbose=verbose, **job_kwargs)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\sortinganalyzer.py", line 1614, in compute_several_extensions
results = run_node_pipeline(
^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\node_pipeline.py", line 631, in run_node_pipeline
processor.run(recording_slices=recording_slices)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\site-packages\spikeinterface\core\job_tools.py", line 517, in run
results = executor.map(process_function_wrapper, recording_slices)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 851, in map
results = super().map(partial(_process_chunk, fn),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures_base.py", line 608, in map
fs = [self.submit(fn, *args) for args in zip(*iterables)]
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 822, in submit
self._adjust_process_count()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 781, in _adjust_process_count
self._spawn_process()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\concurrent\futures\process.py", line 799, in _spawn_process
p.start()
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\context.py", line 337, in _Popen
return Popen(process_obj)
^^^^^^^^^^^^^^^^^^
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\popen_spawn_win32.py", line 95, in init
reduction.dump(process_obj, to_child)
File "C:\Users\Wei Xu.conda\envs\sortinganalyzer_test\Lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
OSError: [Errno 22] Invalid argument

[chrishalcrow]

Hey @wei-xu-edinburgh - can you give some more details? Mostly: which OS are you using, and do you set n_jobs somewhere? Either in set_global_n_jobs or somewhere else?

@zm711 also says it could be that the connection to your external network dropped at some point. Which seems very possible if you're using the Edinburgh DataStore ;)

[wei-xu-edinburgh]

Hi @chrishalcrow: The operating system is windows11, using pycharm, and si.set_global_job_kwargs(n_jobs=10)

[zm711]

I will say we use networked mapped drives from Windows all the time and if there is any network issues than multiprocessing tends to fail typically with Popen based errors. So that would be the first I would check. Your options would be try to do the same analysis locally and see if you still see multiprocessing failures or turn off multiprocessing (which probably isn't ideal!)

[wei-xu-edinburgh]

I tried to getting sorting analyzer to save locally - the problem persists. However, removing parallel processing seems to solve the issue.

[chrishalcrow]

@zm711 , you think this is a problem of the combination (parallel processing) + (network drive)? So @wei-xu-edinburgh 's solution of using n_jobs=1 works, but so would moving the recording to a local drive and using lots of jobs?

[zm711]

yeah the issue is the process pool getting messed up. The pool is only made if using n_jobs > 1. so this specific problem shouldn't happen if you're not making PoolExecutor. This should only happen if you invalidate the pool in some way. The most common way is for drive access issues (which for a network drive could mean poor wifi, computer going to sleep). Typically you don't have drive access issues locally. So I think we would need to know exactly how you're doing it. I would not recommend getting rid of parallelization as it will be quite slow for most recordings. I think it is better for us to figure out why your pool is getting broken.