Docker image returning an error for KS2, KS2_5, KS3, KS4, and IronClust

[SebastianMarten]

Hi,
I am a very new user of Spikeinterface (si) and am having trouble with using the docker_image option. I would like to run a 30 min segment with different sorters, thus si would be perfect in my case. I am using Python 3.13., Spikeinterface 0.102.3 and Docker Desktop on a Windows PC. Kilosort 2, 2.5 and 3 throw the same error: Exception Kilosort returned a non-zero exit code. Kilosort 4 on the other hand prints a RuntimeError related to CUDA and Ironclust returns a ModuleNotFoundError.

Note. After posting this I attempted tried to run Kilosort2 and 2.5 on my computer locally but failed running the mexGPUall.m file from Kilosort2_5 (https://github.com/jiumao2/Kilosort_2_5) I am getting this error (despite using Visual Studio Community 2022:

---------------------------------------------------------------------------
-- unsupported Microsoft Visual Studio version! Only the versions between 2017 and 2022 (inclusive) are supported! The nvcc flag'-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk. 

Providing the nvcc flag does not solve the issue.

For Kilosort 2, 2.5, and 3 I am getting this error in the .log file:

> Warning: X does not support locale C.UTF-8
> Time   0s. Computing whitening matrix.. 
> Getting channel whitening matrix... 
> ----------------------------------------Error using gpuArray.zeros
> GPU device is not supported because it has a higher compute capability (12.0) than is natively supported by the CUDA libraries. Use parallel.gpu.enableCUDAForwardCompatibility(true) to enable forward compatibility for GPU devices. If you enable forward compatibility, you might see unexpected behavior in GPU computations.

Together this makes me think that the issue is related to my GPU (NVIDIA RTX 5070 Ti) and compiler not being support by the older KS versions and Ironclust.

I would greatly appreciate any help on this!

Kilosort 2

sorting = si.run_sorter(sorter_name='kilosort2', recording=rec, folder = r'D:\GL1060_20210817_sleep4_g0_imec0\data\kilosort2_output', docker_image=True, remove_existing_folder=True, with_output=True)

---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Cell In[7], line 1
----> 1 sorting = si.run_sorter(sorter_name='kilosort2', recording=rec, folder = r'D:\GL1060_20210817_sleep4_g0_imec0\data\kilosort2_output', docker_image=True, remove_existing_folder=True, with_output=True)

> File c:\Users\sebastian.marten\AppData\Local\anaconda3\envs\new\Lib\site-packages\spikeinterface\sorters\runsorter.py:192, in run_sorter(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, output_folder, **sorter_params)
>     186         if not has_spython():
>     187             raise RuntimeError(
>     188                 "The python `spython` package must be installed to "
>     189                 "run singularity. Install with `pip install spython`"
>     190             )
> --> 192     return run_sorter_container(
>     193         container_image=container_image,
>     194         mode=mode,
>     195         **common_kwargs,
>     196     )
>     198 return run_sorter_local(**common_kwargs)
> 
> File c:\Users\sebastian.marten\AppData\Local\anaconda3\envs\new\Lib\site-packages\spikeinterface\sorters\runsorter.py:647, in run_sorter_container(sorter_name, recording, mode, container_image, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, delete_container_files, extra_requirements, installation_mode, spikeinterface_version, spikeinterface_folder_source, output_folder, **sorter_params)
>     645 if run_error:
>     646     if raise_error:
> --> 647         raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
>     648 else:
>     649     if with_output:
> 
> SpikeSortingError: Spike sorting in docker failed with the following error:
> /GL1060_20210817_sleep4_g0_imec0/data/in_container_sorter_script.py:23: DeprecationWarning: `output_folder` is deprecated and will be removed in version 0.103.0 Please use folder instead
>   sorting = run_sorter_local(
> Traceback (most recent call last):
>   File "/GL1060_20210817_sleep4_g0_imec0/data/in_container_sorter_script.py", line 23, in <module>
>     sorting = run_sorter_local(
>   File "/root/.local/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py", line 260, in run_sorter_local
>     SorterClass.run_from_folder(folder, raise_error, verbose)
>   File "/root/.local/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 310, in run_from_folder
>     raise SpikeSortingError(
> spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
> Traceback (most recent call last):
>   File "/root/.local/lib/python3.9/site-packages/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
>     SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
>   File "/root/.local/lib/python3.9/site-packages/spikeinterface/sorters/external/kilosortbase.py", line 217, in _run_from_folder
>     raise Exception(f"{cls.sorter_name} returned a non-zero exit code")
> Exception: kilosort2 returned a non-zero exit code
> 
> Spike sorting failed. You can inspect the runtime trace in /GL1060_20210817_sleep4_g0_imec0/data/kilosort2_output/spikeinterface_log.json.

Kilosort4

sorting = si.run_sorter(sorter_name='kilosort4', recording=rec, folder = r'D:\GL1060_20210817_sleep4_g0_imec0\data\kilosort4_output', docker_image=True, remove_existing_folder=True, with_output=True)

---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Cell In[4], line 1
----> 1 sorting = si.run_sorter(sorter_name='kilosort4', recording=rec, folder = r'D:\GL1060_20210817_sleep4_g0_imec0\data\kilosort4_output', docker_image=True, remove_existing_folder=True, with_output=True)

File c:\Users\sebastian.marten\AppData\Local\anaconda3\envs\new\Lib\site-packages\spikeinterface\sorters\runsorter.py:192, in run_sorter(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, output_folder, **sorter_params)
    186         if not has_spython():
    187             raise RuntimeError(
    188                 "The python `spython` package must be installed to "
    189                 "run singularity. Install with `pip install spython`"
    190             )
--> 192     return run_sorter_container(
    193         container_image=container_image,
    194         mode=mode,
    195         **common_kwargs,
    196     )
    198 return run_sorter_local(**common_kwargs)

File c:\Users\sebastian.marten\AppData\Local\anaconda3\envs\new\Lib\site-packages\spikeinterface\sorters\runsorter.py:647, in run_sorter_container(sorter_name, recording, mode, container_image, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, delete_container_files, extra_requirements, installation_mode, spikeinterface_version, spikeinterface_folder_source, output_folder, **sorter_params)
    645 if run_error:
    646     if raise_error:
--> 647         raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
    648 else:
    649     if with_output:

SpikeSortingError: Spike sorting in docker failed with the following error:
/GL1060_20210817_sleep4_g0_imec0/data/in_container_sorter_script.py:23: DeprecationWarning: `output_folder` is deprecated and will be removed in version 0.103.0 Please use folder instead
  sorting = run_sorter_local(
/home/miniconda3/lib/python3.11/site-packages/torch/cuda/__init__.py:235: UserWarning: 
NVIDIA GeForce RTX 5070 Ti with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_37 sm_90.
If you want to use the NVIDIA GeForce RTX 5070 Ti GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(
Traceback (most recent call last):
  File "/GL1060_20210817_sleep4_g0_imec0/data/in_container_sorter_script.py", line 23, in <module>
    sorting = run_sorter_local(
              ^^^^^^^^^^^^^^^^^
  File "/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py", line 260, in run_sorter_local
    SorterClass.run_from_folder(folder, raise_error, verbose)
  File "/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 310, in run_from_folder
    raise SpikeSortingError(
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/root/.local/lib/python3.11/site-packages/spikeinterface/sorters/external/kilosort4.py", line 316, in _run_from_folder
    ops = compute_preprocessing(ops=ops, device=device, tic0=tic0, file_object=file_object)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/lib/python3.11/site-packages/kilosort/run_kilosort.py", line 500, in compute_preprocessing
    hp_filter = preprocessing.get_highpass_filter(fs, cutoff, device=device)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/miniconda3/lib/python3.11/site-packages/kilosort/preprocessing.py", line 135, in get_highpass_filter
    hp_filter = torch.from_numpy(hp_filter).to(device).float()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: no kernel image is available for execution on the device
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.


Spike sorting failed. You can inspect the runtime trace in /GL1060_20210817_sleep4_g0_imec0/data/kilosort4_output/spikeinterface_log.json.

Ironclust

sorting = si.run_sorter(sorter_name='ironclust', recording=rec, folder = r'D:\GL1060_20210817_sleep4_g0_imec0\data\ironclust_output', docker_image=True, remove_existing_folder=True, with_output=True)

---------------------------------------------------------------------------
SpikeSortingError                         Traceback (most recent call last)
Cell In[5], line 1
----> 1 sorting = si.run_sorter(sorter_name='ironclust', recording=rec, folder = r'D:\GL1060_20210817_sleep4_g0_imec0\data\ironclust_output', docker_image=True, remove_existing_folder=True, with_output=True)

File c:\Users\sebastian.marten\AppData\Local\anaconda3\envs\new\Lib\site-packages\spikeinterface\sorters\runsorter.py:192, in run_sorter(sorter_name, recording, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, output_folder, **sorter_params)
    186         if not has_spython():
    187             raise RuntimeError(
    188                 "The python `spython` package must be installed to "
    189                 "run singularity. Install with `pip install spython`"
    190             )
--> 192     return run_sorter_container(
    193         container_image=container_image,
    194         mode=mode,
    195         **common_kwargs,
    196     )
    198 return run_sorter_local(**common_kwargs)

File c:\Users\sebastian.marten\AppData\Local\anaconda3\envs\new\Lib\site-packages\spikeinterface\sorters\runsorter.py:647, in run_sorter_container(sorter_name, recording, mode, container_image, folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, delete_container_files, extra_requirements, installation_mode, spikeinterface_version, spikeinterface_folder_source, output_folder, **sorter_params)
    645 if run_error:
    646     if raise_error:
--> 647         raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
    648 else:
    649     if with_output:

SpikeSortingError: Spike sorting in docker failed with the following error:
Traceback (most recent call last):
  File "/GL1060_20210817_sleep4_g0_imec0/data/in_container_sorter_script.py", line 4, in <module>
    from spikeinterface import load
ModuleNotFoundError: No module named 'spikeinterface'

[zm711]

Sorry we are little behind on updating our containers. @alejoe91 can double check to see if this is a stale container issue or a script issue when he gets a moment.

[alejoe91]

Hi @SebastianMarten

For KS2, 2.5, 3 this was an error that is fixed in main (#4034)

For KS4, your GPU is too new :P basically the image that we provide is with CUDA 12 is too old for your GPU (and we don't currently have a 4.x GPU to install torch against...In this case, I recommend installing KS4 locally (which is easy)

The Ironclust image was outdated and using python 3.8, which is not supported anymore on the SPikeInterface side. I'm building and pushing a new one with Python3.12 that should fix the issue!

[SebastianMarten]

Hi @alejoe91,

May I ask whether you think I will be able to get those running locally or am I bound to KS4 with this GPU now?

Thanks a lot for the quick response, much appreciated!

[zm711]

May I ask whether you think I will be able to get those running locally or am I bound to KS4 with this GPU now?

Could you explain this question a bit more? Are you asking if you could install the other sorters locally ?

[SebastianMarten]

Hi @zm711,

I am trying to get KS 2, KS 2.5 or KS 4 to run but I am experiencing some issues with all of them. Even when attempting to install them locally I run into problems. From what is understand these issues arise since the graphics card or its required software (e.g., NVIDIA CUDA toolkit) is not compatible with for example Matlab. Thus, my question, whether I should consider downgrading to ensure compatibility. I was hoping you could give me advise on that.

Thanks a lot!

[zm711]

Yeah getting KS2 and KS2.5 to work via Matlab is definitely a pain. One issue is that some versions of Matlab often error during running KS because of a change in one of the functions (I believe the change occurred around 2021/2022)--Aurélien tried to submit a PR to fix the issue but they closed it. So in our group people always use Matlab < 2022. Then you do have to make sure your GPU can work with older Matlab and find a CUDA toolkit that fits between them which is a struggle. For KS4 last I checked it worked pretty well to just follow their repo instructions so I guess some further questions would be:

Confirming you're on Windows (and which version)?
Which GPU do you have (Alessio is good at seeing that info but for me it is easier if it just listed)?
And what errors are you actually getting preventing local install?

EDIT: Also Hi back @SebastianMarten !

[SebastianMarten]

I am working with Windows 11 Enterprise 24H2 and an NVIDIA RTX 5070 Ti. Kilosort 4 worked very well with the standard repo installation. I also got Kilosort 2.5 to run with the following versions and parallel.gpu.enableCUDAForwardCompatibility(true) :

• Microsoft Visual Studio Enterprise 2019 (Desktop development with C++ toolkit)
• NVIDIA CUDA Toolkit 11.0
• MATLAB 2021b

I think I will work with this for now.
Thanks a lot for your quick responses!