[Bug]: cuFFT error when running with GPU #45
Description
Contact Details
No response
What happened?
When trying to run with gpu
make run track=test.mp3 gpu=true
I have the following error ⬇️ it might be because of missing cuda and / or torch version? I'm using WSL2 with ubt22.4. But this should not matter with Docker anyway....
In which operating system you found the issue?
Linux
Relevant log output
docker run --rm -i --name=demucs --gpus all -v /home/iart/applications/audio/demucs/input:/data/input -v /home/iart/applications/audio/demucs/output:/data/output -v /home/iart/applications/audio/demucs/models:/data/models xserrat/facebook-demucs:latest python3 -m demucs -n htdemucs --out /data/output --shifts 1 --overlap 0.25 -j 1 "/data/input/test.mp3"
0%| | 0.0/146.25 [00:00<?, ?seconds/s]Selected model is a bag of 1 models. You will see that many progress bars per track.
Separated tracks will be stored in /data/output/htdemucs
Separating track /data/input/test.mp3
0%| | 0.0/146.25 [00:00<?, ?seconds/s]
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/usr/lib/demucs/demucs/__main__.py", line 10, in <module>
main()
File "/usr/lib/demucs/demucs/separate.py", line 168, in main
origin, res = separator.separate_audio_file(track)
File "/usr/lib/demucs/demucs/api.py", line 308, in separate_audio_file
return self.separate_tensor(self._load_audio(file), self.samplerate)
File "/usr/lib/demucs/demucs/api.py", line 271, in separate_tensor
out = apply_model(
File "/usr/lib/demucs/demucs/apply.py", line 216, in apply_model
res = apply_model(sub_model, mix, **kwargs, callback_arg=callback_arg)
File "/usr/lib/demucs/demucs/apply.py", line 251, in apply_model
res = apply_model(model, shifted, **kwargs, callback_arg=callback_arg)
File "/usr/lib/demucs/demucs/apply.py", line 290, in apply_model
chunk_out = future.result() # type: th.Tensor
File "/usr/lib/demucs/demucs/utils.py", line 132, in result
return self.func(*self.args, **self.kwargs)
File "/usr/lib/demucs/demucs/apply.py", line 317, in apply_model
out = model(padded_mix)
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1194, in _call_impl
return forward_call(*input, **kwargs)
File "/usr/lib/demucs/demucs/htdemucs.py", line 538, in forward
z = self._spec(mix)
File "/usr/lib/demucs/demucs/htdemucs.py", line 437, in _spec
z = spectro(x, nfft, hl)[..., :-1, :]
File "/usr/lib/demucs/demucs/spec.py", line 17, in spectro
z = th.stft(x,
File "/usr/local/lib/python3.10/dist-packages/torch/functional.py", line 632, in stft
return _VF.stft(input, n_fft, hop_length, win_length, window, # type: ignore[attr-defined]
RuntimeError: cuFFT error: CUFFT_INTERNAL_ERROR
make: *** [Makefile:54: run] Error 1