Could I run VAD model (jit or onnx) on GPU

[mironnn]

❓ Questions and Help

Hi.
Thank you for the provided repo and models.
Wiki says, that "Using batching or GPU can also improve performance considerably".
I've tried to run VAD models of GPU and faced errors.
Could I run jit or Onnx models on GPU?
Thanks

[snakers4]

Could I run jit or Onnx models on GPU?

It does not make much practical sense, unless you have some very specific conditions or large batches, but you can try.

Make sure to read this before proceeding:

• Are different samples in a batch processed independently? #142

[trzy]

GPU inference would definitely be helpful :) I have a use case where I need to proceed multiple hours of sound captured by a wearable device.

[Dan1aR]

@mironnn
Hi! I ran into the same need of using Silero VAD on GPU
Did you succeeded in it?
Because im facing this issue and can't figure it out:

# Loading model on gpu
device = torch.device("cuda")
model = torch.jit.load("silero_vad.jit", map_location=device)

The while inference chunk-wise with this code:

model.reset_states()
audio = torch.rand([200*1536]).to(device)

audio_length_samples = len(audio)
window_size_samples = 512
sampling_rate = 16_000

speech_probs = []
for current_start_sample in tqdm(range(0, audio_length_samples, window_size_samples)):
    chunk = audio[current_start_sample: current_start_sample + window_size_samples]
    if len(chunk) < window_size_samples:
        chunk = torch.nn.functional.pad(chunk, (0, int(window_size_samples - len(chunk))))
    speech_prob = model(chunk, sampling_rate).item()
    speech_probs.append(speech_prob)
len(f"{len(speech_probs)=}")

On 3-rd iteration i get:

RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: The following operation failed in the TorchScript interpreter.
Traceback of TorchScript (most recent call last):
RuntimeError: vector::_M_range_check: __n (which is 18446744073709551615) >= this->size() (which is 3)

Can you please help me understand why is this happening and how to fix it?

Versions:
torch 1.12.1
cuda 11.6