Explanation of code design/logic

[manickavela29]

Hi @csukuangfj , @pkufool

It would be great if someone can explain the idea behind unlocking the mutex before DecodeStreams and locking it afterwards?

sherpa-onnx/sherpa-onnx/csrc/online-websocket-server-impl.cc

Line 201 in 1a43d1e

recognizer_->DecodeStreams(s_vec.data(), s_vec.size());

as per my understanding,
several instances of DecodeStreams can run concurrently with different sets of s_vec, is it right?

[csukuangfj]

The lock is to protect s_vec

---

several instances of DecodeStreams can run concurrently with different sets of s_vec, is it right?

Yes, you are right.

Recognizer is thread-safe while OnlineStream is not thread safe.

Several threads can use the same recognizer to decode different streams.

The lock protects data members such as ready_connections_.

[manickavela29]

Thanks for the explanation
How does onnxruntime inference behave with this,

does this mean several batch inference can happen at same time on onnxruntime session ?
or is it kind of queued for onnxruntime to infer?

since DecodeStreams is unlocked, several thread(with unique set of streams under active list) might enter Decodestreams together, how is the onnxrt session resources shared between these, or do they happen sequentially

[manickavela29]

Hi @csukuangfj ,

Thanks for the explanation How does onnxruntime inference behave with this,

does this mean several batch inference can happen at same time on onnxruntime session ? or is it kind of queued for onnxruntime to infer?

since DecodeStreams is unlocked, several thread(with unique set of streams under active list) might enter Decodestreams together, how is the onnxrt session resources shared between these, or do they happen sequentially

[csukuangfj]

Please see microsoft/onnxruntime#114

onnxruntime session is thread-safe.

---

, or do they happen sequentially

No, they run concurrently and independently between threads.

[csukuangfj]

By the way, from my experience, onnxruntime session is thread-safe when you use CPU.
It is not the case when you use CUDA.

[manickavela29]

Thanks, I had the very same question with CUDA and was wondering if it was handled.
but it seems to be working fine 😄, so all good then.

so this concurrency strategy is specific to CPU which is just extended to CUDA(TRT too) right?

recognizer_->DecodeStreams(s_vec.data(), s_vec.size());

[csukuangfj]

so this concurrency strategy is specific to CPU which is just extended to CUDA(TRT too) right?

Everything should be fine if it does not throw errors when you switch to CUDA/TensorRT.
Otherwise, you have to hold the lock.