deepspeed using multi - GPUs

[zhangyuanxun]

Hello,

I'm using deepspeed inference with multiple GPUs. However, I have observed that there is no difference in performance between using a single GPU and multiple GPUs when the data (batch size = 1). Is this expected behavior, or could there be an issue with my code?

Thank you.

[EeyoreLee]

this's normal. even slower than using one single gpu since the calculation must be done step by step when batch size is one. unless the cuda cores aren't enough and MP use tensor-parallelism and the bandwidth is much high, maybe it will be faster than using one single gpu.