Feature request: GPU benchmarks for ML workloads.

[ywilke]

I love what you are doing at sparecores!

I was trying to find good benchmarks that compare the GPU instances that different cloud providers offer. Unfortunately I was unable to find any good comparison of the performance for ML workloads between the different GPU instances. Even when I tried the find a comparison between the GPUs that the cloud instances contain I could not find a good benchmark that compares them.
It would be great if you could add some GPU benchmarks to sparecores that would test common ML workloads like LLM/resnet training/inference. I am not sure if there is already a good benchmark suite that you could run.

Maybe this repository is not the correct place for feature requests. Let me know if you want to move it somewhere else.

[daroczig]

Thanks for this request! We are actually currently working on LLM inference speed benchmarks, which I was hoping to ship in a week or so, but we hit a problem with llama-bench from llama.cpp scaling to multiple GPUs [ggml-org/llama.cpp/discussions/11236]. We will see if we can resolve it, or we might need to write up custom benchmarking scripts supporting both CPU and (multi)GPU use cases from tiny to larger models. I will let you know about this here.

We also have plans to support other benchmarks, e.g. we started GBM model training benchmarks on CPU and GPU as well following @szilard's related benchmarks, but that was put back to the backlog due to other priorities. I think we can pick it up after the above-mentioned LLM-speed updates.

[ywilke]

Great to hear that! Looking forward to see the data once it is ready.

P.S. I will add two more feature requests. Maybe you are also already working on those but just wanted to mention that there is interest for them.

[daroczig]

A PR for the LLM inference speed benchmarks is now open at SpareCores/sc-images#1 -- @ywilke, could you please take a look?

I'd love to hear any feedback, e.g. if the covered benchmark scenarios (prompt processing with 16-32k tokens and text generation with 1-8k tokens), the selected models etc seem to be useful, or would you have any recommendations.

[ywilke]

That’s great to see! Unfortunately, I don’t have much insight into how specific ML workloads perform better on one GPU versus another. From what I’ve heard, workloads are often constrained by either memory or compute, but I’m not sure how much that applies differently to LLMs compared to models like ResNet. It would also be interesting to see a distinction between training and inference speed—are there significant differences in how these bottlenecks manifest for LLM inference? Looking forward to the benchmarks!

[daroczig]

Hey @ywilke,

I'm sorry for the long delay with this. Change in priorities and some unseen delays with cloud compute credits pushed this much further in the future than I hoped .. but at last #173 is now live, so if you visit https://sparecores.com/servers, you should be able to select LLM Inference Speed benchmarks -- which has been evaluated for ~500 servers, and will finish the remaining ~1500 in the next couple of days as well.

Please let us know if you have any feedback.

Other GPU benchmarks are expected to follow e.g. for training GBM -- I will report back, but it will take a couple of weeks for sure due to yet another change in priorities :)

Thanks,
Gergely

[ywilke]

Hi Gergely,

Thanks for this update! It will be interesting to see if the different benchmarks correlate highly and if it is possible pick a single score representative for general inference/training speed.
Looking forward to future updates!