llama.cpp performance numbers

[JohannesGaessler]

I've started a Github page for collecting llama.cpp performance numbers.
It's still very much WIP; currently there are no GPU benchmarks.
I'll probably at some point write scripts to automate data collection and add them to the corresponding git repository (once they're somewhat mature I'll make a PR for the llama.cpp main repository).

[ianscrivener]

good stuff @JohannesGaessler. These are really interesting insights.. especially about relative CPU performance!

BTW: @soleblaze and I are also looking at performance and benchmarking... we are all taking different approaches with different goals.

[ianscrivener]

is your test command like?
time ./main -c 2048 -p "" -t 8 -n 128 -m ./models/7B/ggml-llama-7b-q4_0.bin

[JohannesGaessler]

The exact command that I use is usually a variant of

./main --model models/nvme/llama-${model_size}-ggml-${quantization}.bin --seed 1337 --ignore-eos --n-predict 128 --ctx-size 2048 --batch-size 512 --threads 1 --n-gpu-layers 99 | tee chat.txt

And then I take the number provided by llama.cpp for t/s.

[JohannesGaessler]

I should mention: to my knowledge the "eval time" metric does not include CPU overhead from e.g. sampling or creating ggml graphs. For CPU benchmarking this does not matter but once you get to something like 100 t/s it becomes an issue and the actual tokens generated per second are ~10% lower.