more perfo with llamafile tinyblas on x86_64. #10714

Djip007 · 2024-12-08T03:19:15Z

ikawrakow/ik_llama.cpp#71 have a good idea.

I'll figure to add it in llamafile/tinyblas sgemm (and a litle more) and id work great:

AMD Ryzen 9 7940HS (zen4)

Mistral-Nemo-Instruct-2407.BF16.gguf +kv@bf16

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 13B BF16	pp1	2.52	2.51	1.00
llama 13B BF16	pp2	5.00	4.74	0.95
llama 13B BF16	pp3	7.44	7.23	0.97
llama 13B BF16	pp4	9.75	9.91	1.02
llama 13B BF16	pp5	11.95	12.37	1.04
llama 13B BF16	pp6	13.92	14.78	1.06
llama 13B BF16	pp7	15.64	17.09	1.09
llama 13B BF16	pp8	17.24	19.41	1.13
llama 13B BF16	pp9	18.35	21.63	1.18
llama 13B BF16	pp10	19.47	24.02	1.23
llama 13B BF16	pp11	20.48	26.30	1.28
llama 13B BF16	pp12	21.04	28.43	1.35
llama 13B BF16	pp13	21.49	29.41	1.37
llama 13B BF16	pp14	23.10	31.56	1.37
llama 13B BF16	pp15	23.65	33.51	1.42
llama 13B BF16	pp16	23.99	35.87	1.50
llama 13B BF16	pp30	24.19	51.09	2.11
llama 13B BF16	pp32	24.56	51.04	2.08
llama 13B BF16	pp64	25.66	57.23	2.23
llama 13B BF16	pp65	24.57	57.33	2.33
llama 13B BF16	pp120	25.73	65.51	2.55
llama 13B BF16	pp128	25.77	65.81	2.55
llama 13B BF16	pp130	26.86	66.31	2.47
llama 13B BF16	pp240	27.76	70.40	2.54
llama 13B BF16	pp255	26.03	70.32	2.70
llama 13B BF16	pp256	26.04	70.34	2.70
llama 13B BF16	pp510	25.73	68.26	2.65
llama 13B BF16	pp512	25.74	67.97	2.64
llama 13B BF16	pp1024	25.27	66.76	2.64
llama 13B BF16	pp1025	25.04	64.84	2.59
llama 13B BF16	pp2048	24.63	63.96	2.60
llama 13B BF16	tg128	2.52	2.52	1.00

Mistral-Nemo-Instruct-2407.FP16.gguf +kv@fp16

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 13B F16	pp1	2.50	2.50	1.00
llama 13B F16	pp2	4.94	4.81	0.97
llama 13B F16	pp3	7.41	7.19	0.97
llama 13B F16	pp4	9.81	9.82	1.00
llama 13B F16	pp5	12.23	12.25	1.00
llama 13B F16	pp6	7.92	14.58	1.84
llama 13B F16	pp7	9.19	16.52	1.80
llama 13B F16	pp8	10.47	18.76	1.79
llama 13B F16	pp9	11.76	20.86	1.77
llama 13B F16	pp10	22.58	22.80	1.01
llama 13B F16	pp11	14.19	24.90	1.75
llama 13B F16	pp12	15.41	26.70	1.73
llama 13B F16	pp13	16.66	26.99	1.62
llama 13B F16	pp14	17.88	28.74	1.61
llama 13B F16	pp15	31.97	29.67	0.93
llama 13B F16	pp16	19.79	31.23	1.58
llama 13B F16	pp30	38.31	36.86	0.96
llama 13B F16	pp32	29.11	36.60	1.26
llama 13B F16	pp64	32.15	38.95	1.21
llama 13B F16	pp65	38.72	38.91	1.00
llama 13B F16	pp120	39.14	40.36	1.03
llama 13B F16	pp128	35.44	40.19	1.13
llama 13B F16	pp130	39.49	40.24	1.02
llama 13B F16	pp240	36.90	40.76	1.10
llama 13B F16	pp255	35.87	40.66	1.13
llama 13B F16	pp256	33.51	40.43	1.21
llama 13B F16	pp510	27.96	40.09	1.43
llama 13B F16	pp512	27.41	40.08	1.46
llama 13B F16	pp1024	27.27	39.03	1.43
llama 13B F16	pp1025	25.91	38.50	1.49
llama 13B F16	pp2048	26.75	37.95	1.42
llama 13B F16	tg128	2.50	2.51	1.00

on AMD Ryzen 9 5950X 16-Core Processor (znver3) (AVX2)

Mistral-Nemo-Instruct-2407.BF16.gguf +kv@bf16

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 13B BF16	pp1	2.21	2.21	1.00
llama 13B BF16	pp2	4.36	4.31	0.99
llama 13B BF16	pp3	6.44	6.44	1.00
llama 13B BF16	pp4	8.42	8.58	1.02
llama 13B BF16	pp5	10.29	10.71	1.04
llama 13B BF16	pp6	12.00	12.78	1.07
llama 13B BF16	pp7	13.53	14.86	1.10
llama 13B BF16	pp8	14.72	16.92	1.15
llama 13B BF16	pp9	15.61	18.93	1.21
llama 13B BF16	pp10	16.30	20.92	1.28
llama 13B BF16	pp11	16.93	23.01	1.36
llama 13B BF16	pp12	17.35	24.89	1.43
llama 13B BF16	pp13	17.69	26.94	1.52
llama 13B BF16	pp14	17.95	28.78	1.60
llama 13B BF16	pp15	18.21	30.64	1.68
llama 13B BF16	pp16	18.37	32.45	1.77
llama 13B BF16	pp30	19.20	42.87	2.23
llama 13B BF16	pp32	19.36	43.14	2.23
llama 13B BF16	pp64	19.85	45.05	2.27
llama 13B BF16	pp65	19.46	44.94	2.31
llama 13B BF16	pp120	19.98	46.27	2.32
llama 13B BF16	pp128	20.14	46.11	2.29
llama 13B BF16	pp130	19.97	45.93	2.30
llama 13B BF16	pp240	20.23	46.50	2.30
llama 13B BF16	pp255	20.24	46.54	2.30
llama 13B BF16	pp256	20.19	46.40	2.30
llama 13B BF16	pp510	20.09	46.01	2.29
llama 13B BF16	pp512	20.17	45.81	2.27
llama 13B BF16	pp1024	19.94	45.05	2.26
llama 13B BF16	pp1025	19.74	44.18	2.24
llama 13B BF16	pp2048	19.48	43.68	2.24
llama 13B BF16	tg128	2.21	2.21	1.00

Mistral-Nemo-Instruct-2407.FP16.gguf +kv@fp16

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 13B F16	pp1	2.19	2.19	1.00
llama 13B F16	pp2	4.30	4.28	1.00
llama 13B F16	pp3	6.46	6.41	0.99
llama 13B F16	pp4	4.84	8.53	1.76
llama 13B F16	pp5	6.01	10.64	1.77
llama 13B F16	pp6	12.90	12.71	0.99
llama 13B F16	pp7	8.50	14.81	1.74
llama 13B F16	pp8	9.64	16.88	1.75
llama 13B F16	pp9	19.25	18.90	0.98
llama 13B F16	pp10	12.25	20.88	1.70
llama 13B F16	pp11	13.39	22.94	1.71
llama 13B F16	pp12	25.40	24.89	0.98
llama 13B F16	pp13	15.89	26.87	1.69
llama 13B F16	pp14	17.02	28.74	1.69
llama 13B F16	pp15	30.82	30.66	0.99
llama 13B F16	pp16	19.45	32.55	1.67
llama 13B F16	pp30	34.23	54.23	1.58
llama 13B F16	pp32	27.34	55.37	2.03
llama 13B F16	pp64	30.66	58.46	1.91
llama 13B F16	pp65	31.03	57.47	1.85
llama 13B F16	pp120	35.31	58.01	1.64
llama 13B F16	pp128	33.09	57.76	1.75
llama 13B F16	pp130	33.05	58.28	1.76
llama 13B F16	pp240	35.19	58.66	1.67
llama 13B F16	pp255	35.24	58.62	1.66
llama 13B F16	pp256	33.98	58.57	1.72
llama 13B F16	pp510	33.76	57.94	1.72
llama 13B F16	pp512	33.34	57.51	1.72
llama 13B F16	pp1024	33.03	56.41	1.71
llama 13B F16	pp1025	32.59	53.96	1.66
llama 13B F16	pp2048	32.08	54.93	1.71
llama 13B F16	tg128	2.18	2.19	1.00

Djip007 · 2024-12-10T00:49:28Z

Some perplexity with new code. (vs master BF16/zen3)

#> zen3:
./build/bin/./llama-perplexity -ctk bf16 -ctv bf16 --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.BF16.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9452 ±    0.5268       0.00070 ±    0.00062       0.00003 ±    0.00000     0.186 ±  0.018 %    99.608 ±  0.392 %
   2       5.4448 ±    0.6044       0.00187 ±    0.00151       0.00004 ±    0.00000     0.183 ±  0.013 %    99.804 ±  0.196 %
   3       4.6848 ±    0.4030       0.00093 ±    0.00104       0.00006 ±    0.00000     0.255 ±  0.030 %    99.869 ±  0.131 %
   4       5.0051 ±    0.3673       0.00039 ±    0.00080       0.00005 ±    0.00000     0.243 ±  0.024 %    99.902 ±  0.098 %
   5       5.2917 ±    0.3433       0.00004 ±    0.00067       0.00006 ±    0.00000     0.243 ±  0.020 %    99.922 ±  0.078 %
   6       5.8289 ±    0.3542       0.00000 ±    0.00057       0.00006 ±    0.00000     0.233 ±  0.017 %    99.869 ±  0.092 %
   7       6.2242 ±    0.3544       0.00025 ±    0.00054       0.00006 ±    0.00000     0.228 ±  0.015 %    99.832 ±  0.097 %
   8       6.4312 ±    0.3454       0.00041 ±    0.00048       0.00006 ±    0.00000     0.229 ±  0.014 %    99.755 ±  0.110 %
   9       6.8865 ±    0.3580       0.00036 ±    0.00043       0.00006 ±    0.00000     0.227 ±  0.013 %    99.739 ±  0.107 %
  10       7.2362 ±    0.3590       0.00026 ±    0.00039       0.00006 ±    0.00000     0.224 ±  0.012 %    99.765 ±  0.096 %
  11       7.2572 ±    0.3420       0.00018 ±    0.00036       0.00005 ±    0.00000     0.218 ±  0.011 %    99.750 ±  0.094 %
  12       7.2827 ±    0.3297       0.00015 ±    0.00033       0.00007 ±    0.00001     0.230 ±  0.011 %    99.673 ±  0.103 %
  13       7.4379 ±    0.3228       0.00011 ±    0.00031       0.00007 ±    0.00001     0.226 ±  0.011 %    99.608 ±  0.109 %
  14       7.3367 ±    0.3061       0.00016 ±    0.00030       0.00007 ±    0.00001     0.224 ±  0.010 %    99.636 ±  0.101 %
  15       7.1258 ±    0.2859       0.00012 ±    0.00028       0.00007 ±    0.00001     0.222 ±  0.010 %    99.634 ±  0.098 %
  16       7.1695 ±    0.2792       0.00012 ±    0.00026       0.00007 ±    0.00001     0.223 ±  0.009 %    99.657 ±  0.092 %
  17       6.8048 ±    0.2538       0.00008 ±    0.00025       0.00007 ±    0.00001     0.223 ±  0.009 %    99.677 ±  0.086 %
  18       6.8631 ±    0.2517       0.00016 ±    0.00024       0.00007 ±    0.00001     0.221 ±  0.008 %    99.651 ±  0.087 %
  19       6.9983 ±    0.2515       0.00016 ±    0.00023       0.00007 ±    0.00001     0.220 ±  0.008 %    99.670 ±  0.082 %
  20       6.7969 ±    0.2383       0.00013 ±    0.00022       0.00007 ±    0.00001     0.229 ±  0.008 %    99.667 ±  0.081 %

./build/bin/./llama-perplexity --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.BF16.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9430 ±    0.5263       0.00016 ±    0.00050       0.00003 ±    0.00000     0.159 ±  0.014 %    100.000 ±  0.000 %
   2       5.4442 ±    0.6042       0.00177 ±    0.00153       0.00003 ±    0.00000     0.155 ±  0.009 %    99.804 ±  0.196 %
   3       4.6852 ±    0.4029       0.00103 ±    0.00104       0.00003 ±    0.00000     0.179 ±  0.010 %    99.869 ±  0.131 %
   4       5.0069 ±    0.3673       0.00074 ±    0.00079       0.00003 ±    0.00000     0.178 ±  0.009 %    99.902 ±  0.098 %
   5       5.2939 ±    0.3435       0.00046 ±    0.00064       0.00003 ±    0.00000     0.171 ±  0.008 %    99.922 ±  0.078 %
   6       5.8312 ±    0.3544       0.00039 ±    0.00053       0.00003 ±    0.00000     0.165 ±  0.007 %    99.935 ±  0.065 %
   7       6.2260 ±    0.3545       0.00055 ±    0.00050       0.00003 ±    0.00000     0.165 ±  0.006 %    99.888 ±  0.079 %
   8       6.4317 ±    0.3454       0.00047 ±    0.00044       0.00003 ±    0.00000     0.169 ±  0.007 %    99.853 ±  0.085 %
   9       6.8872 ±    0.3580       0.00047 ±    0.00039       0.00003 ±    0.00000     0.168 ±  0.006 %    99.826 ±  0.087 %
  10       7.2376 ±    0.3590       0.00045 ±    0.00036       0.00003 ±    0.00000     0.167 ±  0.006 %    99.804 ±  0.088 %
  11       7.2592 ±    0.3421       0.00045 ±    0.00033       0.00003 ±    0.00000     0.166 ±  0.005 %    99.822 ±  0.080 %
  12       7.2841 ±    0.3298       0.00033 ±    0.00031       0.00003 ±    0.00000     0.172 ±  0.005 %    99.837 ±  0.073 %
  13       7.4398 ±    0.3229       0.00036 ±    0.00029       0.00003 ±    0.00000     0.171 ±  0.005 %    99.849 ±  0.067 %
  14       7.3379 ±    0.3062       0.00033 ±    0.00027       0.00003 ±    0.00000     0.168 ±  0.005 %    99.860 ±  0.063 %
  15       7.1275 ±    0.2859       0.00035 ±    0.00025       0.00003 ±    0.00000     0.167 ±  0.005 %    99.843 ±  0.064 %
  16       7.1714 ±    0.2793       0.00039 ±    0.00024       0.00003 ±    0.00000     0.171 ±  0.005 %    99.828 ±  0.065 %
  17       6.8067 ±    0.2539       0.00036 ±    0.00023       0.00003 ±    0.00000     0.169 ±  0.004 %    99.839 ±  0.061 %
  18       6.8643 ±    0.2518       0.00033 ±    0.00022       0.00003 ±    0.00000     0.168 ±  0.004 %    99.804 ±  0.065 %
  19       6.9991 ±    0.2515       0.00027 ±    0.00021       0.00003 ±    0.00000     0.166 ±  0.004 %    99.814 ±  0.062 %
  20       6.7977 ±    0.2383       0.00026 ±    0.00020       0.00003 ±    0.00000     0.168 ±  0.004 %    99.824 ±  0.059 %

./build/bin/./llama-perplexity --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.F16.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9425 ±    0.5261       0.00002 ±    0.00006       0.00000 ±    0.00000     0.022 ±  0.002 %    100.000 ±  0.000 %
   2       5.4427 ±    0.6040       0.00148 ±    0.00151       0.00000 ±    0.00000     0.023 ±  0.002 %    100.000 ±  0.000 %
   3       4.6851 ±    0.4029       0.00100 ±    0.00101       0.00000 ±    0.00000     0.029 ±  0.003 %    100.000 ±  0.000 %
   4       5.0068 ±    0.3674       0.00073 ±    0.00075       0.00000 ±    0.00000     0.029 ±  0.002 %    100.000 ±  0.000 %
   5       5.2945 ±    0.3436       0.00057 ±    0.00060       0.00000 ±    0.00000     0.028 ±  0.002 %    100.000 ±  0.000 %
   6       5.8317 ±    0.3545       0.00048 ±    0.00050       0.00000 ±    0.00000     0.027 ±  0.002 %    100.000 ±  0.000 %
   7       6.2264 ±    0.3545       0.00061 ±    0.00047       0.00000 ±    0.00000     0.027 ±  0.001 %    100.000 ±  0.000 %
   8       6.4321 ±    0.3454       0.00054 ±    0.00041       0.00000 ±    0.00000     0.027 ±  0.001 %    100.000 ±  0.000 %
   9       6.8873 ±    0.3580       0.00049 ±    0.00036       0.00000 ±    0.00000     0.027 ±  0.001 %    100.000 ±  0.000 %
  10       7.2376 ±    0.3591       0.00045 ±    0.00033       0.00000 ±    0.00000     0.027 ±  0.001 %    99.961 ±  0.039 %
  11       7.2589 ±    0.3421       0.00041 ±    0.00030       0.00000 ±    0.00000     0.026 ±  0.001 %    99.964 ±  0.036 %
  12       7.2846 ±    0.3299       0.00040 ±    0.00027       0.00000 ±    0.00000     0.027 ±  0.001 %    99.967 ±  0.033 %
  13       7.4399 ±    0.3229       0.00038 ±    0.00025       0.00000 ±    0.00000     0.026 ±  0.001 %    99.970 ±  0.030 %
  14       7.3381 ±    0.3062       0.00035 ±    0.00023       0.00000 ±    0.00000     0.026 ±  0.001 %    99.972 ±  0.028 %
  15       7.1273 ±    0.2860       0.00033 ±    0.00022       0.00000 ±    0.00000     0.026 ±  0.001 %    99.974 ±  0.026 %
  16       7.1709 ±    0.2793       0.00031 ±    0.00021       0.00000 ±    0.00000     0.026 ±  0.001 %    99.975 ±  0.025 %
  17       6.8063 ±    0.2539       0.00030 ±    0.00019       0.00000 ±    0.00000     0.027 ±  0.001 %    99.977 ±  0.023 %
  18       6.8639 ±    0.2518       0.00028 ±    0.00018       0.00000 ±    0.00000     0.027 ±  0.001 %    99.956 ±  0.031 %
  19       6.9991 ±    0.2515       0.00027 ±    0.00017       0.00000 ±    0.00000     0.027 ±  0.001 %    99.959 ±  0.029 %
  20       6.7978 ±    0.2384       0.00026 ±    0.00016       0.00000 ±    0.00000     0.027 ±  0.001 %    99.961 ±  0.028 %

./build/bin/./llama-perplexity --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.Q8_0.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9483 ±    0.5259       0.00149 ±    0.00252       0.00062 ±    0.00005     0.837 ±  0.073 %    99.216 ±  0.554 %
   2       5.4523 ±    0.6051       0.00324 ±    0.00270       0.00075 ±    0.00006     0.820 ±  0.050 %    99.020 ±  0.437 %
   3       4.6906 ±    0.4032       0.00217 ±    0.00211       0.00088 ±    0.00005     0.945 ±  0.056 %    98.824 ±  0.390 %
   4       5.0117 ±    0.3678       0.00170 ±    0.00173       0.00085 ±    0.00004     0.935 ±  0.047 %    98.824 ±  0.338 %
   5       5.3001 ±    0.3441       0.00161 ±    0.00147       0.00085 ±    0.00004     0.928 ±  0.040 %    98.745 ±  0.312 %
   6       5.8416 ±    0.3554       0.00217 ±    0.00130       0.00085 ±    0.00003     0.902 ±  0.036 %    98.889 ±  0.268 %
   7       6.2372 ±    0.3555       0.00234 ±    0.00124       0.00086 ±    0.00003     0.932 ±  0.040 %    98.824 ±  0.255 %
   8       6.4396 ±    0.3462       0.00171 ±    0.00114       0.00087 ±    0.00003     0.923 ±  0.036 %    98.578 ±  0.262 %
   9       6.8928 ±    0.3586       0.00129 ±    0.00106       0.00089 ±    0.00003     0.927 ±  0.036 %    98.562 ±  0.249 %
  10       7.2420 ±    0.3596       0.00106 ±    0.00099       0.00088 ±    0.00002     0.905 ±  0.033 %    98.549 ±  0.237 %
  11       7.2616 ±    0.3424       0.00079 ±    0.00092       0.00085 ±    0.00002     0.887 ±  0.031 %    98.610 ±  0.221 %
  12       7.2866 ±    0.3302       0.00068 ±    0.00087       0.00086 ±    0.00002     0.888 ±  0.029 %    98.529 ±  0.218 %
  13       7.4416 ±    0.3232       0.00061 ±    0.00084       0.00087 ±    0.00002     0.878 ±  0.027 %    98.431 ±  0.216 %
  14       7.3400 ±    0.3065       0.00061 ±    0.00080       0.00087 ±    0.00002     0.869 ±  0.026 %    98.403 ±  0.210 %
  15       7.1295 ±    0.2862       0.00063 ±    0.00076       0.00086 ±    0.00002     0.863 ±  0.025 %    98.431 ±  0.201 %
  16       7.1739 ±    0.2795       0.00074 ±    0.00074       0.00088 ±    0.00002     0.879 ±  0.024 %    98.407 ±  0.196 %
  17       6.8092 ±    0.2541       0.00074 ±    0.00071       0.00086 ±    0.00002     0.873 ±  0.024 %    98.431 ±  0.189 %
  18       6.8646 ±    0.2519       0.00038 ±    0.00069       0.00085 ±    0.00002     0.869 ±  0.023 %    98.453 ±  0.182 %
  19       6.9997 ±    0.2516       0.00035 ±    0.00067       0.00085 ±    0.00002     0.865 ±  0.022 %    98.431 ±  0.179 %
  20       6.7990 ±    0.2385       0.00044 ±    0.00065       0.00085 ±    0.00002     0.877 ±  0.021 %    98.451 ±  0.173 %

#> zen4:
./build/bin/./llama-perplexity -ctk bf16 -ctv bf16 --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.BF16.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9443 ±    0.5267       0.00048 ±    0.00053       0.00004 ±    0.00001     0.169 ±  0.019 %    99.608 ±  0.392 %
   2       5.4419 ±    0.6039       0.00133 ±    0.00153       0.00005 ±    0.00001     0.167 ±  0.012 %    99.412 ±  0.339 %
   3       4.6835 ±    0.4027       0.00066 ±    0.00105       0.00006 ±    0.00001     0.236 ±  0.021 %    99.608 ±  0.226 %
   4       5.0057 ±    0.3672       0.00051 ±    0.00080       0.00005 ±    0.00000     0.231 ±  0.017 %    99.608 ±  0.196 %
   5       5.2931 ±    0.3434       0.00030 ±    0.00065       0.00005 ±    0.00000     0.220 ±  0.014 %    99.686 ±  0.157 %
   6       5.8307 ±    0.3543       0.00030 ±    0.00055       0.00005 ±    0.00000     0.216 ±  0.012 %    99.739 ±  0.131 %
   7       6.2255 ±    0.3544       0.00047 ±    0.00052       0.00005 ±    0.00000     0.210 ±  0.011 %    99.664 ±  0.137 %
   8       6.4316 ±    0.3454       0.00047 ±    0.00046       0.00005 ±    0.00000     0.218 ±  0.010 %    99.657 ±  0.130 %
   9       6.8874 ±    0.3580       0.00050 ±    0.00041       0.00005 ±    0.00000     0.213 ±  0.010 %    99.608 ±  0.130 %
  10       7.2365 ±    0.3589       0.00030 ±    0.00038       0.00005 ±    0.00000     0.209 ±  0.009 %    99.569 ±  0.130 %
  11       7.2584 ±    0.3420       0.00034 ±    0.00035       0.00005 ±    0.00000     0.205 ±  0.008 %    99.572 ±  0.123 %
  12       7.2835 ±    0.3298       0.00025 ±    0.00032       0.00005 ±    0.00000     0.211 ±  0.009 %    99.542 ±  0.122 %
  13       7.4389 ±    0.3228       0.00025 ±    0.00030       0.00005 ±    0.00000     0.209 ±  0.009 %    99.578 ±  0.113 %
  14       7.3370 ±    0.3061       0.00021 ±    0.00029       0.00005 ±    0.00000     0.208 ±  0.008 %    99.524 ±  0.115 %
  15       7.1270 ±    0.2859       0.00028 ±    0.00027       0.00005 ±    0.00000     0.209 ±  0.008 %    99.529 ±  0.111 %
  16       7.1706 ±    0.2792       0.00027 ±    0.00026       0.00005 ±    0.00000     0.216 ±  0.007 %    99.510 ±  0.109 %
  17       6.8060 ±    0.2538       0.00026 ±    0.00025       0.00006 ±    0.00000     0.215 ±  0.007 %    99.539 ±  0.103 %
  18       6.8641 ±    0.2517       0.00030 ±    0.00024       0.00006 ±    0.00000     0.213 ±  0.007 %    99.521 ±  0.102 %
  19       6.9992 ±    0.2515       0.00028 ±    0.00023       0.00006 ±    0.00000     0.214 ±  0.007 %    99.546 ±  0.097 %
  20       6.7980 ±    0.2384       0.00030 ±    0.00022       0.00006 ±    0.00000     0.217 ±  0.007 %    99.569 ±  0.092 %

./build/bin/./llama-perplexity --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.BF16.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9433 ±    0.5262       0.00024 ±    0.00045       0.00003 ±    0.00000     0.166 ±  0.028 %    99.216 ±  0.554 %
   2       5.4430 ±    0.6041       0.00154 ±    0.00158       0.00003 ±    0.00000     0.167 ±  0.016 %    99.216 ±  0.391 %
   3       4.6865 ±    0.4030       0.00130 ±    0.00108       0.00004 ±    0.00001     0.205 ±  0.024 %    99.477 ±  0.261 %
   4       5.0071 ±    0.3673       0.00079 ±    0.00082       0.00003 ±    0.00000     0.193 ±  0.019 %    99.510 ±  0.219 %
   5       5.2951 ±    0.3436       0.00068 ±    0.00066       0.00003 ±    0.00000     0.185 ±  0.016 %    99.608 ±  0.175 %
   6       5.8319 ±    0.3544       0.00051 ±    0.00056       0.00003 ±    0.00000     0.178 ±  0.014 %    99.608 ±  0.160 %
   7       6.2266 ±    0.3545       0.00064 ±    0.00051       0.00003 ±    0.00000     0.174 ±  0.012 %    99.608 ±  0.148 %
   8       6.4322 ±    0.3455       0.00056 ±    0.00045       0.00003 ±    0.00000     0.177 ±  0.011 %    99.657 ±  0.130 %
   9       6.8885 ±    0.3581       0.00066 ±    0.00041       0.00003 ±    0.00000     0.175 ±  0.010 %    99.608 ±  0.130 %
  10       7.2386 ±    0.3592       0.00059 ±    0.00037       0.00003 ±    0.00000     0.172 ±  0.009 %    99.647 ±  0.117 %
  11       7.2603 ±    0.3422       0.00060 ±    0.00034       0.00003 ±    0.00000     0.170 ±  0.009 %    99.679 ±  0.107 %
  12       7.2852 ±    0.3299       0.00049 ±    0.00031       0.00003 ±    0.00000     0.173 ±  0.008 %    99.673 ±  0.103 %
  13       7.4408 ±    0.3230       0.00049 ±    0.00029       0.00003 ±    0.00000     0.172 ±  0.008 %    99.698 ±  0.095 %
  14       7.3386 ±    0.3063       0.00041 ±    0.00028       0.00003 ±    0.00000     0.170 ±  0.007 %    99.692 ±  0.093 %
  15       7.1278 ±    0.2860       0.00040 ±    0.00026       0.00003 ±    0.00000     0.167 ±  0.007 %    99.686 ±  0.090 %
  16       7.1714 ±    0.2793       0.00038 ±    0.00024       0.00003 ±    0.00000     0.166 ±  0.006 %    99.706 ±  0.085 %
  17       6.8064 ±    0.2539       0.00032 ±    0.00023       0.00003 ±    0.00000     0.164 ±  0.006 %    99.723 ±  0.080 %
  18       6.8641 ±    0.2518       0.00031 ±    0.00022       0.00003 ±    0.00000     0.162 ±  0.006 %    99.695 ±  0.081 %
  19       6.9994 ±    0.2515       0.00032 ±    0.00021       0.00003 ±    0.00000     0.161 ±  0.006 %    99.711 ±  0.077 %
  20       6.7979 ±    0.2384       0.00028 ±    0.00020       0.00003 ±    0.00000     0.165 ±  0.005 %    99.706 ±  0.076 %

./build/bin/./llama-perplexity --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.F16.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9432 ±    0.5262       0.00021 ±    0.00007       0.00000 ±    0.00000     0.023 ±  0.003 %    100.000 ±  0.000 %
   2       5.4435 ±    0.6041       0.00163 ±    0.00150       0.00000 ±    0.00000     0.025 ±  0.002 %    100.000 ±  0.000 %
   3       4.6856 ±    0.4029       0.00111 ±    0.00100       0.00000 ±    0.00000     0.030 ±  0.002 %    100.000 ±  0.000 %
   4       5.0072 ±    0.3674       0.00081 ±    0.00075       0.00000 ±    0.00000     0.029 ±  0.002 %    100.000 ±  0.000 %
   5       5.2951 ±    0.3437       0.00067 ±    0.00060       0.00000 ±    0.00000     0.030 ±  0.002 %    100.000 ±  0.000 %
   6       5.8323 ±    0.3545       0.00057 ±    0.00050       0.00000 ±    0.00000     0.029 ±  0.002 %    100.000 ±  0.000 %
   7       6.2269 ±    0.3546       0.00069 ±    0.00047       0.00000 ±    0.00000     0.028 ±  0.001 %    100.000 ±  0.000 %
   8       6.4324 ±    0.3455       0.00059 ±    0.00041       0.00000 ±    0.00000     0.028 ±  0.001 %    100.000 ±  0.000 %
   9       6.8876 ±    0.3581       0.00053 ±    0.00036       0.00000 ±    0.00000     0.027 ±  0.001 %    100.000 ±  0.000 %
  10       7.2379 ±    0.3591       0.00049 ±    0.00033       0.00000 ±    0.00000     0.027 ±  0.001 %    99.961 ±  0.039 %
  11       7.2590 ±    0.3421       0.00043 ±    0.00030       0.00000 ±    0.00000     0.027 ±  0.001 %    99.964 ±  0.036 %
  12       7.2847 ±    0.3299       0.00041 ±    0.00027       0.00000 ±    0.00000     0.028 ±  0.001 %    99.935 ±  0.046 %
  13       7.4400 ±    0.3229       0.00039 ±    0.00025       0.00000 ±    0.00000     0.028 ±  0.001 %    99.940 ±  0.043 %
  14       7.3381 ±    0.3062       0.00035 ±    0.00023       0.00000 ±    0.00000     0.028 ±  0.001 %    99.888 ±  0.056 %
  15       7.1273 ±    0.2860       0.00032 ±    0.00022       0.00000 ±    0.00000     0.028 ±  0.001 %    99.895 ±  0.052 %
  16       7.1708 ±    0.2793       0.00030 ±    0.00021       0.00000 ±    0.00000     0.030 ±  0.001 %    99.902 ±  0.049 %
  17       6.8062 ±    0.2539       0.00029 ±    0.00019       0.00000 ±    0.00000     0.030 ±  0.001 %    99.908 ±  0.046 %
  18       6.8639 ±    0.2518       0.00027 ±    0.00018       0.00000 ±    0.00000     0.030 ±  0.001 %    99.891 ±  0.049 %
  19       6.9991 ±    0.2515       0.00027 ±    0.00017       0.00000 ±    0.00000     0.030 ±  0.001 %    99.897 ±  0.046 %
  20       6.7978 ±    0.2384       0.00027 ±    0.00016       0.00000 ±    0.00000     0.030 ±  0.001 %    99.882 ±  0.048 %

./build/bin/./llama-perplexity --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.Q8_0.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9525 ±    0.5268       0.00257 ±    0.00200       0.00062 ±    0.00006     0.780 ±  0.079 %    98.824 ±  0.677 %
   2       5.4551 ±    0.6050       0.00375 ±    0.00244       0.00074 ±    0.00005     0.790 ±  0.052 %    99.020 ±  0.437 %
   3       4.6895 ±    0.4028       0.00195 ±    0.00192       0.00094 ±    0.00007     0.974 ±  0.066 %    98.824 ±  0.390 %
   4       5.0104 ±    0.3675       0.00144 ±    0.00155       0.00089 ±    0.00005     0.958 ±  0.054 %    98.922 ±  0.324 %
   5       5.2958 ±    0.3435       0.00082 ±    0.00133       0.00089 ±    0.00004     0.924 ±  0.046 %    98.902 ±  0.292 %
   6       5.8327 ±    0.3544       0.00064 ±    0.00118       0.00087 ±    0.00004     0.897 ±  0.040 %    98.824 ±  0.276 %
   7       6.2283 ±    0.3547       0.00092 ±    0.00114       0.00088 ±    0.00003     0.888 ±  0.036 %    98.768 ±  0.261 %
   8       6.4327 ±    0.3453       0.00063 ±    0.00107       0.00088 ±    0.00003     0.877 ±  0.033 %    98.529 ±  0.267 %
   9       6.8834 ±    0.3577      -0.00008 ±    0.00100       0.00088 ±    0.00003     0.869 ±  0.030 %    98.562 ±  0.249 %
  10       7.2338 ±    0.3588      -0.00007 ±    0.00094       0.00087 ±    0.00002     0.859 ±  0.028 %    98.431 ±  0.246 %
  11       7.2576 ±    0.3420       0.00023 ±    0.00088       0.00086 ±    0.00002     0.843 ±  0.026 %    98.503 ±  0.229 %
  12       7.2853 ±    0.3300       0.00051 ±    0.00084       0.00088 ±    0.00002     0.851 ±  0.025 %    98.529 ±  0.218 %
  13       7.4384 ±    0.3229       0.00018 ±    0.00080       0.00087 ±    0.00002     0.841 ±  0.024 %    98.522 ±  0.210 %
  14       7.3364 ±    0.3062       0.00011 ±    0.00076       0.00087 ±    0.00002     0.838 ±  0.023 %    98.543 ±  0.201 %
  15       7.1259 ±    0.2859       0.00013 ±    0.00073       0.00086 ±    0.00002     0.833 ±  0.022 %    98.614 ±  0.189 %
  16       7.1687 ±    0.2792       0.00001 ±    0.00070       0.00086 ±    0.00002     0.841 ±  0.021 %    98.603 ±  0.184 %
  17       6.8051 ±    0.2538       0.00013 ±    0.00067       0.00085 ±    0.00002     0.838 ±  0.020 %    98.570 ±  0.180 %
  18       6.8615 ±    0.2516      -0.00007 ±    0.00066       0.00084 ±    0.00002     0.839 ±  0.020 %    98.540 ±  0.177 %
  19       6.9961 ±    0.2514      -0.00016 ±    0.00064       0.00084 ±    0.00002     0.838 ±  0.019 %    98.514 ±  0.174 %
  20       6.7965 ±    0.2384       0.00008 ±    0.00062       0.00085 ±    0.00002     0.855 ±  0.020 %    98.529 ±  0.169 %

Djip007 · 2024-12-10T00:52:07Z

For me look good.
@ggerganov @slaren what do you think with the 3 failed?

slaren · 2024-12-10T21:20:03Z

@ggerganov @slaren what do you think with the 3 failed?

Not sure, try merging the current master to see if it is some issue in the server that has already been fixed.

Djip007 · 2024-12-11T03:14:52Z

--------------------------- Captured stdout teardown ---------------------------
Stopping server with pid=4213
=========================== short test summary info ============================
FAILED unit/test_completion.py::test_consistent_result_same_seed[2] - AssertionError: assert ' making. Eve...hen, they saw' == ' making. Eve...ining and dan'
  
     making. Everyone are very hungry.
  - One day, it is time to go to the park with his mom. They had a quiet window. They were shining and dan
  + One day, it is time to go to the park with his mom. They had a talking eye to rest. But then, they saw
!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!
======================== 1 failed, 32 passed in 22.85s =========================

look like a small diff in result.

unit/test_completion.py::test_consistent_result_same_seed[1] PASSED      [ 34%]
unit/test_completion.py::test_consistent_result_same_seed[2] FAILED      [ 35%]

@pytest.mark.parametrize("n_slots", [1, 2])
def test_consistent_result_same_seed(n_slots: int):
    global server
    server.n_slots = n_slots

what is n_slots?

I have to check some elements in my code tomorrow...

slaren · 2024-12-11T14:01:23Z

what is n_slots?

I am not sure what's the effect of increasing the number of slots for this test. I suspect that this error might indicate there is a buffer overflow somewhere, and random data beyond the tensor buffer may be causing it to generate different sequences despite using the same seed.

Djip007 · 2024-12-11T18:58:13Z

I suspect that this error might indicate there is a buffer overflow somewhere, and random data beyond the tensor buffer may be causing it to generate different sequences despite using the same seed.

That's what I was thinking last night, but it was too late. I have a little idea, but I was too tired to check/correct it.

ggerganov · 2024-12-11T19:18:17Z

The failing test seems to be using 2 slots. With 2 slots, the KV cache buffer is shared among the two generations. Initially, the buffer is empty:

..................................................................

Then the first request is processed by slot 0 and thus the beginning of the buffer is occupied:

000000000000000000000000000000000000..............................

The second request is processed on slot 1, so the old data remains in the buffer:

0000000000000000000000000000000000001111111111111111111111111111..

Because we compute the attention on the entire buffer by masking out the cross-sequence values, it is actually possible to get different results between the 2 generations. This happens due to summing floating-point across the length of the KV buffer. In the next example, even-though the data in the buffer is the same, it can lead to different numerical results during the V*QK matrix multiplication simply because the data occupies different cells and the SIMD groups would produce different reults:

000000000000000000000000000000000000..1111111111111111111111111111

I'm thinking that maybe there isn't a bug in the implementation in this PR, and it's a side-effect of the unified KV cache. Probably this test for n_slots > 1 should be disabled for now.

Djip007 · 2024-12-11T20:01:57Z

@ggerganov
Great! Thanks for these explanations. Very clear. And then very nice to learn how it works.

On the other hand, by going over my code step by step, there are a small number of cases (2 to ~5?) where I do too much calculation and write outside the wrong value (possibly by overwriting correct data that I just calculated...)

So I corrected that. It remains to be seen if the test passes.

Djip007 · 2024-12-11T20:05:46Z

Well that wasn't enough,
at least I fixed a small bug.

I'm doing another pass on the perlexity to be sure with my last correction.

slaren

I have been running random tests with test-backend-ops and I haven't seen any failure, so I am fairly confident that this is correct. Let's just disable the server test for 2 slots.

Djip007 · 2024-12-11T20:17:09Z

I have been running random tests with test-backend-ops and I haven't seen any failure, so I am fairly confident that this is correct. Let's just disable the server test for 2 slots.

not sur how to do it:

# replace 
# @pytest.mark.parametrize("n_slots", [1, 2])
# with that?
@pytest.mark.parametrize("n_slots", [1])
def test_consistent_result_same_seed(n_slots: int):
    global server
    server.n_slots = n_slots
    server.start()
    last_res = None
    for _ in range(4):
        res = server.make_request("POST", "/completion", data={
            "prompt": "I believe the meaning of life is",
            "seed": 42,
            "temperature": 1.0,
            "cache_prompt": False,  # TODO: remove this once test_cache_vs_nocache_prompt is fixed
        })
        if last_res is not None:
            assert res.body["content"] == last_res.body["content"]
        last_res = res

ggerganov · 2024-12-11T20:26:49Z

A different test is failing now. Add:

--- a/examples/server/tests/unit/test_completion.py
+++ b/examples/server/tests/unit/test_completion.py
@@ -116,6 +116,7 @@ def test_different_result_different_seed(n_slots: int):
 def test_consistent_result_different_batch_size(n_batch: int, temperature: float):
     global server
     server.n_batch = n_batch
+    server.n_slots = 1
     server.start()
     last_res = None
     for _ in range(4):

slaren · 2024-12-11T20:27:05Z

On my system (intel 13900k) I see better performance with BF16, but worse with F16 in some cases:

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 7B BF16	pp32	33.94	42.27	1.25
llama 7B BF16	pp64	34.82	43.27	1.24
llama 7B BF16	pp128	33.22	43.69	1.32
llama 7B BF16	tg32	6.75	6.41	0.95
llama 7B F16	pp32	41.45	28.85	0.70
llama 7B F16	pp64	42.88	26.34	0.61
llama 7B F16	pp128	43.69	29.75	0.68
llama 7B F16	tg32	6.82	6.47	0.95

With different numbers of threads:

Model	Threads	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 7B F16	8	pp64	51.98	59.14	1.14
llama 7B F16	16	pp64	35.20	28.97	0.82
llama 7B F16	24	pp64	63.18	43.40	0.69
llama 7B F16	32	pp64	75.45	54.18	0.72

Djip007 · 2024-12-11T20:33:18Z

On my system (intel 13900k)

It is a AVX512 or a AVX2?

slaren · 2024-12-11T20:34:22Z

AVX2

Djip007 · 2024-12-11T20:40:48Z

This is the "bad" intel CPU: that disable avx512 on effichent core

Nombre de Performance-cores 8
Nombre d'Efficient-cores 16

look faster on perf core slower on other...

can you bench with that:

#elif (defined(__AVX__) || defined(__AVX2__)) && defined(__F16C__)
// do not convert B to FP16
        if (Btype == GGML_TYPE_F32) {
            tinyBLAS<8, __m256, __m256, ggml_fp16_t, float, float> tb{ k,
                (const ggml_fp16_t *)A, lda,
                (const float *)B, ldb,
                (float *)C, ldc,
                ith, nth};
            return tb.matmul(m, n);
        }

and may be with the BF16 too...

slaren · 2024-12-11T20:42:39Z

Not sure where to change that, can you show me the diff of this change (change it locally and run git diff)?

Djip007 · 2024-12-11T20:45:23Z

ggml/src/ggml-cpu/llamafile/sgemm.cpp

-        tb.matmul(m, n);
-        return true;
+        if (Btype == GGML_TYPE_F16) {
+            tinyBLAS<8, __m256, __m256, ggml_fp16_t, ggml_fp16_t, float> tb{ k,


#elif (defined(__AVX__) || defined(__AVX2__)) && defined(__F16C__) // do not convert B to FP16 what was did before... if (Btype == GGML_TYPE_F32) { tinyBLAS<8, __m256, __m256, ggml_fp16_t, float, float> tb{ k, (const ggml_fp16_t *)A, lda, (const float *)B, ldb, (float *)C, ldc, ith, nth}; return tb.matmul(m, n); }

I' will try the same on my zen3 ...

This is definitely faster, but still slower than master in some cases.

Model Threads Test t/s master t/s perfo/tinyblas Speedup

llama 7B F16 8 pp64 52.68 61.85 1.17

llama 7B F16 16 pp64 42.89 37.64 0.88

llama 7B F16 24 pp64 63.02 56.05 0.89

llama 7B F16 32 pp64 77.93 68.80 0.88

With these heterogeneous CPUs, the dispatch would need to be refined. For the moment, the calculations are distributed evenly between the Cores, so there is a "chance" that we will wait for the eco cores....

My guess is that the tinyblas implementation doesn't play very well with the e-cores or multi-threading. The ggml implementation has dynamic chunking so that the faster threads will get more work, but I don't think that this implemented in tinyblas.

My RAM is 3600 vs 2133 it may explain the diff in TG...
CPU is the same.

I have 4x32Go config as interleaved
did you have 4x16Go ? with what config?

g++ --version g++ (GCC) 14.2.1 20240912 (Red Hat 14.2.1-3)

OS: fedora 41 vs Ubuntu 11.4

Mistral-7b : look we have the same behavior with it so it's not due to the model.

Model Test t/s master t/s PR Speedup PC

llama 7B F16 pp120 55.95 84.83 1.52 djip007

llama 7B F16 pp120 70.64 44.11 0.62 ggerganov

For now I don't understand what's happening...

something with Memory config ?

Memory Device Array Handle: 0x0032 Error Information Handle: 0x003F Total Width: 64 bits Data Width: 64 bits Size: 32 GB Form Factor: DIMM Set: None Locator: DIMM_B1 Bank Locator: BANK 2 Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 3600 MT/s Manufacturer: CRUCIAL Serial Number: ----------- Asset Tag: Not Specified Part Number: BL32G36C16U4B.M16FB1 Rank: 2 Configured Memory Speed: 3600 MT/s Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Memory Technology: DRAM Memory Operating Mode Capability: Volatile memory Firmware Version: Unknown Module Manufacturer ID: Bank 6, Hex 0x9B Module Product ID: Unknown Memory Subsystem Controller Manufacturer ID: Unknown Memory Subsystem Controller Product ID: Unknown Non-Volatile Size: None Volatile Size: 32 GB Cache Size: None Logical Size: None

$ lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD Model name: AMD Ryzen 9 5950X 16-Core Processor CPU family: 25 Model: 33 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 0 Frequency boost: enabled CPU(s) scaling MHz: 65% CPU max MHz: 5084,0000 CPU min MHz: 550,0000 BogoMIPS: 6800,74 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdp e1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma c x16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy extapic cr8_legacy abm sse4a misalignsse 3dnowpr efetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs i bpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv 1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_loc k nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku o spke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap Caches (sum of all): L1d: 512 KiB (16 instances) L1i: 512 KiB (16 instances) L2: 8 MiB (16 instances) L3: 64 MiB (2 instances) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-31 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Reg file data sampling: Not affected Retbleed: Not affected Spec rstack overflow: Vulnerable: Safe RET, no microcode Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Srbds: Not affected Tsx async abort: Not affected

memory addressing:

Handle 0x003B, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x01FFFFFFFFF Range Size: 128 GB Physical Device Handle: 0x003A Memory Array Mapped Address Handle: 0x0034 Partition Row Position: Unknown Interleave Position: Unknown Interleaved Data Depth: Unknown Handle 0x003E, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x01FFFFFFFFF Range Size: 128 GB Physical Device Handle: 0x003D Memory Array Mapped Address Handle: 0x0034 Partition Row Position: Unknown Interleave Position: Unknown Interleaved Data Depth: Unknown Handle 0x0041, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x01FFFFFFFFF Range Size: 128 GB Physical Device Handle: 0x0040 Memory Array Mapped Address Handle: 0x0034 Partition Row Position: Unknown Interleave Position: Unknown Interleaved Data Depth: Unknown Handle 0x0044, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x01FFFFFFFFF Range Size: 128 GB Physical Device Handle: 0x0043 Memory Array Mapped Address Handle: 0x0034 Partition Row Position: Unknown Interleave Position: Unknown Interleaved Data Depth: Unknown

You have Corsair Vengeance RGB RS 16 Go DDR4 3200 MHz CL16 RAM and clock it a 2133 MHz ?

My RAM is 3600 vs 2133 it may explain the diff in TG...

Yes, probably that's it.

I have 4x32Go config as interleaved
did you have 4x16Go ? with what config?

Yes, these are 4x16GB. I don't know if they are interleaved and how to check. Here is the full dmidecode if it helps:

$ sudo dmidecode -t memory # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Handle 0x000A, DMI type 16, 23 bytes Physical Memory Array Location: System Board Or Motherboard Use: System Memory Error Correction Type: None Maximum Capacity: 128 GB Error Information Handle: 0x0009 Number Of Devices: 4 Handle 0x0012, DMI type 17, 92 bytes Memory Device Array Handle: 0x000A Error Information Handle: 0x0011 Total Width: 64 bits Data Width: 64 bits Size: 16 GB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: P0 CHANNEL A Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2133 MT/s Manufacturer: Unknown Serial Number: 00000000 Asset Tag: Not Specified Part Number: CMG16GX4M1E3200C16 Rank: 1 Configured Memory Speed: 2133 MT/s Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Memory Technology: DRAM Memory Operating Mode Capability: Volatile memory Firmware Version: Unknown Module Manufacturer ID: Bank 3, Hex 0x9E Module Product ID: Unknown Memory Subsystem Controller Manufacturer ID: Unknown Memory Subsystem Controller Product ID: Unknown Non-Volatile Size: None Volatile Size: 16 GB Cache Size: None Logical Size: None Handle 0x0015, DMI type 17, 92 bytes Memory Device Array Handle: 0x000A Error Information Handle: 0x0014 Total Width: 64 bits Data Width: 64 bits Size: 16 GB Form Factor: DIMM Set: None Locator: DIMM 1 Bank Locator: P0 CHANNEL A Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2133 MT/s Manufacturer: Unknown Serial Number: 00000000 Asset Tag: Not Specified Part Number: CMG16GX4M1E3200C16 Rank: 1 Configured Memory Speed: 2133 MT/s Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Memory Technology: DRAM Memory Operating Mode Capability: Volatile memory Firmware Version: Unknown Module Manufacturer ID: Bank 3, Hex 0x9E Module Product ID: Unknown Memory Subsystem Controller Manufacturer ID: Unknown Memory Subsystem Controller Product ID: Unknown Non-Volatile Size: None Volatile Size: 16 GB Cache Size: None Logical Size: None Handle 0x0018, DMI type 17, 92 bytes Memory Device Array Handle: 0x000A Error Information Handle: 0x0017 Total Width: 64 bits Data Width: 64 bits Size: 16 GB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: P0 CHANNEL B Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2133 MT/s Manufacturer: Unknown Serial Number: 00000000 Asset Tag: Not Specified Part Number: CMG16GX4M1E3200C16 Rank: 1 Configured Memory Speed: 2133 MT/s Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Memory Technology: DRAM Memory Operating Mode Capability: Volatile memory Firmware Version: Unknown Module Manufacturer ID: Bank 3, Hex 0x9E Module Product ID: Unknown Memory Subsystem Controller Manufacturer ID: Unknown Memory Subsystem Controller Product ID: Unknown Non-Volatile Size: None Volatile Size: 16 GB Cache Size: None Logical Size: None Handle 0x001B, DMI type 17, 92 bytes Memory Device Array Handle: 0x000A Error Information Handle: 0x001A Total Width: 64 bits Data Width: 64 bits Size: 16 GB Form Factor: DIMM Set: None Locator: DIMM 1 Bank Locator: P0 CHANNEL B Type: DDR4 Type Detail: Synchronous Unbuffered (Unregistered) Speed: 2133 MT/s Manufacturer: Unknown Serial Number: 00000000 Asset Tag: Not Specified Part Number: CMG16GX4M1E3200C16 Rank: 1 Configured Memory Speed: 2133 MT/s Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V Memory Technology: DRAM Memory Operating Mode Capability: Volatile memory Firmware Version: Unknown Module Manufacturer ID: Bank 3, Hex 0x9E Module Product ID: Unknown Memory Subsystem Controller Manufacturer ID: Unknown Memory Subsystem Controller Product ID: Unknown Non-Volatile Size: None Volatile Size: 16 GB Cache Size: None Logical Size: None $ sudo dmidecode -t 20 # dmidecode 3.3 Getting SMBIOS data from sysfs. SMBIOS 3.3.0 present. Handle 0x0013, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x00FFFFFFFFF Range Size: 64 GB Physical Device Handle: 0x0012 Memory Array Mapped Address Handle: 0x000C Partition Row Position: Unknown Interleave Position: Unknown Interleaved Data Depth: Unknown Handle 0x0016, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x00FFFFFFFFF Range Size: 64 GB Physical Device Handle: 0x0015 Memory Array Mapped Address Handle: 0x000C Partition Row Position: Unknown Interleave Position: Unknown Interleaved Data Depth: Unknown Handle 0x0019, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x00FFFFFFFFF Range Size: 64 GB Physical Device Handle: 0x0018 Memory Array Mapped Address Handle: 0x000C Partition Row Position: Unknown Interleave Position: Unknown Interleaved Data Depth: Unknown Handle 0x001C, DMI type 20, 35 bytes Memory Device Mapped Address Starting Address: 0x00000000000 Ending Address: 0x00FFFFFFFFF Range Size: 64 GB Physical Device Handle: 0x001B Memory Array Mapped Address Handle: 0x000C Partition Row Position: Unknown Interleave Position: Unknown Interleaved Data Depth: Unknown

lscpu:

Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 48 bits physical, 48 bits virtual Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Vendor ID: AuthenticAMD Model name: AMD Ryzen 9 5950X 16-Core Processor CPU family: 25 Model: 33 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 1 Stepping: 2 Frequency boost: enabled CPU max MHz: 5083,3979 CPU min MHz: 2200,0000 BogoMIPS: 6787.87 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy extapic cr 8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm _total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap Caches (sum of all): L1d: 512 KiB (16 instances) L1i: 512 KiB (16 instances) L2: 8 MiB (16 instances) L3: 64 MiB (2 instances) NUMA: NUMA node(s): 1 NUMA node0 CPU(s): 0-31 Vulnerabilities: Gather data sampling: Not affected Itlb multihit: Not affected L1tf: Not affected Mds: Not affected Meltdown: Not affected Mmio stale data: Not affected Reg file data sampling: Not affected Retbleed: Not affected Spec rstack overflow: Vulnerable: Safe RET, no microcode Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected Srbds: Not affected Tsx async abort: Not affected

You have Corsair Vengeance RGB RS 16 Go DDR4 3200 MHz CL16 RAM and clock it a 2133 MHz ?

Hm, it's possible. It's an old machine that I currently keep remotely and I remember that at some point in the past I was adjusting some settings in the BIOS with the main goal to reduce the CPU fan noises. Though it's completely possible that I have underclocked the memory by accident.

Anyway, I am OK to ignore this datapoint for now since it is very likely a misconfiguration on my side. Next time I have access to the machine, will check what the BIOS settings are. But no need to worry about this for now.

Here are the tests you asked for.

Model Threads Test t/s master t/s perfo/tinyblas Speedup

llama 7B BF16 8 pp1 6.54 6.85 1.05

llama 7B BF16 8 pp2 11.15 14.50 1.30

llama 7B BF16 8 pp3 14.71 21.46 1.46

llama 7B BF16 8 pp4 17.29 25.63 1.48

llama 7B BF16 8 pp5 19.17 32.46 1.69

llama 7B BF16 8 pp6 20.78 37.58 1.81

llama 7B BF16 8 pp7 22.28 40.30 1.81

llama 7B BF16 8 pp8 23.07 44.56 1.93

llama 7B BF16 8 pp9 22.69 47.97 2.11

llama 7B BF16 8 pp10 22.85 50.19 2.20

llama 7B BF16 8 pp11 23.28 52.97 2.28

llama 7B BF16 8 pp12 23.64 55.29 2.34

llama 7B BF16 8 pp13 23.93 55.26 2.31

llama 7B BF16 8 pp14 17.59 55.09 3.13

llama 7B BF16 8 pp15 23.55 57.91 2.46

llama 7B BF16 8 pp16 24.62 57.64 2.34

llama 7B BF16 8 pp30 26.04 63.60 2.44

llama 7B BF16 8 pp31 25.55 63.00 2.47

llama 7B BF16 8 pp32 25.07 60.48 2.41

llama 7B BF16 8 pp64 25.38 63.79 2.51

llama 7B BF16 8 pp65 24.97 66.64 2.67

llama 7B BF16 8 pp66 25.25 68.75 2.72

llama 7B BF16 8 pp120 23.24 67.85 2.92

llama 7B BF16 8 pp128 25.64 66.69 2.60

llama 7B BF16 8 pp130 23.48 54.18 2.31

llama 7B BF16 8 pp240 24.38 68.85 2.82

llama 7B BF16 8 pp255 24.51 66.71 2.72

llama 7B BF16 8 pp256 23.19 66.83 2.88

llama 7B BF16 8 pp510 22.54 60.81 2.70

llama 7B BF16 8 pp512 23.94 60.61 2.53

llama 7B BF16 8 pp1023 23.68 59.02 2.49

llama 7B BF16 8 pp1024 23.78 59.15 2.49

llama 7B BF16 8 pp1025 23.73 58.89 2.48

llama 7B BF16 8 pp2048 23.29 58.20 2.50

llama 7B BF16 8 tg128 6.69 6.42 0.96

llama 7B BF16 16 pp1 6.89 6.90 1.00

llama 7B BF16 16 pp2 13.49 14.19 1.05

llama 7B BF16 16 pp3 18.20 20.99 1.15

llama 7B BF16 16 pp4 21.92 27.21 1.24

llama 7B BF16 16 pp5 24.63 34.20 1.39

llama 7B BF16 16 pp6 26.45 39.95 1.51

llama 7B BF16 16 pp7 26.87 45.12 1.68

llama 7B BF16 16 pp8 20.01 49.56 2.48

llama 7B BF16 16 pp9 28.62 54.41 1.90

llama 7B BF16 16 pp10 30.72 57.11 1.86

llama 7B BF16 16 pp11 31.19 62.05 1.99

llama 7B BF16 16 pp12 31.64 65.01 2.05

llama 7B BF16 16 pp13 33.39 66.48 1.99

llama 7B BF16 16 pp14 33.71 69.13 2.05

llama 7B BF16 16 pp15 34.12 71.64 2.10

llama 7B BF16 16 pp16 33.81 71.72 2.12

llama 7B BF16 16 pp30 34.43 81.93 2.38

llama 7B BF16 16 pp31 34.60 80.99 2.34

llama 7B BF16 16 pp32 34.67 81.60 2.35

llama 7B BF16 16 pp64 35.44 85.54 2.41

llama 7B BF16 16 pp65 34.26 85.67 2.50

llama 7B BF16 16 pp66 36.01 86.00 2.39

llama 7B BF16 16 pp120 36.28 86.63 2.39

llama 7B BF16 16 pp128 32.27 87.02 2.70

llama 7B BF16 16 pp130 35.74 86.62 2.42

llama 7B BF16 16 pp240 33.41 75.09 2.25

llama 7B BF16 16 pp255 33.79 86.10 2.55

llama 7B BF16 16 pp256 33.65 86.45 2.57

llama 7B BF16 16 pp510 32.94 82.24 2.50

llama 7B BF16 16 pp512 34.21 75.92 2.22

llama 7B BF16 16 pp1023 33.01 76.81 2.33

llama 7B BF16 16 pp1024 32.96 77.04 2.34

llama 7B BF16 16 pp1025 33.54 72.84 2.17

llama 7B BF16 16 pp2048 32.17 72.21 2.24

llama 7B BF16 16 tg128 6.64 6.49 0.98

llama 7B BF16 24 pp1 6.45 6.46 1.00

llama 7B BF16 24 pp2 13.87 14.12 1.02

llama 7B BF16 24 pp3 19.54 21.26 1.09

llama 7B BF16 24 pp4 23.84 27.43 1.15

llama 7B BF16 24 pp5 27.16 34.66 1.28

llama 7B BF16 24 pp6 29.39 40.81 1.39

llama 7B BF16 24 pp7 31.82 46.53 1.46

llama 7B BF16 24 pp8 32.90 52.14 1.58

llama 7B BF16 24 pp9 34.22 57.32 1.67

llama 7B BF16 24 pp10 34.64 61.83 1.78

llama 7B BF16 24 pp11 35.80 67.03 1.87

llama 7B BF16 24 pp12 35.61 70.34 1.98

llama 7B BF16 24 pp13 26.88 72.89 2.71

llama 7B BF16 24 pp14 36.83 76.42 2.08

llama 7B BF16 24 pp15 36.29 83.74 2.31

llama 7B BF16 24 pp16 37.80 84.55 2.24

llama 7B BF16 24 pp30 39.96 96.48 2.41

llama 7B BF16 24 pp31 40.38 97.68 2.42

llama 7B BF16 24 pp32 40.44 98.66 2.44

llama 7B BF16 24 pp64 42.22 98.08 2.32

llama 7B BF16 24 pp65 41.12 95.11 2.31

llama 7B BF16 24 pp66 33.80 95.15 2.82

llama 7B BF16 24 pp120 41.40 74.86 1.81

llama 7B BF16 24 pp128 41.64 97.67 2.35

llama 7B BF16 24 pp130 35.86 99.80 2.78

llama 7B BF16 24 pp240 41.78 96.56 2.31

llama 7B BF16 24 pp255 41.48 81.96 1.98

llama 7B BF16 24 pp256 38.66 98.73 2.55

llama 7B BF16 24 pp510 39.75 85.03 2.14

llama 7B BF16 24 pp512 39.44 94.90 2.41

llama 7B BF16 24 pp1023 39.18 88.19 2.25

llama 7B BF16 24 pp1024 38.70 87.89 2.27

llama 7B BF16 24 pp1025 38.99 85.98 2.20

llama 7B BF16 24 pp2048 37.26 83.63 2.24

llama 7B BF16 24 tg128 6.12 6.14 1.00

llama 7B BF16 32 pp1 6.61 6.49 0.98

llama 7B BF16 32 pp2 13.41 12.63 0.94

llama 7B BF16 32 pp3 16.67 19.67 1.18

llama 7B BF16 32 pp4 24.53 26.30 1.07

llama 7B BF16 32 pp5 25.11 30.75 1.22

llama 7B BF16 32 pp6 30.08 35.58 1.18

llama 7B BF16 32 pp7 32.79 44.60 1.36

llama 7B BF16 32 pp8 33.60 47.41 1.41

llama 7B BF16 32 pp9 35.68 49.91 1.40

llama 7B BF16 32 pp10 36.61 61.07 1.67

llama 7B BF16 32 pp11 34.87 60.74 1.74

llama 7B BF16 32 pp12 38.80 58.61 1.51

llama 7B BF16 32 pp13 38.92 67.56 1.74

llama 7B BF16 32 pp14 37.30 71.50 1.92

llama 7B BF16 32 pp15 38.23 52.93 1.38

llama 7B BF16 32 pp16 39.77 77.02 1.94

llama 7B BF16 32 pp30 42.88 93.16 2.17

llama 7B BF16 32 pp31 41.70 92.21 2.21

llama 7B BF16 32 pp32 42.97 96.90 2.26

llama 7B BF16 32 pp64 45.44 106.63 2.35

llama 7B BF16 32 pp65 44.98 105.98 2.36

llama 7B BF16 32 pp66 45.10 105.50 2.34

llama 7B BF16 32 pp120 46.50 104.57 2.25

llama 7B BF16 32 pp128 45.95 104.48 2.27

llama 7B BF16 32 pp130 46.34 101.40 2.19

llama 7B BF16 32 pp240 43.26 87.72 2.03

llama 7B BF16 32 pp255 46.97 109.11 2.32

llama 7B BF16 32 pp256 46.94 104.42 2.22

llama 7B BF16 32 pp510 44.52 92.37 2.07

llama 7B BF16 32 pp512 44.46 92.70 2.08

llama 7B BF16 32 pp1023 43.70 93.70 2.14

llama 7B BF16 32 pp1024 43.72 94.72 2.17

llama 7B BF16 32 pp1025 43.49 93.42 2.15

llama 7B BF16 32 pp2048 42.64 90.61 2.12

llama 7B BF16 32 tg128 5.84 5.89 1.01

Model Threads Test t/s master t/s perfo/tinyblas Speedup

llama 7B F16 8 pp1 6.38 7.03 1.10

llama 7B F16 8 pp2 12.49 14.03 1.12

llama 7B F16 8 pp3 19.06 20.65 1.08

llama 7B F16 8 pp4 12.97 25.21 1.94

llama 7B F16 8 pp5 18.14 31.46 1.73

llama 7B F16 8 pp6 28.90 36.29 1.26

llama 7B F16 8 pp7 20.51 39.36 1.92

llama 7B F16 8 pp8 27.07 43.70 1.61

llama 7B F16 8 pp9 39.64 46.82 1.18

llama 7B F16 8 pp10 30.42 49.03 1.61

llama 7B F16 8 pp11 23.51 51.16 2.18

llama 7B F16 8 pp12 44.81 53.55 1.20

llama 7B F16 8 pp13 35.45 53.97 1.52

llama 7B F16 8 pp14 37.40 55.91 1.50

llama 7B F16 8 pp15 47.63 57.52 1.21

llama 7B F16 8 pp16 38.27 56.69 1.48

llama 7B F16 8 pp30 55.66 62.84 1.13

llama 7B F16 8 pp31 46.17 61.79 1.34

llama 7B F16 8 pp32 46.22 62.78 1.36

llama 7B F16 8 pp64 51.43 65.10 1.27

llama 7B F16 8 pp65 51.57 65.04 1.26

llama 7B F16 8 pp66 55.29 50.84 0.92

llama 7B F16 8 pp120 56.77 65.83 1.16

llama 7B F16 8 pp128 55.29 65.64 1.19

llama 7B F16 8 pp130 54.22 66.17 1.22

llama 7B F16 8 pp240 51.06 58.46 1.14

llama 7B F16 8 pp255 56.89 66.11 1.16

llama 7B F16 8 pp256 56.60 58.38 1.03

llama 7B F16 8 pp510 49.56 59.74 1.21

llama 7B F16 8 pp512 48.86 59.75 1.22

llama 7B F16 8 pp1023 50.73 58.39 1.15

llama 7B F16 8 pp1024 46.55 58.51 1.26

llama 7B F16 8 pp1025 50.70 58.01 1.14

llama 7B F16 8 pp2048 48.65 56.76 1.17

llama 7B F16 8 tg128 6.57 6.84 1.04

llama 7B F16 16 pp1 7.36 7.04 0.96

llama 7B F16 16 pp2 11.80 13.98 1.18

llama 7B F16 16 pp3 16.18 20.90 1.29

llama 7B F16 16 pp4 14.60 26.77 1.83

llama 7B F16 16 pp5 17.95 32.82 1.83

llama 7B F16 16 pp6 29.38 38.84 1.32

llama 7B F16 16 pp7 24.25 42.92 1.77

llama 7B F16 16 pp8 26.31 47.11 1.79

llama 7B F16 16 pp9 36.26 51.87 1.43

llama 7B F16 16 pp10 30.21 53.56 1.77

llama 7B F16 16 pp11 31.95 57.11 1.79

llama 7B F16 16 pp12 26.38 60.18 2.28

llama 7B F16 16 pp13 31.82 60.02 1.89

llama 7B F16 16 pp14 35.95 63.33 1.76

llama 7B F16 16 pp15 41.48 65.03 1.57

llama 7B F16 16 pp16 36.87 65.43 1.77

llama 7B F16 16 pp30 46.23 74.36 1.61

llama 7B F16 16 pp31 43.34 73.61 1.70

llama 7B F16 16 pp32 44.02 74.51 1.69

llama 7B F16 16 pp64 44.73 77.52 1.73

llama 7B F16 16 pp65 43.67 77.59 1.78

llama 7B F16 16 pp66 36.00 77.94 2.17

llama 7B F16 16 pp120 46.27 78.65 1.70

llama 7B F16 16 pp128 44.67 64.10 1.43

llama 7B F16 16 pp130 38.55 75.47 1.96

llama 7B F16 16 pp240 47.56 79.61 1.67

llama 7B F16 16 pp255 43.95 67.90 1.54

llama 7B F16 16 pp256 43.84 78.00 1.78

llama 7B F16 16 pp510 47.37 67.67 1.43

llama 7B F16 16 pp512 47.73 69.11 1.45

llama 7B F16 16 pp1023 46.70 69.32 1.48

llama 7B F16 16 pp1024 46.85 69.70 1.49

llama 7B F16 16 pp1025 46.43 69.02 1.49

llama 7B F16 16 pp2048 41.15 64.47 1.57

llama 7B F16 16 tg128 6.77 6.79 1.00

llama 7B F16 24 pp1 6.72 6.70 1.00

llama 7B F16 24 pp2 13.08 13.80 1.06

llama 7B F16 24 pp3 19.12 19.80 1.04

llama 7B F16 24 pp4 17.18 25.52 1.49

llama 7B F16 24 pp5 20.68 31.70 1.53

llama 7B F16 24 pp6 33.86 25.86 0.76

llama 7B F16 24 pp7 20.49 42.41 2.07

llama 7B F16 24 pp8 32.02 49.30 1.54

llama 7B F16 24 pp9 45.35 54.32 1.20

llama 7B F16 24 pp10 38.55 58.23 1.51

llama 7B F16 24 pp11 41.01 62.25 1.52

llama 7B F16 24 pp12 54.11 65.30 1.21

llama 7B F16 24 pp13 45.66 66.65 1.46

llama 7B F16 24 pp14 48.00 68.97 1.44

llama 7B F16 24 pp15 58.72 74.53 1.27

llama 7B F16 24 pp16 50.72 74.72 1.47

llama 7B F16 24 pp30 65.48 84.78 1.29

llama 7B F16 24 pp31 60.81 84.96 1.40

llama 7B F16 24 pp32 61.80 85.63 1.39

llama 7B F16 24 pp64 65.16 85.38 1.31

llama 7B F16 24 pp65 64.48 84.86 1.32

llama 7B F16 24 pp66 66.96 84.38 1.26

llama 7B F16 24 pp120 55.89 85.90 1.54

llama 7B F16 24 pp128 65.76 68.28 1.04

llama 7B F16 24 pp130 65.48 85.78 1.31

llama 7B F16 24 pp240 59.03 87.17 1.48

llama 7B F16 24 pp255 66.37 84.71 1.28

llama 7B F16 24 pp256 65.55 86.56 1.32

llama 7B F16 24 pp510 64.20 74.53 1.16

llama 7B F16 24 pp512 63.80 82.39 1.29

llama 7B F16 24 pp1023 58.59 76.50 1.31

llama 7B F16 24 pp1024 58.57 73.12 1.25

llama 7B F16 24 pp1025 58.36 75.34 1.29

llama 7B F16 24 pp2048 53.68 71.46 1.33

llama 7B F16 24 tg128 6.31 6.50 1.03

llama 7B F16 32 pp1 6.95 6.52 0.94

llama 7B F16 32 pp2 10.93 13.09 1.20

llama 7B F16 32 pp3 20.45 16.78 0.82

llama 7B F16 32 pp4 18.21 25.07 1.38

llama 7B F16 32 pp5 21.85 31.60 1.45

llama 7B F16 32 pp6 37.12 33.09 0.89

llama 7B F16 32 pp7 29.10 43.65 1.50

llama 7B F16 32 pp8 35.45 49.34 1.39

llama 7B F16 32 pp9 45.19 46.98 1.04

llama 7B F16 32 pp10 43.03 58.52 1.36

llama 7B F16 32 pp11 42.45 55.39 1.30

llama 7B F16 32 pp12 61.48 64.59 1.05

llama 7B F16 32 pp13 47.92 67.93 1.42

llama 7B F16 32 pp14 53.72 63.25 1.18

llama 7B F16 32 pp15 62.70 74.37 1.19

llama 7B F16 32 pp16 58.18 65.05 1.12

llama 7B F16 32 pp30 81.05 86.72 1.07

llama 7B F16 32 pp31 70.93 85.87 1.21

llama 7B F16 32 pp32 69.70 86.73 1.24

llama 7B F16 32 pp64 75.62 85.96 1.14

llama 7B F16 32 pp65 80.69 88.52 1.10

llama 7B F16 32 pp66 86.93 88.50 1.02

llama 7B F16 32 pp120 88.89 90.04 1.01

llama 7B F16 32 pp128 85.38 72.80 0.85

llama 7B F16 32 pp130 82.46 88.17 1.07

llama 7B F16 32 pp240 73.02 90.63 1.24

llama 7B F16 32 pp255 88.03 90.93 1.03

llama 7B F16 32 pp256 84.65 78.67 0.93

llama 7B F16 32 pp510 78.80 87.88 1.12

llama 7B F16 32 pp512 73.98 88.27 1.19

llama 7B F16 32 pp1023 71.23 82.14 1.15

llama 7B F16 32 pp1024 74.38 82.08 1.10

llama 7B F16 32 pp1025 73.94 80.76 1.09

llama 7B F16 32 pp2048 65.88 75.73 1.15

llama 7B F16 32 tg128 5.92 5.98 1.01

Model Threads Test t/s master t/s perfo/tinyblas Speedup

llama 7B all F32 8 pp1 3.29 3.60 1.10

llama 7B all F32 8 pp2 6.29 7.10 1.13

llama 7B all F32 8 pp3 9.88 10.64 1.08

llama 7B all F32 8 pp4 7.28 13.62 1.87

llama 7B all F32 8 pp5 9.33 16.96 1.82

llama 7B all F32 8 pp6 18.77 20.40 1.09

llama 7B all F32 8 pp7 12.29 22.44 1.83

llama 7B all F32 8 pp8 14.00 17.29 1.24

llama 7B all F32 8 pp9 27.70 27.44 0.99

llama 7B all F32 8 pp10 12.74 28.43 2.23

llama 7B all F32 8 pp11 18.08 31.63 1.75

llama 7B all F32 8 pp12 34.07 36.68 1.08

llama 7B all F32 8 pp13 21.59 37.41 1.73

llama 7B all F32 8 pp14 23.54 41.72 1.77

llama 7B all F32 8 pp15 42.10 45.17 1.07

llama 7B all F32 8 pp16 26.58 45.03 1.69

llama 7B all F32 8 pp30 60.69 68.16 1.12

llama 7B all F32 8 pp31 42.30 66.59 1.57

llama 7B all F32 8 pp32 43.45 70.13 1.61

llama 7B all F32 8 pp64 60.06 81.96 1.36

llama 7B all F32 8 pp65 58.62 83.51 1.42

llama 7B all F32 8 pp66 52.86 80.40 1.52

llama 7B all F32 8 pp120 75.73 66.10 0.87

llama 7B all F32 8 pp128 71.88 79.70 1.11

llama 7B all F32 8 pp130 69.67 84.03 1.21

llama 7B all F32 8 pp240 57.52 76.86 1.34

llama 7B all F32 8 pp255 68.61 63.99 0.93

llama 7B all F32 8 pp256 57.10 76.73 1.34

llama 7B all F32 8 pp510 58.15 63.27 1.09

llama 7B all F32 8 pp512 57.39 65.34 1.14

llama 7B all F32 8 pp1023 56.31 61.57 1.09

llama 7B all F32 8 pp1024 56.42 63.13 1.12

llama 7B all F32 8 pp1025 55.91 64.92 1.16

llama 7B all F32 8 pp2048 54.29 64.00 1.18

llama 7B all F32 8 tg128 3.30 3.30 1.00

llama 7B all F32 16 pp1 3.55 3.57 1.01

llama 7B all F32 16 pp2 5.44 7.26 1.33

llama 7B all F32 16 pp3 8.37 10.87 1.30

llama 7B all F32 16 pp4 6.61 14.35 2.17

llama 7B all F32 16 pp5 7.75 17.88 2.31

llama 7B all F32 16 pp6 15.87 21.44 1.35

llama 7B all F32 16 pp7 11.32 24.51 2.17

llama 7B all F32 16 pp8 12.19 28.14 2.31

llama 7B all F32 16 pp9 22.32 31.75 1.42

llama 7B all F32 16 pp10 15.65 34.38 2.20

llama 7B all F32 16 pp11 12.30 37.65 3.06

llama 7B all F32 16 pp12 28.21 41.11 1.46

llama 7B all F32 16 pp13 19.83 43.12 2.17

llama 7B all F32 16 pp14 20.17 46.27 2.29

llama 7B all F32 16 pp15 33.49 49.29 1.47

llama 7B all F32 16 pp16 23.59 50.51 2.14

llama 7B all F32 16 pp30 48.91 80.46 1.65

llama 7B all F32 16 pp31 37.14 79.76 2.15

llama 7B all F32 16 pp32 36.78 82.64 2.25

llama 7B all F32 16 pp64 52.07 103.75 1.99

llama 7B all F32 16 pp65 39.03 76.48 1.96

llama 7B all F32 16 pp66 57.92 104.47 1.80

llama 7B all F32 16 pp120 63.20 112.86 1.79

llama 7B all F32 16 pp128 58.30 111.95 1.92

llama 7B all F32 16 pp130 57.93 112.51 1.94

llama 7B all F32 16 pp240 58.98 103.05 1.75

llama 7B all F32 16 pp255 62.83 84.99 1.35

llama 7B all F32 16 pp256 60.20 102.69 1.71

llama 7B all F32 16 pp510 58.07 82.11 1.41

llama 7B all F32 16 pp512 56.77 88.96 1.57

llama 7B all F32 16 pp1023 57.33 84.32 1.47

llama 7B all F32 16 pp1024 57.65 83.80 1.45

llama 7B all F32 16 pp1025 54.17 85.64 1.58

llama 7B all F32 16 pp2048 53.54 77.67 1.45

llama 7B all F32 16 tg128 3.38 3.32 0.98

llama 7B all F32 24 pp1 3.57 3.58 1.00

llama 7B all F32 24 pp2 4.44 7.18 1.62

llama 7B all F32 24 pp3 9.34 10.78 1.15

llama 7B all F32 24 pp4 7.36 14.26 1.94

llama 7B all F32 24 pp5 8.86 17.80 2.01

llama 7B all F32 24 pp6 18.73 21.35 1.14

llama 7B all F32 24 pp7 13.51 24.65 1.82

llama 7B all F32 24 pp8 14.72 28.18 1.91

llama 7B all F32 24 pp9 27.34 31.47 1.15

llama 7B all F32 24 pp10 19.02 34.47 1.81

llama 7B all F32 24 pp11 20.65 37.93 1.84

llama 7B all F32 24 pp12 36.08 41.20 1.14

llama 7B all F32 24 pp13 25.13 43.69 1.74

llama 7B all F32 24 pp14 25.49 47.22 1.85

llama 7B all F32 24 pp15 40.66 50.46 1.24

llama 7B all F32 24 pp16 28.48 52.54 1.84

llama 7B all F32 24 pp30 60.87 85.70 1.41

llama 7B all F32 24 pp31 46.29 85.91 1.86

llama 7B all F32 24 pp32 33.68 88.61 2.63

llama 7B all F32 24 pp64 63.22 115.66 1.83

llama 7B all F32 24 pp65 62.47 115.16 1.84

llama 7B all F32 24 pp66 72.74 116.25 1.60

llama 7B all F32 24 pp120 82.94 125.57 1.51

llama 7B all F32 24 pp128 71.71 125.38 1.75

llama 7B all F32 24 pp130 58.06 123.70 2.13

llama 7B all F32 24 pp240 71.65 115.68 1.61

llama 7B all F32 24 pp255 68.59 97.20 1.42

llama 7B all F32 24 pp256 67.18 114.58 1.71

llama 7B all F32 24 pp510 62.15 106.84 1.72

llama 7B all F32 24 pp512 60.77 107.10 1.76

llama 7B all F32 24 pp1023 59.90 94.97 1.59

llama 7B all F32 24 pp1024 59.98 96.93 1.62

llama 7B all F32 24 pp1025 59.06 94.01 1.59

llama 7B all F32 24 pp2048 59.30 91.46 1.54

llama 7B all F32 24 tg128 3.40 3.40 1.00

llama 7B all F32 32 pp1 3.44 3.03 0.88

llama 7B all F32 32 pp2 6.29 5.22 0.83

llama 7B all F32 32 pp3 9.26 10.28 1.11

llama 7B all F32 32 pp4 7.41 14.41 1.94

llama 7B all F32 32 pp5 8.90 16.51 1.85

llama 7B all F32 32 pp6 13.72 20.73 1.51

llama 7B all F32 32 pp7 12.95 25.04 1.93

llama 7B all F32 32 pp8 14.24 26.06 1.83

llama 7B all F32 32 pp9 26.56 29.83 1.12

llama 7B all F32 32 pp10 18.28 33.25 1.82

llama 7B all F32 32 pp11 19.46 33.97 1.75

llama 7B all F32 32 pp12 35.39 38.65 1.09

llama 7B all F32 32 pp13 23.80 42.63 1.79

llama 7B all F32 32 pp14 25.25 42.34 1.68

llama 7B all F32 32 pp15 44.43 43.85 0.99

llama 7B all F32 32 pp16 27.65 33.80 1.22

llama 7B all F32 32 pp30 62.07 76.51 1.23

llama 7B all F32 32 pp31 48.28 79.72 1.65

llama 7B all F32 32 pp32 50.50 83.07 1.64

llama 7B all F32 32 pp64 62.50 114.16 1.83

llama 7B all F32 32 pp65 72.63 119.77 1.65

llama 7B all F32 32 pp66 89.07 121.06 1.36

llama 7B all F32 32 pp120 91.80 131.97 1.44

llama 7B all F32 32 pp128 84.08 130.04 1.55

llama 7B all F32 32 pp130 84.99 128.60 1.51

llama 7B all F32 32 pp240 75.48 123.00 1.63

llama 7B all F32 32 pp255 82.80 125.00 1.51

llama 7B all F32 32 pp256 80.86 125.54 1.55

llama 7B all F32 32 pp510 79.07 103.85 1.31

llama 7B all F32 32 pp512 72.10 118.05 1.64

llama 7B all F32 32 pp1023 69.98 102.26 1.46

llama 7B all F32 32 pp1024 73.17 112.25 1.53

llama 7B all F32 32 pp1025 71.53 102.11 1.43

llama 7B all F32 32 pp2048 68.90 98.77 1.43

llama 7B all F32 32 tg128 3.13 3.16 1.01

@slaren Thanks for the benchmark !
Look good with the intel CPU. the FP32 is really good for PP...
with 24 thread it look well balanced now.

@ggerganov : it's really strange these differences. for now I don't see why. I'll try to do other tests later. it would be good to find out why, if it's a question of config it would be good to be able to indicate it.
I tried by disabling interlacing, the tg is half but the pp has almost the same peak.

Will try to make some change so the pp don't slow down. 🤞

Even if you don't find a solution to this, I think it is fine to ignore the results from my Ryzen, because it is very likely that I have misconfigured something in the BIOS.

- add bf16 suport - change dispache strategie (thanks: ikawrakow/ik_llama.cpp#71 ) - reduce memory bandwidth simple tinyblas dispache and more cache freindly

- show-progress is not part of GNU Wget2

Djip007 · 2024-12-24T14:06:25Z

OK code look good and I get good perf with Ryzen9 5950X and 7945HS.

Need to "remove" not working test in "Server" check

Djip007 · 2024-12-24T14:30:50Z

Some last bench with ("without" u-batch):

./scripts/compare-commits.sh master perfo/tinyblas -ctk bf16 -ctv bf16 -r 3 -ub 2048 -b 2048 -m "Mistral-Nemo-Instruct-2407.BF16.gguf,Mistral-7B-Instruct-v0.3-BF16.gguf" -p "1,1,1,2,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,30,31,32,33,34,35,60,61,62,63,64,65,127,128,129,130,131,132,255,256,257,258,259,260,510,511,512,513,514,515,1023,1024,1025,1026,1027,1028,2048"
./scripts/compare-commits.sh master perfo/tinyblas -ctk  f16 -ctv  f16 -r 3 -ub 2048 -b 2048 -m "Mistral-Nemo-Instruct-2407.F16.gguf,Mistral-7B-Instruct-v0.3-FP16.gguf"  -p "1,1,1,2,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,30,31,32,33,34,35,60,61,62,63,64,65,127,128,129,130,131,132,255,256,257,258,259,260,510,511,512,513,514,515,1023,1024,1025,1026,1027,1028,2048"

Do not direct compare with preview result, I have some change on Bios config (PBO / max TDP...)

AMD Ryzen 9 7940HS

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 13B BF16	pp1	2.50	2.50	1.00
llama 13B BF16	pp2	4.93	4.92	1.00
llama 13B BF16	pp3	7.29	7.36	1.01
llama 13B BF16	pp4	9.43	9.77	1.04
llama 13B BF16	pp5	11.52	12.19	1.06
llama 13B BF16	pp6	13.38	14.52	1.08
llama 13B BF16	pp7	14.93	16.82	1.13
llama 13B BF16	pp8	16.25	19.12	1.18
llama 13B BF16	pp9	16.62	21.29	1.28
llama 13B BF16	pp10	17.23	23.61	1.37
llama 13B BF16	pp11	17.79	25.73	1.45
llama 13B BF16	pp12	18.14	27.73	1.53
llama 13B BF16	pp13	18.42	29.29	1.59
llama 13B BF16	pp14	18.72	31.34	1.67
llama 13B BF16	pp15	18.99	32.94	1.73
llama 13B BF16	pp16	19.20	34.85	1.82
llama 13B BF16	pp30	20.39	46.59	2.28
llama 13B BF16	pp31	20.49	42.57	2.08
llama 13B BF16	pp32	20.56	42.90	2.09
llama 13B BF16	pp33	20.87	44.29	2.12
llama 13B BF16	pp34	19.86	44.94	2.26
llama 13B BF16	pp35	20.01	46.20	2.31
llama 13B BF16	pp60	21.04	47.46	2.26
llama 13B BF16	pp61	21.08	46.64	2.21
llama 13B BF16	pp62	21.12	47.06	2.23
llama 13B BF16	pp63	21.17	47.59	2.25
llama 13B BF16	pp64	21.20	47.77	2.25
llama 13B BF16	pp65	20.49	47.88	2.34
llama 13B BF16	pp127	21.41	54.12	2.53
llama 13B BF16	pp128	21.46	54.14	2.52
llama 13B BF16	pp129	22.27	54.93	2.47
llama 13B BF16	pp130	20.71	53.92	2.60
llama 13B BF16	pp131	21.08	53.82	2.55
llama 13B BF16	pp132	20.99	54.07	2.58
llama 13B BF16	pp255	23.21	55.18	2.38
llama 13B BF16	pp256	22.95	55.09	2.40
llama 13B BF16	pp257	22.02	55.01	2.50
llama 13B BF16	pp258	22.19	55.11	2.48
llama 13B BF16	pp259	22.24	54.95	2.47
llama 13B BF16	pp260	22.31	55.03	2.47
llama 13B BF16	pp510	22.73	54.50	2.40
llama 13B BF16	pp511	22.82	54.48	2.39
llama 13B BF16	pp512	22.82	54.48	2.39
llama 13B BF16	pp513	22.23	54.56	2.45
llama 13B BF16	pp514	22.24	54.64	2.46
llama 13B BF16	pp515	22.24	54.62	2.46
llama 13B BF16	pp1023	20.64	52.60	2.55
llama 13B BF16	pp1024	20.78	52.32	2.52
llama 13B BF16	pp1025	20.37	52.84	2.59
llama 13B BF16	pp1026	20.44	52.78	2.58
llama 13B BF16	pp1027	20.46	52.77	2.58
llama 13B BF16	pp1028	20.47	52.63	2.57
llama 13B BF16	pp2048	19.91	49.12	2.47
llama 13B BF16	tg128	2.50	2.50	1.00
llama 7B BF16	pp1	4.06	4.05	1.00
llama 7B BF16	pp2	8.03	8.01	1.00
llama 7B BF16	pp3	11.85	11.93	1.01
llama 7B BF16	pp4	15.41	15.77	1.02
llama 7B BF16	pp5	18.63	19.70	1.06
llama 7B BF16	pp6	21.46	23.47	1.09
llama 7B BF16	pp7	23.87	27.15	1.14
llama 7B BF16	pp8	26.02	30.93	1.19
llama 7B BF16	pp9	27.46	34.42	1.25
llama 7B BF16	pp10	28.66	38.16	1.33
llama 7B BF16	pp11	29.45	41.58	1.41
llama 7B BF16	pp12	30.16	44.81	1.49
llama 7B BF16	pp13	30.69	47.66	1.55
llama 7B BF16	pp14	31.27	50.85	1.63
llama 7B BF16	pp15	31.66	53.77	1.70
llama 7B BF16	pp16	32.04	56.74	1.77
llama 7B BF16	pp30	33.27	68.29	2.05
llama 7B BF16	pp31	30.95	73.07	2.36
llama 7B BF16	pp32	31.91	74.05	2.32
llama 7B BF16	pp33	30.81	75.33	2.45
llama 7B BF16	pp34	30.98	75.20	2.43
llama 7B BF16	pp35	31.13	77.35	2.48
llama 7B BF16	pp60	32.62	77.80	2.38
llama 7B BF16	pp61	32.70	75.81	2.32
llama 7B BF16	pp62	32.78	75.54	2.30
llama 7B BF16	pp63	32.83	76.46	2.33
llama 7B BF16	pp64	32.90	77.16	2.35
llama 7B BF16	pp65	31.68	76.93	2.43
llama 7B BF16	pp127	35.38	83.55	2.36
llama 7B BF16	pp128	35.02	74.07	2.12
llama 7B BF16	pp129	33.97	73.78	2.17
llama 7B BF16	pp130	34.04	73.64	2.16
llama 7B BF16	pp131	34.09	73.67	2.16
llama 7B BF16	pp132	34.13	73.72	2.16
llama 7B BF16	pp255	35.70	86.47	2.42
llama 7B BF16	pp256	35.73	87.25	2.44
llama 7B BF16	pp257	34.33	87.09	2.54
llama 7B BF16	pp258	34.41	87.25	2.54
llama 7B BF16	pp259	34.51	86.82	2.52
llama 7B BF16	pp260	34.60	86.93	2.51
llama 7B BF16	pp510	32.46	85.43	2.63
llama 7B BF16	pp511	32.47	85.47	2.63
llama 7B BF16	pp512	32.51	85.47	2.63
llama 7B BF16	pp513	31.65	85.64	2.71
llama 7B BF16	pp514	31.49	85.62	2.72
llama 7B BF16	pp515	31.75	85.63	2.70
llama 7B BF16	pp1023	31.20	69.76	2.24
llama 7B BF16	pp1024	31.25	69.64	2.23
llama 7B BF16	pp1025	30.55	70.17	2.30
llama 7B BF16	pp1026	30.61	69.97	2.29
llama 7B BF16	pp1027	30.60	69.98	2.29
llama 7B BF16	pp1028	30.68	69.88	2.28
llama 7B BF16	pp2048	29.64	64.58	2.18
llama 7B BF16	tg128	4.07	4.07	1.00

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 13B F16	pp1	2.47	2.48	1.00
llama 13B F16	pp2	4.90	4.86	0.99
llama 13B F16	pp3	7.27	7.21	0.99
llama 13B F16	pp4	9.69	9.63	0.99
llama 13B F16	pp5	12.01	11.94	0.99
llama 13B F16	pp6	7.79	14.23	1.83
llama 13B F16	pp7	9.03	16.41	1.82
llama 13B F16	pp8	10.32	18.58	1.80
llama 13B F16	pp9	11.52	20.50	1.78
llama 13B F16	pp10	22.45	22.34	1.00
llama 13B F16	pp11	13.82	24.07	1.74
llama 13B F16	pp12	14.94	25.39	1.70
llama 13B F16	pp13	16.22	25.89	1.60
llama 13B F16	pp14	17.37	26.51	1.53
llama 13B F16	pp15	28.39	26.82	0.94
llama 13B F16	pp16	18.73	27.45	1.47
llama 13B F16	pp30	31.39	30.91	0.98
llama 13B F16	pp31	24.75	30.41	1.23
llama 13B F16	pp32	25.41	30.70	1.21
llama 13B F16	pp33	25.97	30.76	1.18
llama 13B F16	pp34	26.37	30.95	1.17
llama 13B F16	pp35	31.53	31.17	0.99
llama 13B F16	pp60	31.42	32.06	1.02
llama 13B F16	pp61	26.56	31.79	1.20
llama 13B F16	pp62	26.87	31.76	1.18
llama 13B F16	pp63	27.19	31.72	1.17
llama 13B F16	pp64	25.92	31.28	1.21
llama 13B F16	pp65	29.19	32.07	1.10
llama 13B F16	pp127	27.97	31.86	1.14
llama 13B F16	pp128	28.12	32.41	1.15
llama 13B F16	pp129	29.07	32.59	1.12
llama 13B F16	pp130	31.41	32.36	1.03
llama 13B F16	pp131	28.84	31.41	1.09
llama 13B F16	pp132	28.93	32.42	1.12
llama 13B F16	pp255	30.65	32.66	1.07
llama 13B F16	pp256	29.24	32.66	1.12
llama 13B F16	pp257	28.49	32.60	1.14
llama 13B F16	pp258	28.40	32.60	1.15
llama 13B F16	pp259	28.37	32.56	1.15
llama 13B F16	pp260	28.85	32.60	1.13
llama 13B F16	pp510	23.79	32.06	1.35
llama 13B F16	pp511	23.37	32.15	1.38
llama 13B F16	pp512	23.30	32.13	1.38
llama 13B F16	pp513	23.65	31.86	1.35
llama 13B F16	pp514	23.40	32.12	1.37
llama 13B F16	pp515	23.58	32.14	1.36
llama 13B F16	pp1023	16.67	30.88	1.85
llama 13B F16	pp1024	16.46	30.87	1.88
llama 13B F16	pp1025	16.83	30.78	1.83
llama 13B F16	pp1026	16.55	30.81	1.86
llama 13B F16	pp1027	16.51	30.83	1.87
llama 13B F16	pp1028	16.77	30.83	1.84
llama 13B F16	pp2048	14.40	29.17	2.03
llama 13B F16	tg128	2.48	2.48	1.00
llama 7B F16	pp1	4.03	4.01	0.99
llama 7B F16	pp2	7.96	7.82	0.98
llama 7B F16	pp3	11.82	11.63	0.98
llama 7B F16	pp4	15.73	15.44	0.98
llama 7B F16	pp5	19.48	19.25	0.99
llama 7B F16	pp6	12.65	22.85	1.81
llama 7B F16	pp7	14.67	26.31	1.79
llama 7B F16	pp8	16.70	29.88	1.79
llama 7B F16	pp9	18.61	32.89	1.77
llama 7B F16	pp10	36.35	35.95	0.99
llama 7B F16	pp11	22.44	38.63	1.72
llama 7B F16	pp12	24.33	40.96	1.68
llama 7B F16	pp13	26.21	41.44	1.58
llama 7B F16	pp14	27.96	42.89	1.53
llama 7B F16	pp15	46.01	43.46	0.94
llama 7B F16	pp16	30.31	44.67	1.47
llama 7B F16	pp30	49.97	48.86	0.98
llama 7B F16	pp31	39.56	48.18	1.22
llama 7B F16	pp32	40.60	48.55	1.20
llama 7B F16	pp33	41.30	48.53	1.18
llama 7B F16	pp34	42.17	49.00	1.16
llama 7B F16	pp35	50.29	49.02	0.97
llama 7B F16	pp60	50.50	50.52	1.00
llama 7B F16	pp61	42.49	50.14	1.18
llama 7B F16	pp62	43.11	50.26	1.17
llama 7B F16	pp63	43.70	50.44	1.15
llama 7B F16	pp64	44.22	50.64	1.15
llama 7B F16	pp65	50.40	50.32	1.00
llama 7B F16	pp127	46.20	50.67	1.10
llama 7B F16	pp128	46.47	48.38	1.04
llama 7B F16	pp129	46.58	48.64	1.04
llama 7B F16	pp130	49.76	49.00	0.98
llama 7B F16	pp131	45.91	49.13	1.07
llama 7B F16	pp132	46.23	49.10	1.06
llama 7B F16	pp255	46.41	50.86	1.10
llama 7B F16	pp256	45.00	50.82	1.13
llama 7B F16	pp257	44.22	50.70	1.15
llama 7B F16	pp258	44.25	50.72	1.15
llama 7B F16	pp259	43.94	50.64	1.15
llama 7B F16	pp260	45.03	50.68	1.13
llama 7B F16	pp510	37.94	50.18	1.32
llama 7B F16	pp511	37.36	50.13	1.34
llama 7B F16	pp512	37.27	50.13	1.35
llama 7B F16	pp513	37.42	49.84	1.33
llama 7B F16	pp514	37.26	50.24	1.35
llama 7B F16	pp515	38.03	50.25	1.32
llama 7B F16	pp1023	29.16	45.71	1.57
llama 7B F16	pp1024	28.57	45.47	1.59
llama 7B F16	pp1025	28.08	45.52	1.62
llama 7B F16	pp1026	27.90	45.59	1.63
llama 7B F16	pp1027	28.40	45.51	1.60
llama 7B F16	pp1028	29.58	45.46	1.54
llama 7B F16	pp2048	21.20	42.23	1.99
llama 7B F16	tg128	4.02	4.03	1.00

AMD Ryzen 9 5950X

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 13B BF16	pp1	2.13	2.13	1.00
llama 13B BF16	pp2	4.23	4.15	0.98
llama 13B BF16	pp3	6.30	6.21	0.99
llama 13B BF16	pp4	8.27	8.29	1.00
llama 13B BF16	pp5	10.14	10.33	1.02
llama 13B BF16	pp6	11.87	12.37	1.04
llama 13B BF16	pp7	13.43	14.45	1.08
llama 13B BF16	pp8	14.75	16.49	1.12
llama 13B BF16	pp9	15.77	18.52	1.17
llama 13B BF16	pp10	16.59	20.51	1.24
llama 13B BF16	pp11	17.26	22.56	1.31
llama 13B BF16	pp12	17.88	24.53	1.37
llama 13B BF16	pp13	18.35	26.66	1.45
llama 13B BF16	pp14	18.79	28.53	1.52
llama 13B BF16	pp15	19.12	30.48	1.59
llama 13B BF16	pp16	19.49	32.59	1.67
llama 13B BF16	pp30	20.52	46.80	2.28
llama 13B BF16	pp32	20.74	46.93	2.26
llama 13B BF16	pp64	21.39	49.16	2.30
llama 13B BF16	pp65	20.72	49.68	2.40
llama 13B BF16	pp66	20.78	49.94	2.40
llama 13B BF16	pp127	21.70	50.44	2.32
llama 13B BF16	pp128	21.66	50.56	2.33
llama 13B BF16	pp129	21.34	50.60	2.37
llama 13B BF16	pp255	21.87	50.76	2.32
llama 13B BF16	pp256	21.91	50.62	2.31
llama 13B BF16	pp257	21.40	50.55	2.36
llama 13B BF16	pp510	21.83	50.18	2.30
llama 13B BF16	pp511	21.90	50.07	2.29
llama 13B BF16	pp512	21.92	50.12	2.29
llama 13B BF16	pp513	21.66	50.19	2.32
llama 13B BF16	pp1023	21.33	48.41	2.27
llama 13B BF16	pp1024	21.40	48.46	2.26
llama 13B BF16	pp1025	21.25	48.37	2.28
llama 13B BF16	pp2048	20.46	45.49	2.22
llama 13B BF16	tg128	2.13	2.13	1.00
llama 7B BF16	pp1	3.48	3.48	1.00
llama 7B BF16	pp2	6.87	6.74	0.98
llama 7B BF16	pp3	10.17	10.09	0.99
llama 7B BF16	pp4	13.38	13.45	1.01
llama 7B BF16	pp5	16.39	16.77	1.02
llama 7B BF16	pp6	19.07	20.10	1.05
llama 7B BF16	pp7	21.51	23.44	1.09
llama 7B BF16	pp8	23.70	26.76	1.13
llama 7B BF16	pp9	25.33	30.11	1.19
llama 7B BF16	pp10	26.74	33.45	1.25
llama 7B BF16	pp11	27.85	36.79	1.32
llama 7B BF16	pp12	28.54	39.95	1.40
llama 7B BF16	pp13	29.35	43.38	1.48
llama 7B BF16	pp14	29.98	46.44	1.55
llama 7B BF16	pp15	30.44	49.54	1.63
llama 7B BF16	pp16	30.88	52.88	1.71
llama 7B BF16	pp30	32.19	74.83	2.32
llama 7B BF16	pp32	32.63	74.31	2.28
llama 7B BF16	pp64	33.74	78.01	2.31
llama 7B BF16	pp65	32.78	78.36	2.39
llama 7B BF16	pp66	32.86	78.76	2.40
llama 7B BF16	pp127	34.04	79.20	2.33
llama 7B BF16	pp128	33.99	79.43	2.34
llama 7B BF16	pp129	33.53	79.34	2.37
llama 7B BF16	pp255	34.28	79.33	2.31
llama 7B BF16	pp256	34.35	79.11	2.30
llama 7B BF16	pp257	33.62	78.86	2.35
llama 7B BF16	pp510	34.10	78.20	2.29
llama 7B BF16	pp511	34.18	78.01	2.28
llama 7B BF16	pp512	34.20	77.96	2.28
llama 7B BF16	pp513	33.85	77.83	2.30
llama 7B BF16	pp1023	33.14	74.47	2.25
llama 7B BF16	pp1024	33.09	74.55	2.25
llama 7B BF16	pp1025	32.89	74.52	2.27
llama 7B BF16	pp2048	31.33	69.52	2.22
llama 7B BF16	tg128	3.48	3.48	1.00

Model	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 13B F16	pp1	2.12	2.12	1.00
llama 13B F16	pp2	4.14	4.12	1.00
llama 13B F16	pp3	6.23	6.19	0.99
llama 13B F16	pp4	4.80	8.23	1.71
llama 13B F16	pp5	5.93	10.28	1.73
llama 13B F16	pp6	12.43	12.30	0.99
llama 13B F16	pp7	8.38	14.34	1.71
llama 13B F16	pp8	9.47	16.39	1.73
llama 13B F16	pp9	18.64	18.39	0.99
llama 13B F16	pp10	12.13	20.43	1.69
llama 13B F16	pp11	13.25	22.45	1.69
llama 13B F16	pp12	24.73	24.41	0.99
llama 13B F16	pp13	15.87	26.46	1.67
llama 13B F16	pp14	16.95	28.43	1.68
llama 13B F16	pp15	30.41	30.38	1.00
llama 13B F16	pp16	19.48	32.39	1.66
llama 13B F16	pp30	36.04	57.18	1.59
llama 13B F16	pp32	27.90	58.94	2.11
llama 13B F16	pp64	33.24	72.24	2.17
llama 13B F16	pp65	33.64	72.34	2.15
llama 13B F16	pp66	37.38	72.77	1.95
llama 13B F16	pp127	36.07	74.40	2.06
llama 13B F16	pp128	36.18	74.69	2.06
llama 13B F16	pp129	37.63	74.48	1.98
llama 13B F16	pp255	37.67	75.16	2.00
llama 13B F16	pp256	36.81	74.99	2.04
llama 13B F16	pp257	36.91	74.86	2.03
llama 13B F16	pp510	36.38	74.30	2.04
llama 13B F16	pp511	35.98	74.15	2.06
llama 13B F16	pp512	36.04	74.07	2.05
llama 13B F16	pp513	36.37	73.95	2.03
llama 13B F16	pp1023	33.32	70.80	2.12
llama 13B F16	pp1024	33.02	70.94	2.15
llama 13B F16	pp1025	32.96	70.97	2.15
llama 13B F16	pp2048	28.15	65.91	2.34
llama 13B F16	tg128	2.12	2.12	1.00
llama 7B F16	pp1	3.42	3.46	1.01
llama 7B F16	pp2	6.66	6.69	1.00
llama 7B F16	pp3	9.99	10.05	1.01
llama 7B F16	pp4	8.00	13.33	1.67
llama 7B F16	pp5	9.86	16.66	1.69
llama 7B F16	pp6	19.72	19.97	1.01
llama 7B F16	pp7	13.79	23.26	1.69
llama 7B F16	pp8	15.62	26.52	1.70
llama 7B F16	pp9	28.72	29.79	1.04
llama 7B F16	pp10	19.63	33.07	1.68
llama 7B F16	pp11	21.45	36.34	1.69
llama 7B F16	pp12	36.86	39.47	1.07
llama 7B F16	pp13	25.34	42.86	1.69
llama 7B F16	pp14	27.18	45.88	1.69
llama 7B F16	pp15	46.50	49.01	1.05
llama 7B F16	pp16	30.97	52.40	1.69
llama 7B F16	pp30	57.24	91.35	1.60
llama 7B F16	pp32	45.19	95.17	2.11
llama 7B F16	pp64	52.39	113.43	2.16
llama 7B F16	pp65	52.47	113.70	2.17
llama 7B F16	pp66	57.30	114.33	2.00
llama 7B F16	pp127	56.24	117.03	2.08
llama 7B F16	pp128	57.08	117.17	2.05
llama 7B F16	pp129	59.33	116.79	1.97
llama 7B F16	pp255	59.08	118.12	2.00
llama 7B F16	pp256	57.56	117.96	2.05
llama 7B F16	pp257	57.57	117.57	2.04
llama 7B F16	pp510	56.74	114.89	2.02
llama 7B F16	pp511	56.17	114.73	2.04
llama 7B F16	pp512	56.04	114.67	2.05
llama 7B F16	pp513	56.68	114.29	2.02
llama 7B F16	pp1023	52.08	108.43	2.08
llama 7B F16	pp1024	51.50	108.26	2.10
llama 7B F16	pp1025	51.55	108.39	2.10
llama 7B F16	pp2048	45.31	99.35	2.19
llama 7B F16	tg128	3.46	3.46	1.00

Djip007 · 2024-12-24T14:39:05Z

Perplexity look good.

./build/bin/./llama-perplexity -ctk bf16 -ctv bf16 --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.BF16.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9443 ±    0.5267       0.00048 ±    0.00053       0.00004 ±    0.00001     0.169 ±  0.019 %    99.608 ±  0.392 %
   2       5.4419 ±    0.6039       0.00133 ±    0.00153       0.00005 ±    0.00001     0.167 ±  0.012 %    99.412 ±  0.339 %
   3       4.6835 ±    0.4027       0.00066 ±    0.00105       0.00006 ±    0.00001     0.236 ±  0.021 %    99.608 ±  0.226 %
   4       5.0057 ±    0.3672       0.00051 ±    0.00080       0.00005 ±    0.00000     0.231 ±  0.017 %    99.608 ±  0.196 %
   5       5.2931 ±    0.3434       0.00030 ±    0.00065       0.00005 ±    0.00000     0.220 ±  0.014 %    99.686 ±  0.157 %
   6       5.8307 ±    0.3543       0.00030 ±    0.00055       0.00005 ±    0.00000     0.216 ±  0.012 %    99.739 ±  0.131 %
   7       6.2255 ±    0.3544       0.00047 ±    0.00052       0.00005 ±    0.00000     0.210 ±  0.011 %    99.664 ±  0.137 %
   8       6.4316 ±    0.3454       0.00047 ±    0.00046       0.00005 ±    0.00000     0.218 ±  0.010 %    99.657 ±  0.130 %
   9       6.8874 ±    0.3580       0.00050 ±    0.00041       0.00005 ±    0.00000     0.213 ±  0.010 %    99.608 ±  0.130 %
  10       7.2365 ±    0.3589       0.00030 ±    0.00038       0.00005 ±    0.00000     0.209 ±  0.009 %    99.569 ±  0.130 %

./build/bin/./llama-perplexity                     --kl-divergence-base ~/LLM/Mistral-Nemo-Instruct-2407.BF16.kld --kl-divergence -s 31337 -m ~/LLM/Mistral-Nemo-Instruct-2407.F16.gguf
chunk             PPL               ln(PPL(Q)/PPL(base))          KL Divergence              Δp RMS            Same top p
   1       3.9432 ±    0.5262       0.00021 ±    0.00007       0.00000 ±    0.00000     0.023 ±  0.003 %    100.000 ±  0.000 %
   2       5.4435 ±    0.6041       0.00163 ±    0.00150       0.00000 ±    0.00000     0.025 ±  0.002 %    100.000 ±  0.000 %
   3       4.6856 ±    0.4029       0.00111 ±    0.00100       0.00000 ±    0.00000     0.030 ±  0.002 %    100.000 ±  0.000 %
   4       5.0072 ±    0.3674       0.00081 ±    0.00075       0.00000 ±    0.00000     0.029 ±  0.002 %    100.000 ±  0.000 %
   5       5.2951 ±    0.3437       0.00067 ±    0.00060       0.00000 ±    0.00000     0.030 ±  0.002 %    100.000 ±  0.000 %
   6       5.8323 ±    0.3545       0.00057 ±    0.00050       0.00000 ±    0.00000     0.029 ±  0.002 %    100.000 ±  0.000 %
   7       6.2269 ±    0.3546       0.00069 ±    0.00047       0.00000 ±    0.00000     0.028 ±  0.001 %    100.000 ±  0.000 %
   8       6.4324 ±    0.3455       0.00059 ±    0.00041       0.00000 ±    0.00000     0.028 ±  0.001 %    100.000 ±  0.000 %
   9       6.8876 ±    0.3581       0.00053 ±    0.00036       0.00000 ±    0.00000     0.027 ±  0.001 %    100.000 ±  0.000 %
  10       7.2379 ±    0.3591       0.00049 ±    0.00033       0.00000 ±    0.00000     0.027 ±  0.001 %    99.961 ±  0.039 %

examples/server/tests/unit/test_completion.py

Djip007 · 2024-12-24T18:31:59Z

Thanks !

github-actions bot added the ggml changes relating to the ggml tensor library for machine learning label Dec 8, 2024

Djip007 marked this pull request as draft December 8, 2024 03:25

Djip007 force-pushed the perfo/tinyblas branch 5 times, most recently from f7c5a68 to b1c72b9 Compare December 9, 2024 22:34

Djip007 marked this pull request as ready for review December 10, 2024 00:52

invisiblepancake approved these changes Dec 11, 2024

View reviewed changes

Djip007 force-pushed the perfo/tinyblas branch from b1c72b9 to d4a2a20 Compare December 11, 2024 03:07

Djip007 marked this pull request as draft December 11, 2024 18:58

Djip007 force-pushed the perfo/tinyblas branch from d4a2a20 to 6c398db Compare December 11, 2024 19:46

Djip007 force-pushed the perfo/tinyblas branch from 6c398db to 01ba9f5 Compare December 11, 2024 20:10

slaren approved these changes Dec 11, 2024

View reviewed changes

github-actions bot added examples python python script changes server labels Dec 11, 2024

Djip007 commented Dec 11, 2024

View reviewed changes

invisiblepancake approved these changes Dec 11, 2024

View reviewed changes

Djip007 force-pushed the perfo/tinyblas branch 3 times, most recently from b2dab60 to 30ae0d2 Compare December 14, 2024 21:06

invisiblepancake approved these changes Dec 16, 2024

View reviewed changes

Djip007 added 3 commits December 23, 2024 00:23

more perfo with llamafile tinyblas on x86_64.

3f2bc65

- add bf16 suport - change dispache strategie (thanks: ikawrakow/ik_llama.cpp#71 ) - reduce memory bandwidth simple tinyblas dispache and more cache freindly

tinyblas dynamic dispaching

d732874

sgemm: add M blocs.

ac2b53c

Djip007 force-pushed the perfo/tinyblas branch from 30ae0d2 to 94cd488 Compare December 22, 2024 23:45

github-actions bot added the script Script related label Dec 22, 2024

slaren mentioned this pull request Dec 23, 2024

llamafile_sgemm API - INT8 implementation #10912

Open

ggerganov approved these changes Dec 23, 2024

View reviewed changes

- git 2.47 use short id of len 9.

22924d8

- show-progress is not part of GNU Wget2

Djip007 force-pushed the perfo/tinyblas branch from 94cd488 to 4bf8cd9 Compare December 24, 2024 14:02

Djip007 force-pushed the perfo/tinyblas branch from 4bf8cd9 to 7b9119b Compare December 24, 2024 14:10

Djip007 marked this pull request as ready for review December 24, 2024 14:40

Djip007 requested a review from ngxson as a code owner December 24, 2024 14:40

ngxson reviewed Dec 24, 2024

View reviewed changes

examples/server/tests/unit/test_completion.py Outdated Show resolved Hide resolved

remove not stable test

2c6864a

Djip007 force-pushed the perfo/tinyblas branch from 7b9119b to 2c6864a Compare December 24, 2024 16:41

ngxson approved these changes Dec 24, 2024

View reviewed changes

slaren merged commit 2cd43f4 into ggerganov:master Dec 24, 2024
50 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

more perfo with llamafile tinyblas on x86_64. #10714

more perfo with llamafile tinyblas on x86_64. #10714

Djip007 commented Dec 8, 2024 •

edited

Loading

Djip007 commented Dec 10, 2024

Djip007 commented Dec 10, 2024

slaren commented Dec 10, 2024

Djip007 commented Dec 11, 2024 •

edited

Loading

slaren commented Dec 11, 2024

Djip007 commented Dec 11, 2024

ggerganov commented Dec 11, 2024

Djip007 commented Dec 11, 2024

Djip007 commented Dec 11, 2024

slaren left a comment

Djip007 commented Dec 11, 2024

ggerganov commented Dec 11, 2024

slaren commented Dec 11, 2024

Djip007 commented Dec 11, 2024

slaren commented Dec 11, 2024

Djip007 commented Dec 11, 2024 •

edited

Loading

slaren commented Dec 11, 2024

Djip007 Dec 11, 2024 •

edited

Loading

Djip007 Dec 11, 2024

slaren Dec 11, 2024

Djip007 Dec 11, 2024

slaren Dec 11, 2024

Djip007 Dec 15, 2024 •

edited

Loading

ggerganov Dec 16, 2024

slaren Dec 16, 2024

Djip007 Dec 16, 2024

ggerganov Dec 23, 2024

Djip007 commented Dec 24, 2024

Djip007 commented Dec 24, 2024

Djip007 commented Dec 24, 2024

Djip007 commented Dec 24, 2024

Model	Threads	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 7B F16	8	pp64	52.68	61.85	1.17
llama 7B F16	16	pp64	42.89	37.64	0.88
llama 7B F16	24	pp64	63.02	56.05	0.89
llama 7B F16	32	pp64	77.93	68.80	0.88

Model	Test	t/s master	t/s PR	Speedup	PC
llama 7B F16	pp120	55.95	84.83	1.52	djip007
llama 7B F16	pp120	70.64	44.11	0.62	ggerganov

Model	Threads	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 7B BF16	8	pp1	6.54	6.85	1.05
llama 7B BF16	8	pp2	11.15	14.50	1.30
llama 7B BF16	8	pp3	14.71	21.46	1.46
llama 7B BF16	8	pp4	17.29	25.63	1.48
llama 7B BF16	8	pp5	19.17	32.46	1.69
llama 7B BF16	8	pp6	20.78	37.58	1.81
llama 7B BF16	8	pp7	22.28	40.30	1.81
llama 7B BF16	8	pp8	23.07	44.56	1.93
llama 7B BF16	8	pp9	22.69	47.97	2.11
llama 7B BF16	8	pp10	22.85	50.19	2.20
llama 7B BF16	8	pp11	23.28	52.97	2.28
llama 7B BF16	8	pp12	23.64	55.29	2.34
llama 7B BF16	8	pp13	23.93	55.26	2.31
llama 7B BF16	8	pp14	17.59	55.09	3.13
llama 7B BF16	8	pp15	23.55	57.91	2.46
llama 7B BF16	8	pp16	24.62	57.64	2.34
llama 7B BF16	8	pp30	26.04	63.60	2.44
llama 7B BF16	8	pp31	25.55	63.00	2.47
llama 7B BF16	8	pp32	25.07	60.48	2.41
llama 7B BF16	8	pp64	25.38	63.79	2.51
llama 7B BF16	8	pp65	24.97	66.64	2.67
llama 7B BF16	8	pp66	25.25	68.75	2.72
llama 7B BF16	8	pp120	23.24	67.85	2.92
llama 7B BF16	8	pp128	25.64	66.69	2.60
llama 7B BF16	8	pp130	23.48	54.18	2.31
llama 7B BF16	8	pp240	24.38	68.85	2.82
llama 7B BF16	8	pp255	24.51	66.71	2.72
llama 7B BF16	8	pp256	23.19	66.83	2.88
llama 7B BF16	8	pp510	22.54	60.81	2.70
llama 7B BF16	8	pp512	23.94	60.61	2.53
llama 7B BF16	8	pp1023	23.68	59.02	2.49
llama 7B BF16	8	pp1024	23.78	59.15	2.49
llama 7B BF16	8	pp1025	23.73	58.89	2.48
llama 7B BF16	8	pp2048	23.29	58.20	2.50
llama 7B BF16	8	tg128	6.69	6.42	0.96
llama 7B BF16	16	pp1	6.89	6.90	1.00
llama 7B BF16	16	pp2	13.49	14.19	1.05
llama 7B BF16	16	pp3	18.20	20.99	1.15
llama 7B BF16	16	pp4	21.92	27.21	1.24
llama 7B BF16	16	pp5	24.63	34.20	1.39
llama 7B BF16	16	pp6	26.45	39.95	1.51
llama 7B BF16	16	pp7	26.87	45.12	1.68
llama 7B BF16	16	pp8	20.01	49.56	2.48
llama 7B BF16	16	pp9	28.62	54.41	1.90
llama 7B BF16	16	pp10	30.72	57.11	1.86
llama 7B BF16	16	pp11	31.19	62.05	1.99
llama 7B BF16	16	pp12	31.64	65.01	2.05
llama 7B BF16	16	pp13	33.39	66.48	1.99
llama 7B BF16	16	pp14	33.71	69.13	2.05
llama 7B BF16	16	pp15	34.12	71.64	2.10
llama 7B BF16	16	pp16	33.81	71.72	2.12
llama 7B BF16	16	pp30	34.43	81.93	2.38
llama 7B BF16	16	pp31	34.60	80.99	2.34
llama 7B BF16	16	pp32	34.67	81.60	2.35
llama 7B BF16	16	pp64	35.44	85.54	2.41
llama 7B BF16	16	pp65	34.26	85.67	2.50
llama 7B BF16	16	pp66	36.01	86.00	2.39
llama 7B BF16	16	pp120	36.28	86.63	2.39
llama 7B BF16	16	pp128	32.27	87.02	2.70
llama 7B BF16	16	pp130	35.74	86.62	2.42
llama 7B BF16	16	pp240	33.41	75.09	2.25
llama 7B BF16	16	pp255	33.79	86.10	2.55
llama 7B BF16	16	pp256	33.65	86.45	2.57
llama 7B BF16	16	pp510	32.94	82.24	2.50
llama 7B BF16	16	pp512	34.21	75.92	2.22
llama 7B BF16	16	pp1023	33.01	76.81	2.33
llama 7B BF16	16	pp1024	32.96	77.04	2.34
llama 7B BF16	16	pp1025	33.54	72.84	2.17
llama 7B BF16	16	pp2048	32.17	72.21	2.24
llama 7B BF16	16	tg128	6.64	6.49	0.98
llama 7B BF16	24	pp1	6.45	6.46	1.00
llama 7B BF16	24	pp2	13.87	14.12	1.02
llama 7B BF16	24	pp3	19.54	21.26	1.09
llama 7B BF16	24	pp4	23.84	27.43	1.15
llama 7B BF16	24	pp5	27.16	34.66	1.28
llama 7B BF16	24	pp6	29.39	40.81	1.39
llama 7B BF16	24	pp7	31.82	46.53	1.46
llama 7B BF16	24	pp8	32.90	52.14	1.58
llama 7B BF16	24	pp9	34.22	57.32	1.67
llama 7B BF16	24	pp10	34.64	61.83	1.78
llama 7B BF16	24	pp11	35.80	67.03	1.87
llama 7B BF16	24	pp12	35.61	70.34	1.98
llama 7B BF16	24	pp13	26.88	72.89	2.71
llama 7B BF16	24	pp14	36.83	76.42	2.08
llama 7B BF16	24	pp15	36.29	83.74	2.31
llama 7B BF16	24	pp16	37.80	84.55	2.24
llama 7B BF16	24	pp30	39.96	96.48	2.41
llama 7B BF16	24	pp31	40.38	97.68	2.42
llama 7B BF16	24	pp32	40.44	98.66	2.44
llama 7B BF16	24	pp64	42.22	98.08	2.32
llama 7B BF16	24	pp65	41.12	95.11	2.31
llama 7B BF16	24	pp66	33.80	95.15	2.82
llama 7B BF16	24	pp120	41.40	74.86	1.81
llama 7B BF16	24	pp128	41.64	97.67	2.35
llama 7B BF16	24	pp130	35.86	99.80	2.78
llama 7B BF16	24	pp240	41.78	96.56	2.31
llama 7B BF16	24	pp255	41.48	81.96	1.98
llama 7B BF16	24	pp256	38.66	98.73	2.55
llama 7B BF16	24	pp510	39.75	85.03	2.14
llama 7B BF16	24	pp512	39.44	94.90	2.41
llama 7B BF16	24	pp1023	39.18	88.19	2.25
llama 7B BF16	24	pp1024	38.70	87.89	2.27
llama 7B BF16	24	pp1025	38.99	85.98	2.20
llama 7B BF16	24	pp2048	37.26	83.63	2.24
llama 7B BF16	24	tg128	6.12	6.14	1.00
llama 7B BF16	32	pp1	6.61	6.49	0.98
llama 7B BF16	32	pp2	13.41	12.63	0.94
llama 7B BF16	32	pp3	16.67	19.67	1.18
llama 7B BF16	32	pp4	24.53	26.30	1.07
llama 7B BF16	32	pp5	25.11	30.75	1.22
llama 7B BF16	32	pp6	30.08	35.58	1.18
llama 7B BF16	32	pp7	32.79	44.60	1.36
llama 7B BF16	32	pp8	33.60	47.41	1.41
llama 7B BF16	32	pp9	35.68	49.91	1.40
llama 7B BF16	32	pp10	36.61	61.07	1.67
llama 7B BF16	32	pp11	34.87	60.74	1.74
llama 7B BF16	32	pp12	38.80	58.61	1.51
llama 7B BF16	32	pp13	38.92	67.56	1.74
llama 7B BF16	32	pp14	37.30	71.50	1.92
llama 7B BF16	32	pp15	38.23	52.93	1.38
llama 7B BF16	32	pp16	39.77	77.02	1.94
llama 7B BF16	32	pp30	42.88	93.16	2.17
llama 7B BF16	32	pp31	41.70	92.21	2.21
llama 7B BF16	32	pp32	42.97	96.90	2.26
llama 7B BF16	32	pp64	45.44	106.63	2.35
llama 7B BF16	32	pp65	44.98	105.98	2.36
llama 7B BF16	32	pp66	45.10	105.50	2.34
llama 7B BF16	32	pp120	46.50	104.57	2.25
llama 7B BF16	32	pp128	45.95	104.48	2.27
llama 7B BF16	32	pp130	46.34	101.40	2.19
llama 7B BF16	32	pp240	43.26	87.72	2.03
llama 7B BF16	32	pp255	46.97	109.11	2.32
llama 7B BF16	32	pp256	46.94	104.42	2.22
llama 7B BF16	32	pp510	44.52	92.37	2.07
llama 7B BF16	32	pp512	44.46	92.70	2.08
llama 7B BF16	32	pp1023	43.70	93.70	2.14
llama 7B BF16	32	pp1024	43.72	94.72	2.17
llama 7B BF16	32	pp1025	43.49	93.42	2.15
llama 7B BF16	32	pp2048	42.64	90.61	2.12
llama 7B BF16	32	tg128	5.84	5.89	1.01

Model	Threads	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 7B F16	8	pp1	6.38	7.03	1.10
llama 7B F16	8	pp2	12.49	14.03	1.12
llama 7B F16	8	pp3	19.06	20.65	1.08
llama 7B F16	8	pp4	12.97	25.21	1.94
llama 7B F16	8	pp5	18.14	31.46	1.73
llama 7B F16	8	pp6	28.90	36.29	1.26
llama 7B F16	8	pp7	20.51	39.36	1.92
llama 7B F16	8	pp8	27.07	43.70	1.61
llama 7B F16	8	pp9	39.64	46.82	1.18
llama 7B F16	8	pp10	30.42	49.03	1.61
llama 7B F16	8	pp11	23.51	51.16	2.18
llama 7B F16	8	pp12	44.81	53.55	1.20
llama 7B F16	8	pp13	35.45	53.97	1.52
llama 7B F16	8	pp14	37.40	55.91	1.50
llama 7B F16	8	pp15	47.63	57.52	1.21
llama 7B F16	8	pp16	38.27	56.69	1.48
llama 7B F16	8	pp30	55.66	62.84	1.13
llama 7B F16	8	pp31	46.17	61.79	1.34
llama 7B F16	8	pp32	46.22	62.78	1.36
llama 7B F16	8	pp64	51.43	65.10	1.27
llama 7B F16	8	pp65	51.57	65.04	1.26
llama 7B F16	8	pp66	55.29	50.84	0.92
llama 7B F16	8	pp120	56.77	65.83	1.16
llama 7B F16	8	pp128	55.29	65.64	1.19
llama 7B F16	8	pp130	54.22	66.17	1.22
llama 7B F16	8	pp240	51.06	58.46	1.14
llama 7B F16	8	pp255	56.89	66.11	1.16
llama 7B F16	8	pp256	56.60	58.38	1.03
llama 7B F16	8	pp510	49.56	59.74	1.21
llama 7B F16	8	pp512	48.86	59.75	1.22
llama 7B F16	8	pp1023	50.73	58.39	1.15
llama 7B F16	8	pp1024	46.55	58.51	1.26
llama 7B F16	8	pp1025	50.70	58.01	1.14
llama 7B F16	8	pp2048	48.65	56.76	1.17
llama 7B F16	8	tg128	6.57	6.84	1.04
llama 7B F16	16	pp1	7.36	7.04	0.96
llama 7B F16	16	pp2	11.80	13.98	1.18
llama 7B F16	16	pp3	16.18	20.90	1.29
llama 7B F16	16	pp4	14.60	26.77	1.83
llama 7B F16	16	pp5	17.95	32.82	1.83
llama 7B F16	16	pp6	29.38	38.84	1.32
llama 7B F16	16	pp7	24.25	42.92	1.77
llama 7B F16	16	pp8	26.31	47.11	1.79
llama 7B F16	16	pp9	36.26	51.87	1.43
llama 7B F16	16	pp10	30.21	53.56	1.77
llama 7B F16	16	pp11	31.95	57.11	1.79
llama 7B F16	16	pp12	26.38	60.18	2.28
llama 7B F16	16	pp13	31.82	60.02	1.89
llama 7B F16	16	pp14	35.95	63.33	1.76
llama 7B F16	16	pp15	41.48	65.03	1.57
llama 7B F16	16	pp16	36.87	65.43	1.77
llama 7B F16	16	pp30	46.23	74.36	1.61
llama 7B F16	16	pp31	43.34	73.61	1.70
llama 7B F16	16	pp32	44.02	74.51	1.69
llama 7B F16	16	pp64	44.73	77.52	1.73
llama 7B F16	16	pp65	43.67	77.59	1.78
llama 7B F16	16	pp66	36.00	77.94	2.17
llama 7B F16	16	pp120	46.27	78.65	1.70
llama 7B F16	16	pp128	44.67	64.10	1.43
llama 7B F16	16	pp130	38.55	75.47	1.96
llama 7B F16	16	pp240	47.56	79.61	1.67
llama 7B F16	16	pp255	43.95	67.90	1.54
llama 7B F16	16	pp256	43.84	78.00	1.78
llama 7B F16	16	pp510	47.37	67.67	1.43
llama 7B F16	16	pp512	47.73	69.11	1.45
llama 7B F16	16	pp1023	46.70	69.32	1.48
llama 7B F16	16	pp1024	46.85	69.70	1.49
llama 7B F16	16	pp1025	46.43	69.02	1.49
llama 7B F16	16	pp2048	41.15	64.47	1.57
llama 7B F16	16	tg128	6.77	6.79	1.00
llama 7B F16	24	pp1	6.72	6.70	1.00
llama 7B F16	24	pp2	13.08	13.80	1.06
llama 7B F16	24	pp3	19.12	19.80	1.04
llama 7B F16	24	pp4	17.18	25.52	1.49
llama 7B F16	24	pp5	20.68	31.70	1.53
llama 7B F16	24	pp6	33.86	25.86	0.76
llama 7B F16	24	pp7	20.49	42.41	2.07
llama 7B F16	24	pp8	32.02	49.30	1.54
llama 7B F16	24	pp9	45.35	54.32	1.20
llama 7B F16	24	pp10	38.55	58.23	1.51
llama 7B F16	24	pp11	41.01	62.25	1.52
llama 7B F16	24	pp12	54.11	65.30	1.21
llama 7B F16	24	pp13	45.66	66.65	1.46
llama 7B F16	24	pp14	48.00	68.97	1.44
llama 7B F16	24	pp15	58.72	74.53	1.27
llama 7B F16	24	pp16	50.72	74.72	1.47
llama 7B F16	24	pp30	65.48	84.78	1.29
llama 7B F16	24	pp31	60.81	84.96	1.40
llama 7B F16	24	pp32	61.80	85.63	1.39
llama 7B F16	24	pp64	65.16	85.38	1.31
llama 7B F16	24	pp65	64.48	84.86	1.32
llama 7B F16	24	pp66	66.96	84.38	1.26
llama 7B F16	24	pp120	55.89	85.90	1.54
llama 7B F16	24	pp128	65.76	68.28	1.04
llama 7B F16	24	pp130	65.48	85.78	1.31
llama 7B F16	24	pp240	59.03	87.17	1.48
llama 7B F16	24	pp255	66.37	84.71	1.28
llama 7B F16	24	pp256	65.55	86.56	1.32
llama 7B F16	24	pp510	64.20	74.53	1.16
llama 7B F16	24	pp512	63.80	82.39	1.29
llama 7B F16	24	pp1023	58.59	76.50	1.31
llama 7B F16	24	pp1024	58.57	73.12	1.25
llama 7B F16	24	pp1025	58.36	75.34	1.29
llama 7B F16	24	pp2048	53.68	71.46	1.33
llama 7B F16	24	tg128	6.31	6.50	1.03
llama 7B F16	32	pp1	6.95	6.52	0.94
llama 7B F16	32	pp2	10.93	13.09	1.20
llama 7B F16	32	pp3	20.45	16.78	0.82
llama 7B F16	32	pp4	18.21	25.07	1.38
llama 7B F16	32	pp5	21.85	31.60	1.45
llama 7B F16	32	pp6	37.12	33.09	0.89
llama 7B F16	32	pp7	29.10	43.65	1.50
llama 7B F16	32	pp8	35.45	49.34	1.39
llama 7B F16	32	pp9	45.19	46.98	1.04
llama 7B F16	32	pp10	43.03	58.52	1.36
llama 7B F16	32	pp11	42.45	55.39	1.30
llama 7B F16	32	pp12	61.48	64.59	1.05
llama 7B F16	32	pp13	47.92	67.93	1.42
llama 7B F16	32	pp14	53.72	63.25	1.18
llama 7B F16	32	pp15	62.70	74.37	1.19
llama 7B F16	32	pp16	58.18	65.05	1.12
llama 7B F16	32	pp30	81.05	86.72	1.07
llama 7B F16	32	pp31	70.93	85.87	1.21
llama 7B F16	32	pp32	69.70	86.73	1.24
llama 7B F16	32	pp64	75.62	85.96	1.14
llama 7B F16	32	pp65	80.69	88.52	1.10
llama 7B F16	32	pp66	86.93	88.50	1.02
llama 7B F16	32	pp120	88.89	90.04	1.01
llama 7B F16	32	pp128	85.38	72.80	0.85
llama 7B F16	32	pp130	82.46	88.17	1.07
llama 7B F16	32	pp240	73.02	90.63	1.24
llama 7B F16	32	pp255	88.03	90.93	1.03
llama 7B F16	32	pp256	84.65	78.67	0.93
llama 7B F16	32	pp510	78.80	87.88	1.12
llama 7B F16	32	pp512	73.98	88.27	1.19
llama 7B F16	32	pp1023	71.23	82.14	1.15
llama 7B F16	32	pp1024	74.38	82.08	1.10
llama 7B F16	32	pp1025	73.94	80.76	1.09
llama 7B F16	32	pp2048	65.88	75.73	1.15
llama 7B F16	32	tg128	5.92	5.98	1.01

Model	Threads	Test	t/s master	t/s perfo/tinyblas	Speedup
llama 7B all F32	8	pp1	3.29	3.60	1.10
llama 7B all F32	8	pp2	6.29	7.10	1.13
llama 7B all F32	8	pp3	9.88	10.64	1.08
llama 7B all F32	8	pp4	7.28	13.62	1.87
llama 7B all F32	8	pp5	9.33	16.96	1.82
llama 7B all F32	8	pp6	18.77	20.40	1.09
llama 7B all F32	8	pp7	12.29	22.44	1.83
llama 7B all F32	8	pp8	14.00	17.29	1.24
llama 7B all F32	8	pp9	27.70	27.44	0.99
llama 7B all F32	8	pp10	12.74	28.43	2.23
llama 7B all F32	8	pp11	18.08	31.63	1.75
llama 7B all F32	8	pp12	34.07	36.68	1.08
llama 7B all F32	8	pp13	21.59	37.41	1.73
llama 7B all F32	8	pp14	23.54	41.72	1.77
llama 7B all F32	8	pp15	42.10	45.17	1.07
llama 7B all F32	8	pp16	26.58	45.03	1.69
llama 7B all F32	8	pp30	60.69	68.16	1.12
llama 7B all F32	8	pp31	42.30	66.59	1.57
llama 7B all F32	8	pp32	43.45	70.13	1.61
llama 7B all F32	8	pp64	60.06	81.96	1.36
llama 7B all F32	8	pp65	58.62	83.51	1.42
llama 7B all F32	8	pp66	52.86	80.40	1.52
llama 7B all F32	8	pp120	75.73	66.10	0.87
llama 7B all F32	8	pp128	71.88	79.70	1.11
llama 7B all F32	8	pp130	69.67	84.03	1.21
llama 7B all F32	8	pp240	57.52	76.86	1.34
llama 7B all F32	8	pp255	68.61	63.99	0.93
llama 7B all F32	8	pp256	57.10	76.73	1.34
llama 7B all F32	8	pp510	58.15	63.27	1.09
llama 7B all F32	8	pp512	57.39	65.34	1.14
llama 7B all F32	8	pp1023	56.31	61.57	1.09
llama 7B all F32	8	pp1024	56.42	63.13	1.12
llama 7B all F32	8	pp1025	55.91	64.92	1.16
llama 7B all F32	8	pp2048	54.29	64.00	1.18
llama 7B all F32	8	tg128	3.30	3.30	1.00
llama 7B all F32	16	pp1	3.55	3.57	1.01
llama 7B all F32	16	pp2	5.44	7.26	1.33
llama 7B all F32	16	pp3	8.37	10.87	1.30
llama 7B all F32	16	pp4	6.61	14.35	2.17
llama 7B all F32	16	pp5	7.75	17.88	2.31
llama 7B all F32	16	pp6	15.87	21.44	1.35
llama 7B all F32	16	pp7	11.32	24.51	2.17
llama 7B all F32	16	pp8	12.19	28.14	2.31
llama 7B all F32	16	pp9	22.32	31.75	1.42
llama 7B all F32	16	pp10	15.65	34.38	2.20
llama 7B all F32	16	pp11	12.30	37.65	3.06
llama 7B all F32	16	pp12	28.21	41.11	1.46
llama 7B all F32	16	pp13	19.83	43.12	2.17
llama 7B all F32	16	pp14	20.17	46.27	2.29
llama 7B all F32	16	pp15	33.49	49.29	1.47
llama 7B all F32	16	pp16	23.59	50.51	2.14
llama 7B all F32	16	pp30	48.91	80.46	1.65
llama 7B all F32	16	pp31	37.14	79.76	2.15
llama 7B all F32	16	pp32	36.78	82.64	2.25
llama 7B all F32	16	pp64	52.07	103.75	1.99
llama 7B all F32	16	pp65	39.03	76.48	1.96
llama 7B all F32	16	pp66	57.92	104.47	1.80
llama 7B all F32	16	pp120	63.20	112.86	1.79
llama 7B all F32	16	pp128	58.30	111.95	1.92
llama 7B all F32	16	pp130	57.93	112.51	1.94
llama 7B all F32	16	pp240	58.98	103.05	1.75
llama 7B all F32	16	pp255	62.83	84.99	1.35
llama 7B all F32	16	pp256	60.20	102.69	1.71
llama 7B all F32	16	pp510	58.07	82.11	1.41
llama 7B all F32	16	pp512	56.77	88.96	1.57
llama 7B all F32	16	pp1023	57.33	84.32	1.47
llama 7B all F32	16	pp1024	57.65	83.80	1.45
llama 7B all F32	16	pp1025	54.17	85.64	1.58
llama 7B all F32	16	pp2048	53.54	77.67	1.45
llama 7B all F32	16	tg128	3.38	3.32	0.98
llama 7B all F32	24	pp1	3.57	3.58	1.00
llama 7B all F32	24	pp2	4.44	7.18	1.62
llama 7B all F32	24	pp3	9.34	10.78	1.15
llama 7B all F32	24	pp4	7.36	14.26	1.94
llama 7B all F32	24	pp5	8.86	17.80	2.01
llama 7B all F32	24	pp6	18.73	21.35	1.14
llama 7B all F32	24	pp7	13.51	24.65	1.82
llama 7B all F32	24	pp8	14.72	28.18	1.91
llama 7B all F32	24	pp9	27.34	31.47	1.15
llama 7B all F32	24	pp10	19.02	34.47	1.81
llama 7B all F32	24	pp11	20.65	37.93	1.84
llama 7B all F32	24	pp12	36.08	41.20	1.14
llama 7B all F32	24	pp13	25.13	43.69	1.74
llama 7B all F32	24	pp14	25.49	47.22	1.85
llama 7B all F32	24	pp15	40.66	50.46	1.24
llama 7B all F32	24	pp16	28.48	52.54	1.84
llama 7B all F32	24	pp30	60.87	85.70	1.41
llama 7B all F32	24	pp31	46.29	85.91	1.86
llama 7B all F32	24	pp32	33.68	88.61	2.63
llama 7B all F32	24	pp64	63.22	115.66	1.83
llama 7B all F32	24	pp65	62.47	115.16	1.84
llama 7B all F32	24	pp66	72.74	116.25	1.60
llama 7B all F32	24	pp120	82.94	125.57	1.51
llama 7B all F32	24	pp128	71.71	125.38	1.75
llama 7B all F32	24	pp130	58.06	123.70	2.13
llama 7B all F32	24	pp240	71.65	115.68	1.61
llama 7B all F32	24	pp255	68.59	97.20	1.42
llama 7B all F32	24	pp256	67.18	114.58	1.71
llama 7B all F32	24	pp510	62.15	106.84	1.72
llama 7B all F32	24	pp512	60.77	107.10	1.76
llama 7B all F32	24	pp1023	59.90	94.97	1.59
llama 7B all F32	24	pp1024	59.98	96.93	1.62
llama 7B all F32	24	pp1025	59.06	94.01	1.59
llama 7B all F32	24	pp2048	59.30	91.46	1.54
llama 7B all F32	24	tg128	3.40	3.40	1.00
llama 7B all F32	32	pp1	3.44	3.03	0.88
llama 7B all F32	32	pp2	6.29	5.22	0.83
llama 7B all F32	32	pp3	9.26	10.28	1.11
llama 7B all F32	32	pp4	7.41	14.41	1.94
llama 7B all F32	32	pp5	8.90	16.51	1.85
llama 7B all F32	32	pp6	13.72	20.73	1.51
llama 7B all F32	32	pp7	12.95	25.04	1.93
llama 7B all F32	32	pp8	14.24	26.06	1.83
llama 7B all F32	32	pp9	26.56	29.83	1.12
llama 7B all F32	32	pp10	18.28	33.25	1.82
llama 7B all F32	32	pp11	19.46	33.97	1.75
llama 7B all F32	32	pp12	35.39	38.65	1.09
llama 7B all F32	32	pp13	23.80	42.63	1.79
llama 7B all F32	32	pp14	25.25	42.34	1.68
llama 7B all F32	32	pp15	44.43	43.85	0.99
llama 7B all F32	32	pp16	27.65	33.80	1.22
llama 7B all F32	32	pp30	62.07	76.51	1.23
llama 7B all F32	32	pp31	48.28	79.72	1.65
llama 7B all F32	32	pp32	50.50	83.07	1.64
llama 7B all F32	32	pp64	62.50	114.16	1.83
llama 7B all F32	32	pp65	72.63	119.77	1.65
llama 7B all F32	32	pp66	89.07	121.06	1.36
llama 7B all F32	32	pp120	91.80	131.97	1.44
llama 7B all F32	32	pp128	84.08	130.04	1.55
llama 7B all F32	32	pp130	84.99	128.60	1.51
llama 7B all F32	32	pp240	75.48	123.00	1.63
llama 7B all F32	32	pp255	82.80	125.00	1.51
llama 7B all F32	32	pp256	80.86	125.54	1.55
llama 7B all F32	32	pp510	79.07	103.85	1.31
llama 7B all F32	32	pp512	72.10	118.05	1.64
llama 7B all F32	32	pp1023	69.98	102.26	1.46
llama 7B all F32	32	pp1024	73.17	112.25	1.53
llama 7B all F32	32	pp1025	71.53	102.11	1.43
llama 7B all F32	32	pp2048	68.90	98.77	1.43
llama 7B all F32	32	tg128	3.13	3.16	1.01

more perfo with llamafile tinyblas on x86_64. #10714

more perfo with llamafile tinyblas on x86_64. #10714

Conversation

Djip007 commented Dec 8, 2024 • edited Loading

Djip007 commented Dec 10, 2024

Djip007 commented Dec 10, 2024

slaren commented Dec 10, 2024

Djip007 commented Dec 11, 2024 • edited Loading

slaren commented Dec 11, 2024

Djip007 commented Dec 11, 2024

ggerganov commented Dec 11, 2024

Djip007 commented Dec 11, 2024

Djip007 commented Dec 11, 2024

slaren left a comment

Choose a reason for hiding this comment

Djip007 commented Dec 11, 2024

ggerganov commented Dec 11, 2024

slaren commented Dec 11, 2024

Djip007 commented Dec 11, 2024

slaren commented Dec 11, 2024

Djip007 commented Dec 11, 2024 • edited Loading

slaren commented Dec 11, 2024

Djip007 Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Djip007 Dec 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Djip007 commented Dec 24, 2024

Djip007 commented Dec 24, 2024

Djip007 commented Dec 24, 2024

Djip007 commented Dec 24, 2024

Djip007 commented Dec 8, 2024 •

edited

Loading

Djip007 commented Dec 11, 2024 •

edited

Loading

Djip007 commented Dec 11, 2024 •

edited

Loading

Djip007 Dec 11, 2024 •

edited

Loading

Djip007 Dec 15, 2024 •

edited

Loading