llamafile_sgemm API - INT8 implementation

[amritahs-ibm]

This change upstreams llamafile's cpu matrix
multiplication kernels for ppc64le using MMA
builtins for quantised int8 datatype.

This change results in 10%-70% improvement
in total speed(ie all tokens/total time), across
various batch sizes.

The patch is tested with Meta-Lllama-3-8B,
Mistral-7B, Llama-2-7B-chat-hf models on a
IBM POWER10 machine.

gcc-toolset-13 is minimum requirement to build
this patch. Run cmake config and build after path
has been updated with gcc-13:
export PATH=/opt/rh/gcc-toolset-13/root/usr/bin/:$PATH

[amritahs-ibm]

Hi @ggerganov,
Can you please help reviewing this PR. Or suggest any missing actions required from me to get this patch reviewed.

[slaren]

We will need to merge #10714 first, since there may be some conflicts.

[amritahs-ibm]

Sure. Please let me know once that PR is merged. I will fix mine, if any conflicts and resubmit my PR.

[Djip007]

I'll try to submit it today. 🤞

[amritahs-ibm]

I have made the changes suggested by @Djip007 and pushed. @slaren / @Djip007 / @ggerganov Please review the changes.

[Djip007]

These are quick personal comments wait for @slaren @ggerganov before change.
And I read it very quickly.

[Djip007]

@amritahs-ibm
I didn't realize that MMA was "Matrix Multiply Accelerate".
Have you see what was done with amx or "aarch64" kernel? There is now "simple" arch to create kernel that can "repack" the weight, so it fit to the structure needed for such Matrix OP?
That way A is repack at load time once, and only B need to be repack at runtime.

[amritahs-ibm]

All comments are addressed expect for the last MMA one. The updated patch has been committed.
I will look into MMA comment and get back to you.

[amritahs-ibm]

@amritahs-ibm I didn't realize that MMA was "Matrix Multiply Accelerate". Have you see what was done with amx or "aarch64" kernel? There is now "simple" arch to create kernel that can "repack" the weight, so it fit to the structure needed for such Matrix OP? That way A is repack at load time once, and only B need to be repack at runtime.

@Djip007
Are you referring to gemm4xN and gemmMx4 functions in tinyBLAS_Q0_AVX?

Also in case of PowerPC's MMA for int8 data type, MMA engine requires the data to be packed in a different way. So I came up with a specific function for int8(ie packNormal) to do the packing.

Please find below the MMA guide:
https://www.redbooks.ibm.com/redpapers/pdfs/redp5612.pdf

[amritahs-ibm]

@Djip007 Could you please help me on my latest comment?

[amritahs-ibm]

I have addressed all the comments except for the MMA repacking one. As it is an optimization over the current code, I can take it up in the follow on patch.
@slaren @Djip007 @ggerganov
Can anyone of you approve the patch and get it merged?