[Question] How was the llama fp8 mlir generated?

[rednoah91]

Hi @sogartar

Could you teach me how to generate this fp8 mlir which only matmul is fp8 and other ops keep in f32?
https://sharkpublic.blob.core.windows.net/sharkpublic/dan/fp8_prefill.mlir
I can not find related doc describes where is this mlir be generated. Maybe you can point me to the doc if any : )

Thanks
Hong-Rong

[sogartar]

I don't know how this specific MLIR got generated.
I don't think there is a doc sadly.
There is a CI job exporting the MLIR. E.g. Llama 3.1 8B instruct.
You will have to follow the bread crumbs to see what the job actually does.
Namely, how it calls the actual export script.

You will need access to the quantized model weights in Azure. They are not public. They can be found here.