Can I quantize Qwen2-Audio-7B-Instruct using GPTQModel

[JeremyGe07]

Do you know how can I quantize Qwen2-Audio-7B-Instruct, and how can I run this quanztied model

[Qubitium]

Do you know how can I quantize Qwen2-Audio-7B-Instruct, and how can I run this quanztied model

It should work, gptqmodel qunatization happen at the lowest model depth and does not care if the tensors represent text, image, or audio.

The problem is integration and getting all the special tokenization/preprocessing to work.

If there is demand, we may add support for it. To start look at how we do the Qwen VL or OVIS model and use that as foundation.