Precision Issues with GPTQ-Quantized Qwen2.5-VL Model

[KarlDe1]

After applying GPTQ quantization to the Qwen2.5-VL model and running it on the test set, I observed a drop in accuracy. Are there any methods to help identify or locate the source of this accuracy degradation?

In this situation, should we be trying to locate where exactly the accuracy issue is introduced, or is this kind of degradation expected/normal after quantization?

Environment
Include all relevant environment information:

1. Ubuntu22.04
2. Python version:3.10.16
3. LLM Compressor version: v0.6.0
4. torch version: 2.6.0+cu124
5. vLLM version: 0.8.5
6. CUDA version: 12.8
7. transformers version: 4.52.4
8. GPU: nvidia H20

recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W4A16",
        sequential_targets=["Qwen2_5_VLDecoderLayer"],
        ignore=["lm_head", "re:visual.*"],
    ),
]

[KarlDe1]

My prompt provides an image and asks the model to make a driving decision. After testing on the dataset, I found that in 3% of the cases, the quantized model made incorrect decisions, whereas the original (non-quantized) model made the correct choices.

[kylesayrs]

Hi @KarlDe1

You can see the RedhatAI official models uploaded here. Each of these model cards shows the measured accuracy as well as performance expectations alongside each scheme.

Hopefully this helps!

[KarlDe1]

Hi @kylesayrs

Thank you for your reply. Is it possible to recover some of the accuracy through certain methods? For example, by ignoring more layers during quantization?

[KarlDe1]

Hi @kylesayrs

Also, when I was quantizing the model using the script provided here by llm-compressor-v0.6.0, I noticed the following log. I believe the intention was to exclude the visual-related layers from quantization, so perhaps the pattern should be updated to "re:model.visual.*"?