The decode stage of Qwen2.5-VL did not perform as well as expected after quantization of GPTQ-W4A16.

[KarlDe1]

Describe the bug
I have quantized Qwen2.5-VL into W4A16 with GPTQ method and compared the time cost during the decode stage.
The first image below shows the time taken to decode a single token using the unquantized version of Qwen2.5-VL-3B-Instruct. The second image shows the time taken to decode a single token after quantization of W4A16. Surprisingly, the time taken after quantization seems even longer than before quantization.

related issue: #1591

Expected behavior
Since the decode phase is typically memory-bound, I would expect quantization to improve performance by reducing memory bandwidth usage. However, the latency of the quantized model appears similar to that of the unquantized one. Why is that the case? Shouldn’t it be faster with quantization?

Environment
Include all relevant environment information:

1. Ubuntu22.04
2. Python version:3.10.16
3. LLM Compressor version: v0.6.0
4. torch version: 2.6.0+cu124
5. vLLM version: 0.8.5
6. CUDA version: 12.8
7. transformers version: 4.52.4
8. GPU: nvidia H20

[KarlDe1]

#1600 (comment)
@brian-dellabetta THX for your reply, I create a new issue for this problem.

[brian-dellabetta]

Hi @KarlDe1 , I can ask vllm maintainers if they have any thoughts. Couple clarifying questions before that:

• Have you seen this is the case for language-only models as well?
• Can you provide the recipe you used for the GPTQ-quanted model?

[KarlDe1]

Hi @brian-dellabetta ,

Understood. I'll share more details as follows:

Have you seen this is the case for language-only models as well?

I haven't tried any other models.

Can you provide the recipe you used for the GPTQ-quanted model?

recipe = [
    GPTQModifier(
        targets="Linear",
        scheme="W4A16",
        sequential_targets=["Qwen2_5_VLDecoderLayer"],
        ignore=["lm_head", "re:visual.*"],
    ),
]

I just quantized model by examples/multimodal_vision/qwen_2_5_vl_example.py.

[brian-dellabetta]

Hi @KarlDe1 , I think you are right in thinking that latency should be lower for the W4A16 model (vllm docs here), but I wonder if checking time to decode a single token is a valid metric for vision language models. I asked around, and have a few recommendations to help troubleshoot:

• Confirm vllm infers it can use an optimized kernel. For example, do you see INFO 06-30 21:31:05 [compressed_tensors_wNa16.py:95] Using MarlinLinearKernel for CompressedTensorsWNA16 in your logs? An H20 should have the compute capability to run the Marlin Kernel.
• Try a language-only model on your benchmark, for example any of those available here against their fp16 original version. Do you see the same thing?
• Try your original and W4A16 models on an lm_eval benchmark (example here), to see if throughput is drastically different. for VL models, you can use --model "vllm-vlm" and --tasks "mmmu_val", which is a multimodal set. It will show time taken in the progress bar
• Try one of the vllm latency benchmarks -- https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_latency.py, which factor in multiple passes, varying batch sizes and lengths, and warm-up.

If we are still unable to see speedup, I can raise with the broader team. But this is one of the bread and butter flows, so I'd like to confirm these points first.