VLLM serve VLM quantized Gemma3 KeyError: 'vision_model.encoder.layers.0.self_attn.qkv_proj.weight'

[giangntapero]

Describe the bug
Quantize Gemma3 follow example at: https://github.com/vllm-project/llm-compressor/blob/main/examples/multimodal_vision/gemma3_example.py

After that, serving model using vllm command:
vllm serve /gemma-3-4b-it-W4A16-G128 --served-model-name google/gemma-3-4b-it --max-model-len 8192 --limit-mm-per-prompt image=1 --gpu-memory-utilization 0.9

Errors
Process EngineCore_0:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 519, in run_engine_core
raise e
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 506, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 390, in init
super().init(vllm_config, executor_class, log_stats,
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/engine/core.py", line 76, in init
self.model_executor = executor_class(vllm_config)
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/executor_base.py", line 53, in init
self._init_executor()
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 48, in _init_executor
self.collective_rpc("load_model")
File "/usr/local/lib/python3.10/dist-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/utils.py", line 2671, in run_method
return func(*args, **kwargs)
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_worker.py", line 180, in load_model
self.model_runner.load_model()
File "/usr/local/lib/python3.10/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1601, in load_model
self.model = model_loader.load_model(
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/base_loader.py", line 41, in load_model
self.load_weights(model, model_config)
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/model_loader/default_loader.py", line 269, in load_weights
loaded_weights = model.load_weights(
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/gemma3_mm.py", line 700, in load_weights
return loader.load_weights(weights)
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 278, in load_weights
autoloaded_weights = set(self._load_module("", self.module, weights))
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 236, in _load_module
yield from self._load_module(prefix,
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/utils.py", line 209, in _load_module
loaded_params = module_load_weights(weights)
File "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/models/siglip.py", line 514, in load_weights
param = params_dict[name]
KeyError: 'vision_model.encoder.layers.0.self_attn.qkv_proj.weight'

[Flink-ddd]

Thanks for the detailed bug report. I've successfully reproduced this KeyError on a cloud GPU environment. It seems to be a naming mismatch for the vision encoder's weights after quantization.

I'm going to start by inspecting the weight keys in the quantized .safetensors file and will look into patching the model loading logic. I'll submit a PR once I have a solution.

[Flink-ddd]

Hi @giangntapero I already submitted pr in vllm project, but it need wait maintainer further feedback. obviously, this issue si not caused by vllm-compressor.

[brian-dellabetta]

Thanks @Flink-ddd for the response and vllm PR to resolve!

[Flink-ddd]

You're very welcome, @brian-dellabetta . Thanks for the confirmation.

The corresponding Pull Request has been submitted to the vllm-project/vllm repository and is now awaiting maintainer review and the final CI checks. For anyone following this issue, all future updates regarding the fix will be on that PR.

[giangntapero]

@Flink-ddd Thanks for your great work. I tried to follow up your pull request on vllm-project/vllm but now, after quantize gemma3 using code sample from vllm-project/llm-compressor and serving it with vllm, I encounter a different error:

Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/shared/vllm/vllm/v1/engine/core.py", line 523, in run_engine_core
raise e
File "/shared/vllm/vllm/v1/engine/core.py", line 510, in run_engine_core
engine_core = EngineCoreProc(*args, **kwargs)
File "/shared/vllm/vllm/v1/engine/core.py", line 394, in init
super().init(vllm_config, executor_class, log_stats,
File "/shared/vllm/vllm/v1/engine/core.py", line 75, in init
self.model_executor = executor_class(vllm_config)
File "/shared/vllm/vllm/executor/executor_base.py", line 53, in init
self._init_executor()
File "/shared/vllm/vllm/executor/uniproc_executor.py", line 48, in _init_executor
self.collective_rpc("load_model")
File "/shared/vllm/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
answer = run_method(self.driver_worker, method, args, kwargs)
File "/shared/vllm/vllm/utils.py", line 2687, in run_method
return func(*args, **kwargs)
File "/shared/vllm/vllm/v1/worker/gpu_worker.py", line 185, in load_model
self.model_runner.load_model()
File "/shared/vllm/vllm/v1/worker/gpu_model_runner.py", line 1702, in load_model
self.model = model_loader.load_model(
File "/shared/vllm/vllm/model_executor/model_loader/base_loader.py", line 41, in load_model
self.load_weights(model, model_config)
File "/shared/vllm/vllm/model_executor/model_loader/default_loader.py", line 269, in load_weights
loaded_weights = model.load_weights(
File "/shared/vllm/vllm/model_executor/models/gemma3_mm.py", line 712, in load_weights
return loader.load_weights(weights, mapper=self.hf_to_vllm_mapper)
File "/shared/vllm/vllm/model_executor/models/utils.py", line 278, in load_weights
autoloaded_weights = set(self._load_module("", self.module, weights))
File "/shared/vllm/vllm/model_executor/models/utils.py", line 264, in _load_module
raise ValueError(msg)
ValueError: There is no module or parameter named 'vision_model' in Gemma3ForConditionalGeneration

---

I tried to build vllm and transformers from source.

My Env:
llmcompressor==0.5.1
vllm==0.9.2.dev205+g493c27535.cu122
transformers==4.52.0

Please kindly advise. Thank you.

[Flink-ddd]

Hi @giangntapero , thank you very much for testing and providing the new traceback. This is extremely helpful!

Based on the new ValueError, I believe the true root cause is inside siglip.py. I have prepared a new potential fix on a separate branch.

but I will tomorrow handle to this issue, when I push code to 'fix-gemma3-keyerror-siglip-test' branch, I will let you know.

at that ttime, Would you be willing to help test it one more time? You would need to use this specific branch from my fork: Flink-ddd/vllm#fix-gemma3-keyerror-siglip-test.

[Flink-ddd]

Hi @giangntapero , Thank you very much for your patience and for providing the new traceback. That ValueError was the final, crucial clue!

Based on that error, I have developed a new, much more robust fix. I was able to test this new solution in a cloud GPU environment, and I can confirm that the loading process now successfully gets past the stage where the ValueError: There is no module or parameter named 'vision_model' occurred.

If you have a moment, would you be willing to help test this definitive fix? To do so, you would need to install vLLM from my specific branch. The easiest way is to use this pip command:

pip install git+https://github.com/Flink-ddd/vllm.git@fix-gemma3-final-v2

Your confirmation would be the final step needed before I submit the new, finalized Pull Request to the vLLM.

Thank you again for your invaluable help in debugging this complex issue!

[Flink-ddd]

Hi @giangntapero , you can use my specific branch to verify, I already submitted new PR, let's wait to maintainer advise or what happen next step. Below are some screenshots of me using the google/gemma-3-4b model to verify the GPU environment.

It seems that the error you gave before will no longer appear thatValueError: There is no module or parameter named 'vision_model' ...