NVMLError_NotSupported: Not Supported

[JIA-HONG-CHU]

I tried your example code in gemma3.

But I encounted

---

NVMLError_NotSupported Traceback (most recent call last)
Cell In[1], line 42
29 recipe = [
30 GPTQModifier(
31 targets="Linear",
(...)
38 ),
39 ]
41 # Perform oneshot
---> 42 oneshot(
43 model=model,
44 tokenizer=model_id,
45 dataset=DATASET_ID,
46 splits=DATASET_SPLIT,
47 recipe=recipe,
48 max_seq_length=MAX_SEQUENCE_LENGTH,
49 num_calibration_samples=NUM_CALIBRATION_SAMPLES,
50 trust_remote_code_model=True,
51 data_collator=data_collator,
52 )
54 # Confirm generations of the quantized model look sane.
55 print("========== SAMPLE GENERATION ==============")

I used AGX Orin 64GB.
llmcompressor 0.6.0

[brian-dellabetta]

Hi @JIA-HONG-CHU , can you provide the full stack trace? Where in oneshot is it failing? This indicates the GPU you are using doesn't support some operation, but it isn't shown in the logs you've provided. Do any of the other examples work on your GPU? It has compute capability 8.7, so I would expect it to work

[JIA-HONG-CHU]

I modified the get_GPU_usage_nv in metric_logging.py to

def get_GPU_usage_nv() -> List[Tuple[float, float]]:
"""
get gpu usage for Nvidia GPUs using nvml lib
"""
return []
and it works.

But i encounted the other issue

---

ValueError                                Traceback (most recent call last)
Cell In[1], line 72
     70 # Note: compile is disabled: https://github.com/huggingface/transformers/issues/38333
     71 inputs = processor(images=raw_image, text=prompt, return_tensors="pt").to("cuda")
---> 72 output = model.generate(**inputs, max_new_tokens=100, disable_compile=True)
     73 print(processor.decode(output[0], skip_special_tokens=True))
     74 print("==========================================")

File /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:116, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    113 @functools.wraps(func)
    114 def decorate_context(*args, **kwargs):
    115     with ctx_factory():
--> 116         return func(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:2493, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, use_model_defaults, **kwargs)
   2485     input_ids, model_kwargs = self._expand_inputs_for_generation(
   2486         input_ids=input_ids,
   2487         expand_size=generation_config.num_return_sequences,
   2488         is_encoder_decoder=self.config.is_encoder_decoder,
   2489         **model_kwargs,
   2490     )
   2492     # 12. run sample (it degenerates to greedy search when `generation_config.do_sample=False`)
-> 2493     result = self._sample(
   2494         input_ids,
   2495         logits_processor=prepared_logits_processor,
   2496         stopping_criteria=prepared_stopping_criteria,
   2497         generation_config=generation_config,
   2498         synced_gpus=synced_gpus,
   2499         streamer=streamer,
   2500         **model_kwargs,
   2501     )
   2503 elif generation_mode in (GenerationMode.BEAM_SAMPLE, GenerationMode.BEAM_SEARCH):
   2504     # 11. interleave input_ids with `num_beams` additional sequences per batch
   2505     input_ids, model_kwargs = self._expand_inputs_for_generation(
   2506         input_ids=input_ids,
   2507         expand_size=generation_config.num_beams,
   2508         is_encoder_decoder=self.config.is_encoder_decoder,
   2509         **model_kwargs,
   2510     )

File /usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:3452, in GenerationMixin._sample(self, input_ids, logits_processor, stopping_criteria, generation_config, synced_gpus, streamer, **model_kwargs)
   3449 model_inputs.update({"output_hidden_states": output_hidden_states} if output_hidden_states else {})
   3451 if is_prefill:
-> 3452     outputs = self(**model_inputs, return_dict=True)
   3453     is_prefill = False
   3454 else:

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:175, in add_hook_to_module.<locals>.new_forward(module, *args, **kwargs)
    173         output = module._old_forward(*args, **kwargs)
    174 else:
--> 175     output = module._old_forward(*args, **kwargs)
    176 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:969, in can_return_tuple.<locals>.wrapper(self, *args, **kwargs)
    966     set_attribute_for_modules(self, "_is_top_level_module", False)
    968 try:
--> 969     output = func(self, *args, **kwargs)
    970     if is_requested_to_return_tuple or (is_configured_to_return_tuple and is_top_level_module):
    971         output = output.to_tuple()

File /usr/local/lib/python3.10/dist-packages/transformers/models/gemma3/modeling_gemma3.py:1295, in Gemma3ForConditionalGeneration.forward(self, input_ids, pixel_values, attention_mask, position_ids, past_key_values, token_type_ids, cache_position, inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, logits_to_keep, **lm_kwargs)
   1293 # Merge text and images
   1294 if pixel_values is not None:
-> 1295     image_features = self.get_image_features(pixel_values)
   1297     if input_ids is None:
   1298         special_image_mask = inputs_embeds == self.get_input_embeddings()(
   1299             torch.tensor(self.config.image_token_id, dtype=torch.long, device=inputs_embeds.device)
   1300         )

File /usr/local/lib/python3.10/dist-packages/transformers/models/gemma3/modeling_gemma3.py:1188, in Gemma3ForConditionalGeneration.get_image_features(self, pixel_values)
   1178 def get_image_features(self, pixel_values: torch.Tensor) -> torch.Tensor:
   1179     """
   1180     Projects the last hidden state from the vision model into language model space.
   1181 
   (...)
   1186         image_features (`torch.Tensor`): Image feature tensor of shape `(num_images, image_length, embed_dim)`).
   1187     """
-> 1188     vision_outputs = self.vision_tower(pixel_values=pixel_values).last_hidden_state
   1189     image_features = self.multi_modal_projector(vision_outputs)
   1190     return image_features

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:175, in add_hook_to_module.<locals>.new_forward(module, *args, **kwargs)
    173         output = module._old_forward(*args, **kwargs)
    174 else:
--> 175     output = module._old_forward(*args, **kwargs)
    176 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:969, in can_return_tuple.<locals>.wrapper(self, *args, **kwargs)
    966     set_attribute_for_modules(self, "_is_top_level_module", False)
    968 try:
--> 969     output = func(self, *args, **kwargs)
    970     if is_requested_to_return_tuple or (is_configured_to_return_tuple and is_top_level_module):
    971         output = output.to_tuple()

File /usr/local/lib/python3.10/dist-packages/transformers/models/siglip/modeling_siglip.py:1024, in SiglipVisionModel.forward(self, pixel_values, output_attentions, output_hidden_states, interpolate_pos_encoding)
    991 @can_return_tuple
    992 @add_start_docstrings_to_model_forward(SIGLIP_VISION_INPUTS_DOCSTRING)
    993 @replace_return_docstrings(output_type=BaseModelOutputWithPooling, config_class=SiglipVisionConfig)
   (...)
    999     interpolate_pos_encoding: bool = False,
   1000 ) -> BaseModelOutputWithPooling:
   1001     r"""
   1002     Returns:
   1003 
   (...)
   1021     >>> pooled_output = outputs.pooler_output  # pooled features
   1022     ```"""
-> 1024     return self.vision_model(
   1025         pixel_values=pixel_values,
   1026         output_attentions=output_attentions,
   1027         output_hidden_states=output_hidden_states,
   1028         interpolate_pos_encoding=interpolate_pos_encoding,
   1029     )

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:175, in add_hook_to_module.<locals>.new_forward(module, *args, **kwargs)
    173         output = module._old_forward(*args, **kwargs)
    174 else:
--> 175     output = module._old_forward(*args, **kwargs)
    176 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:969, in can_return_tuple.<locals>.wrapper(self, *args, **kwargs)
    966     set_attribute_for_modules(self, "_is_top_level_module", False)
    968 try:
--> 969     output = func(self, *args, **kwargs)
    970     if is_requested_to_return_tuple or (is_configured_to_return_tuple and is_top_level_module):
    971         output = output.to_tuple()

File /usr/local/lib/python3.10/dist-packages/transformers/models/siglip/modeling_siglip.py:929, in SiglipVisionTransformer.forward(self, pixel_values, output_attentions, output_hidden_states, interpolate_pos_encoding)
    923 output_hidden_states = (
    924     output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
    925 )
    927 hidden_states = self.embeddings(pixel_values, interpolate_pos_encoding=interpolate_pos_encoding)
--> 929 encoder_outputs: BaseModelOutput = self.encoder(
    930     inputs_embeds=hidden_states,
    931     output_attentions=output_attentions,
    932     output_hidden_states=output_hidden_states,
    933 )
    935 last_hidden_state = encoder_outputs.last_hidden_state
    936 last_hidden_state = self.post_layernorm(last_hidden_state)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:175, in add_hook_to_module.<locals>.new_forward(module, *args, **kwargs)
    173         output = module._old_forward(*args, **kwargs)
    174 else:
--> 175     output = module._old_forward(*args, **kwargs)
    176 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/transformers/utils/generic.py:969, in can_return_tuple.<locals>.wrapper(self, *args, **kwargs)
    966     set_attribute_for_modules(self, "_is_top_level_module", False)
    968 try:
--> 969     output = func(self, *args, **kwargs)
    970     if is_requested_to_return_tuple or (is_configured_to_return_tuple and is_top_level_module):
    971         output = output.to_tuple()

File /usr/local/lib/python3.10/dist-packages/transformers/models/siglip/modeling_siglip.py:748, in SiglipEncoder.forward(self, inputs_embeds, attention_mask, output_attentions, output_hidden_states)
    745 if output_hidden_states:
    746     encoder_states = encoder_states + (hidden_states,)
--> 748 layer_outputs = encoder_layer(
    749     hidden_states,
    750     attention_mask,
    751     output_attentions=output_attentions,
    752 )
    754 hidden_states = layer_outputs[0]
    756 if output_attentions:

File /usr/local/lib/python3.10/dist-packages/transformers/modeling_layers.py:48, in GradientCheckpointingLayer.__call__(self, *args, **kwargs)
     46 if self.gradient_checkpointing and self.training:
     47     return self._gradient_checkpointing_func(partial(super().__call__, **kwargs), *args)
---> 48 return super().__call__(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:175, in add_hook_to_module.<locals>.new_forward(module, *args, **kwargs)
    173         output = module._old_forward(*args, **kwargs)
    174 else:
--> 175     output = module._old_forward(*args, **kwargs)
    176 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/transformers/models/siglip/modeling_siglip.py:506, in SiglipEncoderLayer.forward(self, hidden_states, attention_mask, output_attentions)
    504 residual = hidden_states
    505 hidden_states = self.layer_norm2(hidden_states)
--> 506 hidden_states = self.mlp(hidden_states)
    507 hidden_states = residual + hidden_states
    509 outputs = (hidden_states,)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:175, in add_hook_to_module.<locals>.new_forward(module, *args, **kwargs)
    173         output = module._old_forward(*args, **kwargs)
    174 else:
--> 175     output = module._old_forward(*args, **kwargs)
    176 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/transformers/models/siglip/modeling_siglip.py:465, in SiglipMLP.forward(self, hidden_states)
    463 hidden_states = self.fc1(hidden_states)
    464 hidden_states = self.activation_fn(hidden_states)
--> 465 hidden_states = self.fc2(hidden_states)
    466 return hidden_states

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1751, in Module._wrapped_call_impl(self, *args, **kwargs)
   1749     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1750 else:
-> 1751     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1762, in Module._call_impl(self, *args, **kwargs)
   1757 # If we don't have any hooks, we want to skip the rest of the logic in
   1758 # this function, and just call forward.
   1759 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1760         or _global_backward_pre_hooks or _global_backward_hooks
   1761         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1762     return forward_call(*args, **kwargs)
   1764 result = None
   1765 called_always_called_hooks = set()

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:175, in add_hook_to_module.<locals>.new_forward(module, *args, **kwargs)
    173         output = module._old_forward(*args, **kwargs)
    174 else:
--> 175     output = module._old_forward(*args, **kwargs)
    176 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/compressed_tensors/quantization/lifecycle/forward.py:321, in wrap_module_forward_quantized.<locals>.wrapped_forward(self, *args, **kwargs)
    318 if scheme.weights is not None and not compressed:
    319     # calibrate and (fake) quantize weights when applicable
    320     unquantized_weight = self.weight.data.clone()
--> 321     self.weight.data = forward_quantize(
    322         module, self.weight, "weight", scheme.weights
    323     )
    325 # perform wrapped forward call
    326 output = forward_func_orig.__get__(module, module.__class__)(
    327     input_, *args[1:], **kwargs
    328 )

File /usr/local/lib/python3.10/dist-packages/compressed_tensors/quantization/lifecycle/forward.py:383, in forward_quantize(module, value, base_name, args)
    380     scale = getattr(module, f"{base_name}_scale")
    381     zero_point = getattr(module, f"{base_name}_zero_point", None)
--> 383 return fake_quantize(
    384     x=value,
    385     scale=scale,
    386     zero_point=zero_point,
    387     args=args,
    388     g_idx=g_idx,
    389     global_scale=global_scale,
    390 )

File /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:116, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    113 @functools.wraps(func)
    114 def decorate_context(*args, **kwargs):
    115     with ctx_factory():
--> 116         return func(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/compressed_tensors/quantization/lifecycle/forward.py:165, in fake_quantize(x, scale, zero_point, args, g_idx, global_scale)
    141 @torch.no_grad()
    142 def fake_quantize(
    143     x: torch.Tensor,
   (...)
    148     global_scale: Optional[torch.Tensor] = None,
    149 ) -> torch.Tensor:
    150     """
    151     Fake quantize the input tensor x by quantizing then dequantizing with
    152     the QuantizationStrategy specified in args. Quantization can be done per tensor,
   (...)
    163     :return: fake quantized tensor
    164     """
--> 165     return _process_quantization(
    166         x=x,
    167         scale=scale,
    168         zero_point=zero_point,
    169         args=args,
    170         do_quantize=True,
    171         do_dequantize=True,
    172         g_idx=g_idx,
    173         global_scale=global_scale,
    174     )

File /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:116, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    113 @functools.wraps(func)
    114 def decorate_context(*args, **kwargs):
    115     with ctx_factory():
--> 116         return func(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/compressed_tensors/quantization/lifecycle/forward.py:210, in _process_quantization(x, scale, zero_point, args, g_idx, dtype, do_quantize, do_dequantize, global_scale)
    208 if columns >= group_size:
    209     if columns % group_size != 0:
--> 210         raise ValueError(
    211             "tensor column shape must be divisble "
    212             f"by the given group_size {group_size}"
    213         )
    215 # support column-order (default) quantization as well as other orderings
    216 # such as activation ordering. Below checks if g_idx has been initialized
    217 is_column_order = g_idx is None or -1 in g_idx

ValueError: tensor column shape must be divisble by the given group_size 128

[brian-dellabetta]

Hi @JIA-HONG-CHU , I am able to run examples/multimodal_vision/gemma3_example.py without issue on Python 3.10. Can you confirm you are on

• latest llm-compressor (0.6.0)
• latest compressed-tensors (0.10.2)
• transformers v4.52.4 or higher. They had some changes to model defs recently
  ?

I've uploaded the resulting checkpoint to https://huggingface.co/nm-testing/gemma-3-4b-it-W4A16-G128/tree/main. If updating dependency versions doesn't resolve your issue, can you try downloading the model and running the generation on it, to see if it works?