The size of tensor a (1290) must match the size of tensor b (778) at non-singleton dimension 3

[mrpeerat]

Hi,

I trained my model using gemma2, and the pre-training and fine-tuning went well (the training loss decreased and look normal). But when I tried to inference using the code below:

from tinyllava.eval.run_tiny_llava import eval_model

model_path = "my model path"
prompt = "What are the things I should be cautious about when I visit here?"
image_file = "https://llava-vl.github.io/static/images/view.jpg"
conv_mode = "gemma" # or llama, gemma, etc

args = type('Args', (), {
    "model_path": model_path,
    "model": None,
    "query": prompt,
    "conv_mode": conv_mode,
    "image_file": image_file,
    "sep": ",",
    "temperature": 0,
    "top_p": None,
    "num_beams": 1,
    "max_new_tokens": 512
})()

print(eval_model(args))

I got this error:

--- Logging error ---
Traceback (most recent call last):
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/logging/__init__.py", line 1100, in emit
    msg = self.format(record)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/logging/__init__.py", line 943, in format
    return fmt.format(record)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/logging/__init__.py", line 678, in format
    record.message = record.getMessage()
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/logging/__init__.py", line 368, in getMessage
    msg = msg % self.args
TypeError: not all arguments converted during string formatting
Call stack:
  File "/shared/aisingapore/peerat/TinyLLaVA_Factory/test.py", line 21, in <module>
    print(eval_model(args))
  File "/shared/aisingapore/peerat/TinyLLaVA_Factory/tinyllava/eval/run_tiny_llava.py", line 42, in eval_model
    model, tokenizer, image_processor, context_len = load_pretrained_model(args.model_path)
  File "/shared/aisingapore/peerat/TinyLLaVA_Factory/tinyllava/model/load_model.py", line 37, in load_pretrained_model
    model = TinyLlavaForConditionalGeneration.from_pretrained(model_name_or_path,low_cpu_mem_usage=True,torch_dtype=torch.float16)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/modeling_utils.py", line 4296, in from_pretrained
    model.generation_config = GenerationConfig.from_pretrained(
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/generation/configuration_utils.py", line 1102, in from_pretrained
    config = cls.from_dict(config_dict, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/generation/configuration_utils.py", line 1137, in from_dict
    config = cls(**{**config_dict, **kwargs})
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/generation/configuration_utils.py", line 509, in __init__
    self.validate(is_init=True)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/generation/configuration_utils.py", line 789, in validate
    logger.warning_once(
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/utils/logging.py", line 328, in warning_once
    self.warning(*args, **kwargs)
Message: 'You have set `use_cache` to `False`, but cache_implementation is set to hybrid. cache_implementation will have no effect.'
Arguments: (<class 'UserWarning'>,)
/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/generation/configuration_utils.py:628: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
  warnings.warn(
The 'batch_size' attribute of HybridCache is deprecated and will be removed in v4.49. Use the more precisely named 'self.max_batch_size' attribute instead.
Traceback (most recent call last):
  File "/shared/aisingapore/peerat/TinyLLaVA_Factory/test.py", line 21, in <module>
    eval_model(args)
  File "/shared/aisingapore/peerat/TinyLLaVA_Factory/tinyllava/eval/run_tiny_llava.py", line 81, in eval_model
    output_ids = model.generate(
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/shared/aisingapore/peerat/TinyLLaVA_Factory/tinyllava/model/modeling_tinyllava.py", line 187, in generate
    return self.language_model.generate(
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/generation/utils.py", line 2252, in generate
    result = self._sample(
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/generation/utils.py", line 3251, in _sample
    outputs = self(**model_inputs, return_dict=True)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/models/gemma/modeling_gemma.py", line 1072, in forward
    outputs = self.model(
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/models/gemma/modeling_gemma.py", line 827, in forward
    layer_outputs = decoder_layer(
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/models/gemma/modeling_gemma.py", line 552, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/peerat/.conda/envs/tinyllava_factory/lib/python3.10/site-packages/transformers/models/gemma/modeling_gemma.py", line 280, in forward
    attn_weights = attn_weights + causal_mask
RuntimeError: The size of tensor a (1290) must match the size of tensor b (778) at non-singleton dimension 3

In addition, when I change the image to TinyLLaVA_Factory/tinyllava/serve/examples/waterview.jpg The last line of the error changes to RuntimeError: The size of tensor a (2826) must match the size of tensor b (778) at non-singleton dimension 3

Any suggestion? Thank you.

[ZhangXJ199]

Gemma2 and Gemma might be different, and you may need to modify ‘tinyllava/data/template/gemma_template.py’ and ‘tinyllava/model/llm/gemma.py’.

[mrpeerat]

Thank you for the suggestion. However, the training is smooth and fine. And I think Gemma and Gemma2 have the same architecture and prompt. So, I think the problem might be somewhere else. Especially when I changed only the image, I got different error outputs.