Eval bug: b7256 breaks MiniCPM-V-4_5

[steampunque]

Name and Version

b7255 (works)
b7256 through at least b7552 (broken)

Operating systems

Linux

GGML backends

CUDA

Hardware

9900k/4070

Models

model:
https://huggingface.co/steampunque/MiniCPM-V-4_5-Hybrid-GGUF/blob/main/MiniCPM-V-4_5.Q6_K_H.gguf
mmproj:
https://huggingface.co/steampunque/MiniCPM-V-4_5-Hybrid-GGUF/blob/main/MiniCPM-V-4_5.mmproj.gguf

Problem description & steps to reproduce

Infinite ??? output on b7256 update.

llama-mtmd-cli -m /datahd/models/MiniCPM-V-4_5.Q6_K_H.gguf --mmproj /datahd/models/MiniCPM-V-4_5.mmproj.gguf --image bandwagon_512.png -p "Describe what is shown in the image"

Test image: https://huggingface.co/steampunque/MiniCPM-V-4_5-Hybrid-GGUF/blob/main/bandwagon_512.png

First Bad Commit

2e1c9cd

Relevant log output

Failing : b7256

Details

llama-mtmd-cli -m /datahd/models/MiniCPM-V-4_5.Q6_K_H.gguf --mmproj /datahd/models/MiniCPM-V-4_5.mmproj.gguf --image bandwagon_512.png -p "Describe what is shown in the image"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4070, compute capability 8.9, VMM: yes
build: 7256 (2e1c9cd81) with cc (GCC) 11.2.0 for x86_64-slackware-linux
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4070) (0000:01:00.0) - 11844 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 399 tensors from /datahd/models/MiniCPM-V-4_5.Q6_K_H.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Model
llama_model_loader: - kv   3:                         general.size_label str              = 8.2B
llama_model_loader: - kv   4:                          qwen3.block_count u32              = 36
llama_model_loader: - kv   5:                       qwen3.context_length u32              = 40960
llama_model_loader: - kv   6:                     qwen3.embedding_length u32              = 4096
llama_model_loader: - kv   7:                  qwen3.feed_forward_length u32              = 12288
llama_model_loader: - kv   8:                 qwen3.attention.head_count u32              = 32
llama_model_loader: - kv   9:              qwen3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  10:                       qwen3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  12:                 qwen3.attention.key_length u32              = 128
llama_model_loader: - kv  13:               qwen3.attention.value_length u32              = 128
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,151748]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,151748]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 151644
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 128244
llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - kv  26:                          general.file_type u32              = 18
llama_model_loader: - type  f32:  145 tensors
llama_model_loader: - type q8_0:   28 tensors
llama_model_loader: - type q4_K:   20 tensors
llama_model_loader: - type q5_K:  127 tensors
llama_model_loader: - type q6_K:   79 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q6_K
print_info: file size   = 5.86 GiB (6.15 BPW) 
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 106
load: token to piece cache size = 0.9321 MB
print_info: arch             = qwen3
print_info: vocab_only       = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 4096
print_info: n_embd_inp       = 4096
print_info: n_layer          = 36
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 12288
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_finetuned   = unknown
print_info: model type       = 8B
print_info: model params     = 8.19 B
print_info: general.name     = Model
print_info: vocab type       = BPE
print_info: n_vocab          = 151748
print_info: n_merges         = 151387
print_info: BOS token        = 151644 '<|im_start|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: UNK token        = 128244 '<unk>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 36 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   486.25 MiB
load_tensors:        CUDA0 model buffer size =  5518.65 MiB
......................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_seq     = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: yarn_log_mul  = 0
llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.58 MiB
llama_kv_cache:      CUDA0 KV buffer size =   576.00 MiB
llama_kv_cache: size =  576.00 MiB (  4096 cells,  36 layers,  1/1 seqs), K (f16):  288.00 MiB, V (f16):  288.00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context:      CUDA0 compute buffer size =   304.38 MiB
llama_context:  CUDA_Host compute buffer size =    16.01 MiB
llama_context: graph nodes  = 1267
llama_context: graph splits = 2
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
mtmd_cli_context: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

clip_model_loader: model name:   
clip_model_loader: description:  image encoder for MiniCPM-V
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    455
clip_model_loader: n_kv:         20

clip_model_loader: has vision encoder
clip_ctx: CLIP using CUDA0 backend
load_hparams: projector:          resampler
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            27
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     0

--- vision hparams ---
load_hparams: image_size:         448
load_hparams: patch_size:         14
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   5
load_hparams: n_merge:            0
load_hparams: n_wa_pattern:       0

load_hparams: model size:         1044.36 MiB
load_hparams: metadata size:      0.16 MiB
load_tensors: ffn up/down are swapped
warmup: warmup with image size = 448 x 448
alloc_compute_meta:      CUDA0 compute buffer size =    62.81 MiB
alloc_compute_meta:        CPU compute buffer size =     2.31 MiB
alloc_compute_meta: graph splits = 1, nodes = 893
warmup: flash attention is enabled
main: loading model: /datahd/models/MiniCPM-V-4_5.Q6_K_H.gguf
encoding image slice...
image slice encoded in 117 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 9 ms
encoding image slice...
image slice encoded in 51 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 7 ms
encoding image slice...
image slice encoded in 49 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 7 ms

???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????^C?


llama_perf_context_print:        load time =    1715.07 ms
llama_perf_context_print: prompt eval time =     427.76 ms /   214 tokens (    2.00 ms per token,   500.28 tokens per second)
llama_perf_context_print:        eval time =    1801.19 ms /   131 runs   (   13.75 ms per token,    72.73 tokens per second)
llama_perf_context_print:       total time =    2984.43 ms /   345 tokens
llama_perf_context_print:    graphs reused =        130

Working : b7255

Details

llama-mtmd-cli -m /datahd/models/MiniCPM-V-4_5.Q6_K_H.gguf --mmproj /datahd/models/MiniCPM-V-4_5.mmproj.gguf --image bandwagon_512.png -p "Describe what is shown in the image"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4070, compute capability 8.9, VMM: yes
build: 7255 (190c4838b) with cc (GCC) 11.2.0 for x86_64-slackware-linux
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4070) (0000:01:00.0) - 11844 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 399 tensors from /datahd/models/MiniCPM-V-4_5.Q6_K_H.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Model
llama_model_loader: - kv   3:                         general.size_label str              = 8.2B
llama_model_loader: - kv   4:                          qwen3.block_count u32              = 36
llama_model_loader: - kv   5:                       qwen3.context_length u32              = 40960
llama_model_loader: - kv   6:                     qwen3.embedding_length u32              = 4096
llama_model_loader: - kv   7:                  qwen3.feed_forward_length u32              = 12288
llama_model_loader: - kv   8:                 qwen3.attention.head_count u32              = 32
llama_model_loader: - kv   9:              qwen3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  10:                       qwen3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  12:                 qwen3.attention.key_length u32              = 128
llama_model_loader: - kv  13:               qwen3.attention.value_length u32              = 128
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,151748]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,151748]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 151644
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 128244
llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - kv  26:                          general.file_type u32              = 18
llama_model_loader: - type  f32:  145 tensors
llama_model_loader: - type q8_0:   28 tensors
llama_model_loader: - type q4_K:   20 tensors
llama_model_loader: - type q5_K:  127 tensors
llama_model_loader: - type q6_K:   79 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q6_K
print_info: file size   = 5.86 GiB (6.15 BPW) 
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 106
load: token to piece cache size = 0.9321 MB
print_info: arch             = qwen3
print_info: vocab_only       = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 4096
print_info: n_embd_inp       = 4096
print_info: n_layer          = 36
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 12288
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_finetuned   = unknown
print_info: model type       = 8B
print_info: model params     = 8.19 B
print_info: general.name     = Model
print_info: vocab type       = BPE
print_info: n_vocab          = 151748
print_info: n_merges         = 151387
print_info: BOS token        = 151644 '<|im_start|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: UNK token        = 128244 '<unk>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 36 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   486.25 MiB
load_tensors:        CUDA0 model buffer size =  5518.65 MiB
......................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_seq     = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: yarn_log_mul  = 0
llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.58 MiB
llama_kv_cache:      CUDA0 KV buffer size =   576.00 MiB
llama_kv_cache: size =  576.00 MiB (  4096 cells,  36 layers,  1/1 seqs), K (f16):  288.00 MiB, V (f16):  288.00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context:      CUDA0 compute buffer size =   304.38 MiB
llama_context:  CUDA_Host compute buffer size =    16.01 MiB
llama_context: graph nodes  = 1267
llama_context: graph splits = 2
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
mtmd_cli_context: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

clip_model_loader: model name:   
clip_model_loader: description:  image encoder for MiniCPM-V
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    455
clip_model_loader: n_kv:         20

clip_model_loader: has vision encoder
clip_ctx: CLIP using CUDA0 backend
load_hparams: projector:          resampler
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            27
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     0

--- vision hparams ---
load_hparams: image_size:         448
load_hparams: patch_size:         14
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   5
load_hparams: n_merge:            0
load_hparams: n_wa_pattern:       0

load_hparams: model size:         1044.36 MiB
load_hparams: metadata size:      0.16 MiB
load_tensors: ffn up/down are swapped
warmup: warmup with image size = 448 x 448
alloc_compute_meta:      CUDA0 compute buffer size =    62.81 MiB
alloc_compute_meta:        CPU compute buffer size =     2.31 MiB
alloc_compute_meta: graph splits = 1, nodes = 893
warmup: flash attention is enabled
main: loading model: /datahd/models/MiniCPM-V-4_5.Q6_K_H.gguf
encoding image slice...
image slice encoded in 109 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 8 ms
encoding image slice...
image slice encoded in 53 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 7 ms
encoding image slice...
image slice encoded in 49 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 7 ms

<think>
So, let's analyze the image. First, the title is "The Bandwagon" by Claude E. Shannon. It's from "IRE Transactions—Information Theory". The layout is a two-column text page. The content is about information theory, discussing its development, applications, and the concept of a "bandwagon" effect. The text mentions how information theory has become a scientific bandwagon, starting as a technical tool for communication engineers, then expanding into psychology, biology, economics, etc. It talks about the importance of scientific rigor and avoiding the bandwagon effect, emphasizing that information theory is a branch of mathematics with strict deductive methods. Also, there's mention of Claude Shannon's work and the need for scientific standards in the field.
</think>

The image shows a two-column text page from a publication titled *The Bandwagon* by Claude E. Shannon, featured in *IRE Transactions—Information Theory*. The content discusses the evolution of information theory, describing it as a “scientific bandwagon” that originated as a technical tool for communication engineers but expanded into diverse fields like psychology, biology, economics, and sociology. It emphasizes the importance of maintaining scientific rigor and avoiding the “bandwagon effect” (where ideas gain popularity without solid foundation). The text also highlights Claude Shannon’s contributions and the need for strict deductive methods in information theory, contrasting it with less rigorous approaches in other fields. Decorative elements (like flourishes) frame the text, and the page number “1088” appears at the top.


llama_perf_context_print:        load time =    1707.29 ms
llama_perf_context_print: prompt eval time =     417.95 ms /   214 tokens (    1.95 ms per token,   512.02 tokens per second)
llama_perf_context_print:        eval time =    4262.59 ms /   310 runs   (   13.75 ms per token,    72.73 tokens per second)
llama_perf_context_print:       total time =    5501.85 ms /   524 tokens
llama_perf_context_print:    graphs reused =        307

[ggerganov]

I cannot reproduce neither with Metal nor with CUDA (DGX Spark) using 2e1c9cd and latest master.

[steampunque]

I cannot reproduce neither with Metal nor with CUDA (DGX Spark) using 2e1c9cd and latest master.

Still fails on master for me with updated nvidia driver to latest production release 580.119.02. A different launch line is needed on master due to the fit command issues:

Details

llama-mtmd-cli -m /data3hd/models/MiniCPM-V-4_5.Q6_K_H.gguf -fit off -c 4096 --mmproj /data3hd/models/MiniCPM-V-4_5.mmproj.gguf --image bandwagon_512.png -p "Describe what is shown in the image"
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 4070, compute capability 8.9, VMM: yes
build: 7569 (0c8986403) with GNU 11.2.0 for Linux x86_64
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 4070) (0000:01:00.0) - 11443 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 399 tensors from /data3hd/models/MiniCPM-V-4_5.Q6_K_H.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Model
llama_model_loader: - kv   3:                         general.size_label str              = 8.2B
llama_model_loader: - kv   4:                          qwen3.block_count u32              = 36
llama_model_loader: - kv   5:                       qwen3.context_length u32              = 40960
llama_model_loader: - kv   6:                     qwen3.embedding_length u32              = 4096
llama_model_loader: - kv   7:                  qwen3.feed_forward_length u32              = 12288
llama_model_loader: - kv   8:                 qwen3.attention.head_count u32              = 32
llama_model_loader: - kv   9:              qwen3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  10:                       qwen3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  12:                 qwen3.attention.key_length u32              = 128
llama_model_loader: - kv  13:               qwen3.attention.value_length u32              = 128
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,151748]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,151748]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 151644
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 128244
llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - kv  26:                          general.file_type u32              = 18
llama_model_loader: - type  f32:  145 tensors
llama_model_loader: - type q8_0:   28 tensors
llama_model_loader: - type q4_K:   20 tensors
llama_model_loader: - type q5_K:  127 tensors
llama_model_loader: - type q6_K:   79 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q6_K
print_info: file size   = 5.86 GiB (6.15 BPW) 
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 106
load: token to piece cache size = 0.9321 MB
print_info: arch             = qwen3
print_info: vocab_only       = 0
print_info: no_alloc         = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 4096
print_info: n_embd_inp       = 4096
print_info: n_layer          = 36
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 12288
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_yarn_log_mul= 0.0000
print_info: rope_finetuned   = unknown
print_info: model type       = 8B
print_info: model params     = 8.19 B
print_info: general.name     = Model
print_info: vocab type       = BPE
print_info: n_vocab          = 151748
print_info: n_merges         = 151387
print_info: BOS token        = 151644 '<|im_start|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: UNK token        = 128244 '<unk>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading output layer to GPU
load_tensors: offloading 35 repeating layers to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   486.25 MiB
load_tensors:        CUDA0 model buffer size =  5518.65 MiB
......................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_seq     = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: yarn_log_mul  = 0
llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.58 MiB
llama_kv_cache:      CUDA0 KV buffer size =   576.00 MiB
llama_kv_cache: size =  576.00 MiB (  4096 cells,  36 layers,  1/1 seqs), K (f16):  288.00 MiB, V (f16):  288.00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context:      CUDA0 compute buffer size =   304.38 MiB
llama_context:  CUDA_Host compute buffer size =    16.01 MiB
llama_context: graph nodes  = 1267
llama_context: graph splits = 2
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
mtmd_cli_context: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

clip_model_loader: model name:   
clip_model_loader: description:  image encoder for MiniCPM-V
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    455
clip_model_loader: n_kv:         20

clip_model_loader: has vision encoder
clip_ctx: CLIP using CUDA0 backend
load_hparams: projector:          resampler
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            27
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     0

--- vision hparams ---
load_hparams: image_size:         448
load_hparams: patch_size:         14
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   5
load_hparams: n_merge:            0
load_hparams: n_wa_pattern:       0

load_hparams: model size:         1044.36 MiB
load_hparams: metadata size:      0.16 MiB
load_tensors: ffn up/down are swapped
warmup: warmup with image size = 448 x 448
alloc_compute_meta:      CUDA0 compute buffer size =    62.81 MiB
alloc_compute_meta:        CPU compute buffer size =     2.31 MiB
alloc_compute_meta: graph splits = 1, nodes = 893
warmup: flash attention is enabled
main: loading model: /data3hd/models/MiniCPM-V-4_5.Q6_K_H.gguf
WARN: This is an experimental CLI for testing multimodal capability.
      For normal use cases, please use the standard llama-cli
encoding image slice...
image slice encoded in 15045 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 56367 ms
encoding image slice...
image slice encoded in 60 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 5 ms
encoding image slice...
image slice encoded in 53 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 5 ms

?????????????????????????????????????????????????????????????????????????????????????????????????????????????????^C?


llama_perf_context_print:        load time =    1101.29 ms
llama_perf_context_print: prompt eval time =   71616.06 ms /   214 tokens (  334.65 ms per token,     2.99 tokens per second)
llama_perf_context_print:        eval time =    8403.35 ms /   113 runs   (   74.37 ms per token,    13.45 tokens per second)
llama_perf_context_print:       total time =   80509.40 ms /   327 tokens
llama_perf_context_print:    graphs reused =        112

[ggerganov]

It works ok also on RTX 5090:

CUDA_VISIBLE_DEVICES=0 ./bin/llama-mtmd-cli -hf steampunque/MiniCPM-V-4_5-Hybrid-GGUF:Q6_K_H --image ./bandwagon_512.png -p "Describe what is shown in the image" -fit off -c 4096

Details

CUDA_VISIBLE_DEVICES=0 ./bin/llama-mtmd-cli -hf steampunque/MiniCPM-V-4_5-Hybrid-GGUF:Q6_K_H --image ./bandwagon_512.png -p "Describe what is shown in the image" -fit off -c 4096
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes
common_download_file_single_online: using cached file: /home/ggerganov/.cache/llama.cpp/steampunque_MiniCPM-V-4_5-Hybrid-GGUF_MiniCPM-V-4_5.Q6_K_H.gguf
common_download_file_single_online: using cached file: /home/ggerganov/.cache/llama.cpp/steampunque_MiniCPM-V-4_5-Hybrid-GGUF_MiniCPM-V-4_5.mmproj.gguf
build: 7573 (3595ae596) with GNU 13.3.0 for Linux x86_64
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 5090) (0000:01:00.0) - 31588 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 399 tensors from /home/ggerganov/.cache/llama.cpp/steampunque_MiniCPM-V-4_5-Hybrid-GGUF_MiniCPM-V-4_5.Q6_K_H.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Model
llama_model_loader: - kv   3:                         general.size_label str              = 8.2B
llama_model_loader: - kv   4:                          qwen3.block_count u32              = 36
llama_model_loader: - kv   5:                       qwen3.context_length u32              = 40960
llama_model_loader: - kv   6:                     qwen3.embedding_length u32              = 4096
llama_model_loader: - kv   7:                  qwen3.feed_forward_length u32              = 12288
llama_model_loader: - kv   8:                 qwen3.attention.head_count u32              = 32
llama_model_loader: - kv   9:              qwen3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  10:                       qwen3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  12:                 qwen3.attention.key_length u32              = 128
llama_model_loader: - kv  13:               qwen3.attention.value_length u32              = 128
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,151748]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,151748]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 151644
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 128244
llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - kv  26:                          general.file_type u32              = 18
llama_model_loader: - type  f32:  145 tensors
llama_model_loader: - type q8_0:   28 tensors
llama_model_loader: - type q4_K:   20 tensors
llama_model_loader: - type q5_K:  127 tensors
llama_model_loader: - type q6_K:   79 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q6_K
print_info: file size   = 5.86 GiB (6.15 BPW) 
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 106
load: token to piece cache size = 0.9321 MB
print_info: arch             = qwen3
print_info: vocab_only       = 0
print_info: no_alloc         = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 4096
print_info: n_embd_inp       = 4096
print_info: n_layer          = 36
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 12288
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_yarn_log_mul= 0.0000
print_info: rope_finetuned   = unknown
print_info: model type       = 8B
print_info: model params     = 8.19 B
print_info: general.name     = Model
print_info: vocab type       = BPE
print_info: n_vocab          = 151748
print_info: n_merges         = 151387
print_info: BOS token        = 151644 '<|im_start|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: UNK token        = 128244 '<unk>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading output layer to GPU
load_tensors: offloading 35 repeating layers to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   486.25 MiB
load_tensors:        CUDA0 model buffer size =  5518.65 MiB
......................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 4096
llama_context: n_ctx_seq     = 4096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (4096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     0.58 MiB
llama_kv_cache:      CUDA0 KV buffer size =   576.00 MiB
llama_kv_cache: size =  576.00 MiB (  4096 cells,  36 layers,  1/1 seqs), K (f16):  288.00 MiB, V (f16):  288.00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context:      CUDA0 compute buffer size =   304.38 MiB
llama_context:  CUDA_Host compute buffer size =    16.01 MiB
llama_context: graph nodes  = 1267
llama_context: graph splits = 2
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
mtmd_cli_context: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
clip_model_loader: model name:   
clip_model_loader: description:  image encoder for MiniCPM-V
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    455
clip_model_loader: n_kv:         20
clip_model_loader: has vision encoder
clip_ctx: CLIP using CUDA0 backend
load_hparams: projector:          resampler
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            27
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     0
--- vision hparams ---
load_hparams: image_size:         448
load_hparams: patch_size:         14
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   5
load_hparams: n_merge:            0
load_hparams: n_wa_pattern:       0
load_hparams: model size:         1044.36 MiB
load_hparams: metadata size:      0.16 MiB
load_tensors: ffn up/down are swapped
warmup: warmup with image size = 448 x 448
alloc_compute_meta:      CUDA0 compute buffer size =    62.81 MiB
alloc_compute_meta:        CPU compute buffer size =     2.31 MiB
alloc_compute_meta: graph splits = 1, nodes = 893
warmup: flash attention is enabled
main: loading model: /home/ggerganov/.cache/llama.cpp/steampunque_MiniCPM-V-4_5-Hybrid-GGUF_MiniCPM-V-4_5.Q6_K_H.gguf
WARN: This is an experimental CLI for testing multimodal capability.
      For normal use cases, please use the standard llama-cli
encoding image slice...
image slice encoded in 37 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 23 ms
encoding image slice...
image slice encoded in 20 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 4 ms
encoding image slice...
image slice encoded in 18 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 4 ms
<think>
So, let's analyze the image. First, the image is a page from "The Transactions—Information Theory" volume 8. The title is "The Bandwagon" by Claude E. Shannon. The page has two columns of text. The top has the journal name, volume number, and title. The content is about information theory, mentioning how it has become a scientific bandwagon, discussing its applications and the need for a scientific attitude. The text is dense with technical and theoretical discussion about information theory's role and challenges.
</think>
The image shows a page from a publication titled *The Transactions—Information Theory* (volume 8), featuring an article named "The Bandwagon" by Claude E. Shannon. The page is formatted with two columns of text. At the top, the journal title and volume number are displayed, followed by the article title and author. The content discusses the concept of information theory, its development, and its status as a "scientific bandwagon," touching on topics like its application to various fields, the importance of a scientific approach, and the challenges of maintaining rigor in the subject. The text is dense with technical and theoretical discourse, framed within a scholarly context.
llama_perf_context_print:        load time =     725.14 ms
llama_perf_context_print: prompt eval time =     146.42 ms /   214 tokens (    0.68 ms per token,  1461.54 tokens per second)
llama_perf_context_print:        eval time =    1236.49 ms /   243 runs   (    5.09 ms per token,   196.52 tokens per second)
llama_perf_context_print:       total time =    1616.75 ms /   457 tokens
llama_perf_context_print:    graphs reused =        241

[steampunque]

It works ok also on RTX 5090:

CUDA_VISIBLE_DEVICES=0 ./bin/llama-mtmd-cli -hf steampunque/MiniCPM-V-4_5-Hybrid-GGUF:Q6_K_H --image ./bandwagon_512.png -p "Describe what is shown in the image" -fit off -c 4096

Details

Problem Might be unique to 40 series and or need blackwell to work right, I think dgx was blackwell too. Best guess is blackwell is needed or it will not work with the new cuda code. Also should run with my mmproj just to minimize any possible differences.
edit: I see the command you used automatically pulled my mmproj so that is OK.

[ggerganov]

I can reproduce on GTX 1660 and confirm that #17505 caused the regression. The problematic op is the flash attention call in clip.cpp:

llama.cpp/tools/mtmd/clip.cpp

Lines 606 to 617 in 3b54531

	if (flash_attn_type == CLIP_FLASH_ATTN_TYPE_ENABLED) {
	ggml_tensor * v = ggml_permute(ctx0, v_cur, 0, 2, 1, 3);
	k = ggml_cast(ctx0, k, GGML_TYPE_F16);
	v = ggml_cast(ctx0, v, GGML_TYPE_F16);
	cur = ggml_flash_attn_ext(ctx0, q, k, v, kq_mask, kq_scale, 0.0f, 0.0f);
	ggml_flash_attn_ext_set_prec(cur, GGML_PREC_F32);
	cur = ggml_reshape_2d(ctx0, cur, cur->ne[0]*cur->ne[1], cur->ne[2]*cur->ne[3]);
	} else {

cc @JohannesGaessler

[CISC]

FWIW it works fine on 3090Ti:

Details

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090 Ti, compute capability 8.6, VMM: yes
build: 7563 (9c675c714) with GNU 13.3.0 for Linux x86_64
common_init_result: fitting params to device memory, for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on
llama_params_fit_impl: projected to use 11663 MiB of device memory vs. 23388 MiB of free device memory
llama_params_fit_impl: will leave 11725 >= 1024 MiB of free device memory, no changes needed
llama_params_fit: successfully fit params to free device memory
llama_params_fit: fitting params to free memory took 0.32 seconds
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090 Ti) (0000:01:00.0) - 23388 MiB free
llama_model_loader: loaded meta data with 27 key-value pairs and 399 tensors from /home/cisc/Downloads/MiniCPM-V-4_5.Q6_K_H.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Model
llama_model_loader: - kv   3:                         general.size_label str              = 8.2B
llama_model_loader: - kv   4:                          qwen3.block_count u32              = 36
llama_model_loader: - kv   5:                       qwen3.context_length u32              = 40960
llama_model_loader: - kv   6:                     qwen3.embedding_length u32              = 4096
llama_model_loader: - kv   7:                  qwen3.feed_forward_length u32              = 12288
llama_model_loader: - kv   8:                 qwen3.attention.head_count u32              = 32
llama_model_loader: - kv   9:              qwen3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  10:                       qwen3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  11:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  12:                 qwen3.attention.key_length u32              = 128
llama_model_loader: - kv  13:               qwen3.attention.value_length u32              = 128
llama_model_loader: - kv  14:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  15:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  16:                      tokenizer.ggml.tokens arr[str,151748]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  17:                  tokenizer.ggml.token_type arr[i32,151748]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  18:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  19:                tokenizer.ggml.bos_token_id u32              = 151644
llama_model_loader: - kv  20:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  21:            tokenizer.ggml.unknown_token_id u32              = 128244
llama_model_loader: - kv  22:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  23:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  24:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  25:               general.quantization_version u32              = 2
llama_model_loader: - kv  26:                          general.file_type u32              = 18
llama_model_loader: - type  f32:  145 tensors
llama_model_loader: - type q8_0:   28 tensors
llama_model_loader: - type q4_K:   20 tensors
llama_model_loader: - type q5_K:  127 tensors
llama_model_loader: - type q6_K:   79 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q6_K
print_info: file size   = 5.86 GiB (6.15 BPW) 
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 106
load: token to piece cache size = 0.9321 MB
print_info: arch             = qwen3
print_info: vocab_only       = 0
print_info: no_alloc         = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 4096
print_info: n_embd_inp       = 4096
print_info: n_layer          = 36
print_info: n_head           = 32
print_info: n_head_kv        = 8
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 4
print_info: n_embd_k_gqa     = 1024
print_info: n_embd_v_gqa     = 1024
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 12288
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: n_expert_groups  = 0
print_info: n_group_used     = 0
print_info: causal attn      = 1
print_info: pooling type     = -1
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_yarn_log_mul= 0.0000
print_info: rope_finetuned   = unknown
print_info: model type       = 8B
print_info: model params     = 8.19 B
print_info: general.name     = Model
print_info: vocab type       = BPE
print_info: n_vocab          = 151748
print_info: n_merges         = 151387
print_info: BOS token        = 151644 '<|im_start|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: UNK token        = 128244 '<unk>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading output layer to GPU
load_tensors: offloading 35 repeating layers to GPU
load_tensors: offloaded 37/37 layers to GPU
load_tensors:   CPU_Mapped model buffer size =   486.25 MiB
load_tensors:        CUDA0 model buffer size =  5518.65 MiB
......................................................................................
common_init_result: added <|endoftext|> logit bias = -inf
common_init_result: added <|im_end|> logit bias = -inf
common_init_result: added <|fim_pad|> logit bias = -inf
common_init_result: added <|repo_name|> logit bias = -inf
common_init_result: added <|file_sep|> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 40960
llama_context: n_ctx_seq     = 40960
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context:  CUDA_Host  output buffer size =     0.58 MiB
llama_kv_cache:      CUDA0 KV buffer size =  5760.00 MiB
llama_kv_cache: size = 5760.00 MiB ( 40960 cells,  36 layers,  1/1 seqs), K (f16): 2880.00 MiB, V (f16): 2880.00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context:      CUDA0 compute buffer size =   384.38 MiB
llama_context:  CUDA_Host compute buffer size =    88.01 MiB
llama_context: graph nodes  = 1267
llama_context: graph splits = 2
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
mtmd_cli_context: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant

clip_model_loader: model name:   
clip_model_loader: description:  image encoder for MiniCPM-V
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    455
clip_model_loader: n_kv:         20

clip_model_loader: has vision encoder
clip_ctx: CLIP using CUDA0 backend
load_hparams: projector:          resampler
load_hparams: n_embd:             1152
load_hparams: n_head:             16
load_hparams: n_ff:               4304
load_hparams: n_layer:            27
load_hparams: ffn_op:             gelu
load_hparams: projection_dim:     0

--- vision hparams ---
load_hparams: image_size:         448
load_hparams: patch_size:         14
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   5
load_hparams: n_merge:            0
load_hparams: n_wa_pattern:       0

load_hparams: model size:         1044.36 MiB
load_hparams: metadata size:      0.16 MiB
load_tensors: ffn up/down are swapped
warmup: warmup with image size = 448 x 448
alloc_compute_meta:      CUDA0 compute buffer size =    62.81 MiB
alloc_compute_meta:        CPU compute buffer size =     2.31 MiB
alloc_compute_meta: graph splits = 1, nodes = 893
warmup: flash attention is enabled
main: loading model: /home/cisc/Downloads/MiniCPM-V-4_5.Q6_K_H.gguf
WARN: This is an experimental CLI for testing multimodal capability.
      For normal use cases, please use the standard llama-cli
encoding image slice...
image slice encoded in 84 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 38 ms
encoding image slice...
image slice encoded in 45 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 5 ms
encoding image slice...
image slice encoded in 43 ms
decoding image batch 1/1, n_tokens_batch = 64
image decoded (batch 1/1) in 5 ms

<think>


The image shows a scanned page from a publication titled "THE TRANSACTIONS—INFORMATION THEORY." The page number is 1088, and the article is titled "The Bandwagon" by Claude E. Shannon. The text is formatted in two columns with a decorative border at the top and bottom. The content discusses the concept of information theory, its development, and its applications in various fields such as communication, cybernetics, and automations. It also touches on the challenges and misconceptions surrounding the theory, emphasizing the need for a scientific approach rather than relying on popular trends or superficial applications. The text is dense and technical, reflecting the academic nature of the publication.
</think>

The image displays a page from a publication titled "THE TRANSACTIONS—INFORMATION THEORY," specifically page 1088. The article is titled "The Bandwagon" by Claude E. Shannon. The page features two columns of text with a decorative border at the top and bottom. The content discusses the development and application of information theory, addressing its role in communication, cybernetics, and automations. It critiques the tendency to adopt information theory as a "bandwagon" without proper scientific rigor, emphasizing the need for a disciplined, experimental approach rather than relying on popular trends or superficial applications. The text is dense and technical, reflecting the academic nature of the publication.


llama_perf_context_print:        load time =    1216.44 ms
llama_perf_context_print: prompt eval time =     287.11 ms /   214 tokens (    1.34 ms per token,   745.37 tokens per second)
llama_perf_context_print:        eval time =    2571.71 ms /   280 runs   (    9.18 ms per token,   108.88 tokens per second)
llama_perf_context_print:       total time =    3323.37 ms /   494 tokens
llama_perf_context_print:    graphs reused =        278

[steampunque]

I can reproduce on GTX 1660 and confirm that #17505 caused the regression. The problematic op is the flash attention call in clip.cpp:

llama.cpp/tools/mtmd/clip.cpp

Lines 606 to 617 in 3b54531
if (flash_attn_type == CLIP_FLASH_ATTN_TYPE_ENABLED) {
ggml_tensor * v = ggml_permute(ctx0, v_cur, 0, 2, 1, 3);

 k = ggml_cast(ctx0, k, GGML_TYPE_F16); 
 v = ggml_cast(ctx0, v, GGML_TYPE_F16); 

 cur = ggml_flash_attn_ext(ctx0, q, k, v, kq_mask, kq_scale, 0.0f, 0.0f); 
 ggml_flash_attn_ext_set_prec(cur, GGML_PREC_F32); 

 cur = ggml_reshape_2d(ctx0, cur, cur->ne[0]*cur->ne[1], cur->ne[2]*cur->ne[3]); 

} else {

cc @JohannesGaessler

I ran with FA on and off and got the same ????, so was not localized to FA for me. I am experimenting with rebuilding with different cuda arch enabled (turned out I did not have Ada specified in my config which was created before I had my 4070s) will update if I find anything.

[ggerganov]

You have to dis§ble the FA for the CLIP. There is no CLI arg for that. You can disable it like this:

diff --git a/tools/mtmd/mtmd.cpp b/tools/mtmd/mtmd.cpp
index b9c4fa909..0184c3a01 100644
--- a/tools/mtmd/mtmd.cpp
+++ b/tools/mtmd/mtmd.cpp
@@ -174,7 +174,7 @@ struct mtmd_context {
 
         clip_context_params ctx_clip_params {
             /* use_gpu           */ ctx_params.use_gpu,
-            /* flash_attn_type   */ CLIP_FLASH_ATTN_TYPE_AUTO,
+            /* flash_attn_type   */ CLIP_FLASH_ATTN_TYPE_DISABLED,
             /* image_min_tokens  */ ctx_params.image_min_tokens,
             /* image_max_tokens  */ ctx_params.image_max_tokens,
             /* warmup            */ ctx_params.warmup,

[steampunque]

You have to dis§ble the FA for the CLIP. There is no CLI arg for that. You can disable it like this:

diff --git a/tools/mtmd/mtmd.cpp b/tools/mtmd/mtmd.cpp
index b9c4fa909..0184c3a01 100644
--- a/tools/mtmd/mtmd.cpp
+++ b/tools/mtmd/mtmd.cpp
@@ -174,7 +174,7 @@ struct mtmd_context {

     clip_context_params ctx_clip_params {
         /* use_gpu           */ ctx_params.use_gpu,

•        /* flash_attn_type   */ CLIP_FLASH_ATTN_TYPE_AUTO,
  

•        /* flash_attn_type   */ CLIP_FLASH_ATTN_TYPE_DISABLED,
         /* image_min_tokens  */ ctx_params.image_min_tokens,
         /* image_max_tokens  */ ctx_params.image_max_tokens,
         /* warmup            */ ctx_params.warmup,
  

Ah mystery solved. I wondered what was going on since the pr was involving FA. Thanks.

[steampunque]

I am experimenting with rebuilding with different cuda arch enabled (turned out I did not have Ada specified in my config which was created before I had my 4070s) will update if I find anything.

OK it works for me now when I build with Ada arch enabled. I just enabled most of everything:
export CUDA_ARCH="60;61;70;75;80;86;89"

I think I will move to CUDA_ARCH=native since I build for my local machine anyway.

So it seems new code relies on having Ada arch available or the compute silently cranks out NaNs or otherwise invalid values.

Update:

export CUDA_ARCH="60;61;70;75;80"

also works. The 80 turns on Ampere arch, which is now apparently necessary at build time after 2e1c9cd if target is going to be run on Ampere or Ada.

[JohannesGaessler]

I cannot reproduce this issue at all with any software version on any of my GPUs. In particular I cannot reproduce it by emulating a 4070 with my 4090 by manually setting 56 as the SM count.

OK it works for me now when I build with Ada arch enabled.

At no point in the code do we currently have anything using any features available on Ada Lovelace but not on Ampere. The only explanation where what you're describing would make sense is if somewhere in the kernel there is undefined behavior or if your hardware is unstable.

[steampunque]

I cannot reproduce this issue at all with any software version on any of my GPUs. In particular I cannot reproduce it by emulating a 4070 with my 4090 by manually setting 56 as the SM count.

OK it works for me now when I build with Ada arch enabled.

At no point in the code do we currently have anything using any features available on Ada Lovelace but not on Ampere. The only explanation where what you're describing would make sense is if somewhere in the kernel there is undefined behavior or if your hardware is unstable.

If you build with this cuda config it should expose the problem I was having:

export CUDA_ARCH="60;61;70;75"

Also earlier in the thread it was noted to be failing on 1660 with the same problem.