Eval bug: Incoherence in Mistral 7B Q8_0 on Vulkan backend

[0cc4m]

Name and Version

» build_vk/bin/llama-cli --version
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3090 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
version: 5835 (7209e12d6)
built with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu

Operating systems

Linux

GGML backends

Vulkan

Hardware

RTX 3090 (with and without coopmat2 or coopmat1), AMD Radeon Pro VII, Intel A770

Models

Mistral 7B Q8_0

Problem description & steps to reproduce

Incoherent output (repetitions, missing spaces, random letters, etc) after generating for a while with build_vk/bin/llama-cli -p "Once upon a time in" -c 16384 -b 16384 -ub 512 -n 1024 --ignore-eos -m models/Mistral-7B-Instruct-v0.3-Q8_0.gguf -ngl 99 -no-cnv

First Bad Commit

bd9c981 (#14366)

Relevant log output

build_vk/bin/llama-cli -p "Once upon a time in" -c 16384 -b 16384 -ub 512 -n 1024 --ignore-eos -m models/Mistral-7B-Instruct-v0.3-Q8_0.gguf -ngl 99 -no-cnv

[0cc4m]

@MaggotHATE I think this is the issue you reported in #14366 (comment) . It was not difficult to reproduce using llama-cli.

[0cc4m]

@jeffbolznv We did not consider the GGML_VK_CHECK_RESULTS mode when adding fusion support, it cannot be used to look into the problem here. Do you have an idea on how to update it for fusion?

[mq1]

I identified some broken quants on gemma 3n e4b on my setup (vulkan backend):

• UD-Q4_K_XL
• UD-Q8_K_XL
• UD-Q6_K_XL
• Q8_0
• Q6_K
• Q6_K_L

Q4_K_M and Q5_K_M seem to work fine

[jeffbolznv]

I'll look into this later today.

[l-austenfeld]

I identified some broken quants on gemma 3n e4b on my setup (vulkan backend):

* UD-Q4_K_XL

* UD-Q8_K_XL

* UD-Q6_K_XL

* Q8_0

* Q6_K

* Q6_K_L

Q4_K_M and Q5_K_M seem to work fine

I reported this (or something very similar, I can't tell) in #14525 which seems to be caused by per_layer_token_embd being a Q8_0 quant (Q4_K_M and Q5_K_M have this as Q5_1) and a GET_ROWS call on it resulting in a vector full of zeros instead of the actual values.

The issue reported here seems to be something different. Mistral 7B does not even have per_layer_token_embd weights and the broken output looks very different.

[mq1]

I identified some broken quants on gemma 3n e4b on my setup (vulkan backend):

* UD-Q4_K_XL

* UD-Q8_K_XL

* UD-Q6_K_XL

* Q8_0

* Q6_K

* Q6_K_L

Q4_K_M and Q5_K_M seem to work fine

I reported this (or something very similar, I can't tell) in #14525 which seems to be caused by per_layer_token_embd being a Q8_0 quant (Q4_K_M and Q5_K_M have this as Q5_1) and a GET_ROWS call on it resulting in a vector full of zeros instead of the actual values.

The issue reported here seems to be something different. Mistral 7B does not even have per_layer_token_embd weights and the broken output looks very different.

You’re right

[jeffbolznv]

The bug is due to using the wrong epsilon value (grabbed from the mul tensor rather than the rms_norm tensor). I've also thought of a few more fusion conditions to check, and added support for check_results. I have a bit more testing to do, will make a PR in a while.