Misc. bug: Embedding/pooling: I receive 10xvector not 1xvector

[brunette69-ruby]

Name and Version

./llama-server --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce GTX 750 Ti, compute capability 5.0, VMM: yes
version: 5797 (de56944)
built with cc (Debian 12.2.0-14+deb12u1) 12.2.0 for x86_64-linux-gnu

Operating systems

Linux

Which llama.cpp modules do you know to be affected?

llama-server

Command line

llama-server --embedding --pooling last/mean/any etc....

Problem description & steps to reproduce

Helo,
For text (which turns out to be of ten tokens) i get 10 vectors even though i have --pooling enabled. Am I missing something obvious?
This is the script for server, curl post and the embeddings in file. It outputs 10x1024 vectors, not 1x1024 vector.

Server script
!/bin/bash
LLAMA_MODEL="Qwen3-Embedding-0.6B-Q8_0.gguf"
LLAMA_MODEL_PATH="/home/DATA/GGUF/embed"
LLAMA_OPTS="-c 1024 --temp 0.3 --top-k 40 --top-p 0.9 --n-predict 60 --no-warmup --port 8081 --embedding"
LLAMA_PERF_OPTS="-ngl 99 --mlock --pooling last"

llama-server ${LLAMA_PERF_OPTS} ${LLAMA_OPTS} -m ${LLAMA_MODEL_PATH}/${LLAMA_MODEL} ${@}

curl -s -X POST http://localhost:8081/embedding
-H "Content-Type: application/json"
-d '{
"model": "Qwen3-Embedding-0.6B-Q8_0.gguf",
"input": "The quick brown fox jumps over the lazy dog."
}' > q-test-embedding.txt

ls -l
212K Jul 5 03:51 q-test-embedding.txt
jq '.[].embedding | length' ~/tmp/q-test-embedding.txt
10
grep -o ',' q-test-embedding.txt | wc -l
10240

First Bad Commit

No response

Relevant log output

[brunette69-ruby]

I am testing further. And what i have found out is that at first it works and then after running some script for a while it starts to work in a wrong way - producing vectors per token, not per text/chunk

time 0:
q-test-embedding | grep -o ',' | wc -l
1024

time 1:
ruby main.rb ... I processed a small file into RAG Db, calling same server with same settings

time 2: after running ruby program
~/bin q-test-embedding | grep -o ',' | wc -l
10240

time 3: I stop/start the server

time 4: still doesnt work correcty
~/bin q-test-embedding | grep -o ',' | wc -l
10240

q-test-embedding script:
curl -s -X POST http://localhost:8081/embedding
-H "Content-Type: application/json"
-d '{
"model": "Qwen3-Embedding-0.6B-Q8_0.gguf",
"input": "The quick brown fox jumps over the lazy dog."
}'

My assumption is that something "stays" in VRAM and things dont work anymore because of that.

I will restart machine and put testing data in next post.

[brunette69-ruby]

After restart and poweroff, cable out etc..
At first it didn't work but next request it started to work, producing 1 vector. So the thing is "working erraticaly".
Providing log below.

RECAP:
in both cases n_prompt_tokens=10

1. when not working n_past = 10, n_tokens = 10
2. when working n_past = 10, n_tokens = 1
   WHY: it seems erratic, persists thru reboots of machine even

As far as semantic quality of that 1 vector when "it works" IDK. Is --pooling really working and giving last/mean/...? IDK.

At first not working:
cat 4_request.log
q-test-embedding | grep -o ',' | wc -l
10240
~
slot launch_slot_: id 0 | task 0 | processing task
slot update_slots: id 0 | task 0 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 10
slot update_slots: id 0 | task 0 | kv cache rm [0, end)
slot update_slots: id 0 | task 0 | prompt processing progress, n_past = 10, n_tokens = 10, progress = 1.000000
slot update_slots: id 0 | task 0 | prompt done, n_past = 10, n_tokens = 10
slot release: id 0 | task 0 | stop processing: n_past = 10, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /embedding 127.0.0.1 200

Immediately after starts to work:
~ q-test-embedding | grep -o ',' | wc -l
10240
~ q-test-embedding | grep -o ',' | wc -l
1024
~ q-test-embedding | grep -o ',' | wc -l
1024
~ q-test-embedding | grep -o ',' | wc -l
1024
...

srv log_server_r: request: POST /embedding 127.0.0.1 200
slot launch_slot_: id 0 | task 2 | processing task
slot update_slots: id 0 | task 2 | new prompt, n_ctx_slot = 1024, n_keep = 0, n_prompt_tokens = 10
slot update_slots: id 0 | task 2 | need to evaluate at least 1 token for each active slot, n_past = 10, n_prompt_tokens = 10
slot update_slots: id 0 | task 2 | kv cache rm [9, end)
slot update_slots: id 0 | task 2 | prompt processing progress, n_past = 10, n_tokens = 1, progress = 0.100000
slot update_slots: id 0 | task 2 | prompt done, n_past = 10, n_tokens = 1
slot release: id 0 | task 2 | stop processing: n_past = 10, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /embedding 127.0.0.1 200

[brunette69-ruby]

Difference from logs is:

When not working:
NO EVAL!
NO KV CACHE RM!
OK? progress = 1.000000
N_TOKENS=10 - slot update_slots: id 0 | task 0 | prompt done, n_past = 10, n_tokens = 10

When working:
EVAL - slot update_slots: id 0 | task 2 | need to evaluate at least 1 token for each active slot, n_past = 10, n_prompt_tokens = 10
KVCACHE RM - slot update_slots: id 0 | task 2 | kv cache rm [9, end)
N_TOKENS=1 - slot update_slots: id 0 | task 2 | prompt processing progress, n_past = 10, n_tokens = 1, progress = 0.100000
IS THIS EVEN OK? progress = 0.10000

[brunette69-ruby]

OK. I tried with "--pooling mean". While "--pooling last" sometimes works and sometimes doesn't (usually first time or after some "action" ), "--pooling mean" never works.
Pooling mean always produces many embeded tokens, not one embeded vector. It seems that embedding code is largely non functional.

~/Devel/Q q-test-embedding | grep -o ',' | wc -l
10240
~/Devel/Q q-test-embedding | grep -o ',' | wc -l
10240
~/Devel/Q q-test-embedding | grep -o ',' | wc -l
10240
~/Devel/Q q-test-embedding | grep -o ',' | wc -l
10240
~/Devel/Q q-test-embedding | grep -o ',' | wc -l
10240
~/Devel/Q q-test-embedding | grep -o ',' | wc -l
10240
~/Devel/Q q-test-embedding | grep -o ',' | wc -l
10240

srv log_server_r: request: POST /embedding 127.0.0.1 200
slot launch_slot_: id 0 | task 8 | processing task
slot update_slots: id 0 | task 8 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 10
slot update_slots: id 0 | task 8 | kv cache rm [0, end)
slot update_slots: id 0 | task 8 | prompt processing progress, n_past = 10, n_tokens = 10, progress = 1.000000
slot update_slots: id 0 | task 8 | prompt done, n_past = 10, n_tokens = 10
slot release: id 0 | task 8 | stop processing: n_past = 10, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /embedding 127.0.0.1 200
slot launch_slot_: id 0 | task 10 | processing task
slot update_slots: id 0 | task 10 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 10
slot update_slots: id 0 | task 10 | kv cache rm [0, end)
slot update_slots: id 0 | task 10 | prompt processing progress, n_past = 10, n_tokens = 10, progress = 1.000000
slot update_slots: id 0 | task 10 | prompt done, n_past = 10, n_tokens = 10
slot release: id 0 | task 10 | stop processing: n_past = 10, truncated = 0
srv update_slots: all slots are idle
srv log_server_r: request: POST /embedding 127.0.0.1 200
slot launch_slot_: id 0 | task 12 | processing task
slot update_slots: id 0 | task 12 | new prompt, n_ctx_slot = 4096, n_keep = 0, n_prompt_tokens = 10
slot update_slots: id 0 | task 12 | kv cache rm [0, end)
slot update_slots: id 0 | task 12 | prompt processing progress, n_past = 10, n_tokens = 10, progress = 1.000000
slot update_slots: id 0 | task 12 | prompt done, n_past = 10, n_tokens = 10

server script
#!/bin/bash
LLAMA_MODEL="Qwen3-Embedding-0.6B-Q8_0.gguf"
LLAMA_MODEL_PATH="/home/DATA/GGUF/embed"
llama-server --embedding --pooling mean --port 8081 -m ${LLAMA_MODEL_PATH}/${LLAMA_MODEL} ${@}

testing script q-test-embedding
curl -s -X POST http://localhost:8081/embedding
-H "Content-Type: application/json"
-d '{
"model": "Qwen3-Embedding-0.6B-Q8_0.gguf",
"input": "The quick brown fox jumps over the lazy dog."
}'

[ggerganov]

Qwen3 Embedding and Qwen3 Reranking models are not supported yet.