server : fix format_infill

[ngxson]

Should fix #10691 (comment) , I remove the format_chat/_infill/_rerank from handle_completions_generic but forgot to put it back in handle_infill

[ggerganov]

We should probably return an error if there is more than 1 resulting prompt?

[ngxson]

Because above we already checked if data["prompt"] is string, here we can be sure that we only have one single prompt to deal with. Probably an GGML_ASSERT here make more sense?

(The expected behavior of tokenize_input_prompts is that if prompt is a string, then output vector size is == 1)

[ggerganov]

Got it. It's fine as it is.

[ngxson]

@ggerganov I added a test using Qwen2.5-Coder-1.5B-Instruct-GGUF, you can run it locally using:

SLOW_TESTS=1 ./tests.sh ./unit/test_infill.py -v -x

Feel free to add more complicated test case(s) if you need!

[ngxson]

Btw please note that adding "prompt": "Complete this" to the request makes Qwen model to hallucinate the answer. Looking at the technical report, I don't think the model is trained to follow instruction when doing infill:

[ggerganov]

Ah, the infill endpoint should be used only with the Coder models. Not the Coder-Instruct. The authors confirmed that: https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct/discussions/2#6731a45e0e39be0605a0df20

So the tests should be updated to use the Coder variant.

[ngxson]

OK so I've tried the non-instruction model but I think the problem is related to placement of prompt in the formatted version. What I'm getting is:

<|repo_name|>myproject\n<|file_sep|>llama.h\nLLAMA_API int32_t llama_n_threads();\n<|file_sep|>filename\n<|fim_prefix|>#include <cstdio>\n#include \"llama.h\"\n\nint main() {\n int n_threads = llama_Complete this<|fim_suffix|>}\n<|fim_middle|>

If the prompt is placed at the beginning, it should work:

Complete this<|repo_name|>myproject\n<|file_sep|>llama.h\nLLAMA_API int32_t llama_n_threads();\n<|file_sep|>filename\n<|fim_prefix|>#include <cstdio>\n#include \"llama.h\"\n\nint main() {\n int n_threads = llama_<|fim_suffix|>}\n<|fim_middle|>

We can fix this in another PR, for now I'm gonna comment out the "prompt": "Complete this" and change the model to non-instruction

[ggerganov]

FIM should not add instructions such as "Complete this", as can be seen in the technical report. The "prompt" field in the /infill requests is designed to contain the prefix of the current line on which the cursor is located. This is appended to the "input_prefix" (which contains the previous lines) to obtain the final fim_prefix. So it is working as intended.

[ngxson]

The "prompt" field in the /infill requests is designed to contain the prefix of the current line on which the cursor is located.

Ok thanks for the clarification. I updated the test to reflect this. The "prompt" now contains the current line where the cursor is located:

    res = server.make_request("POST", "/infill", data={
        "input_extra": [{
            "filename": "llama.h",
            "text": "LLAMA_API int32_t llama_n_threads();\n"
        }],
        "input_prefix": "#include <cstdio>\n#include \"llama.h\"\n\nint main() {\n",
        "prompt": "    int n_threads = llama_",
        "input_suffix": "}\n",
    })

[ggerganov]

Yes, perfect. The test_invalid_input_extra_req also needs to be updated like this.