How to Add Grammar/Json Mode Sampling with llama_sampler_chain_add?

[niral28]

Hi I'm trying to add json grammar to my llama sampler. But have been struggling for a bit. I've been following this example:

https://github.com/ggerganov/llama.cpp/blob/master/examples/llama.android/llama/src/main/cpp/llama-android.cpp#L319C5-L319C28

And using this function in the library:

llama.cpp/include/llama.h

Line 1159 in 53ff6b9

LLAMA_API struct llama_sampler * llama_sampler_init_grammar(

And by following some of the source code for llama-cli, this is what I came up with:

    auto sparams = llama_sampler_chain_default_params();
    sparams.no_perf = true;
    llama_sampler * smpl = llama_sampler_chain_init(sparams);
    llama_sampler_chain_add(smpl, llama_sampler_init_grammar(model, json_schema_to_grammar(json::parse("{}")).c_str(), "root")))
    llama_sampler_chain_add(smpl, llama_sampler_init_greedy());

However all my model generates is the opening brace "{" , repeatedly. Am I doing this correctly, can anyone help point out whats wrong?

Also do I need the greedy sampler, it seems to crash without it.

llama.cpp/include/llama.h

Line 1037 in 53ff6b9

// // typically, the chain should end with a sampler such as "greedy", "dist" or "mirostat"

[ggerganov]

However all my model generates is the opening brace "{" , repeatedly. Am I doing this correctly, can anyone help point out whats wrong?

After sampling the token, make sure to call llama_sampler_accept().

[niral28]

And just for clarification, is the greedy/topk/some additional sampler required after the grammar sampler?

[niral28]

Still no luck as well,

I added llama_sampler_accept(sampler, new_token_id) right after this line: https://github.com/ggerganov/llama.cpp/blob/master/examples/llama.android/llama/src/main/cpp/llama-android.cpp#L414