LLaVA-NeXT is not supported ?

[micoraweb]

Hello, it looks like the latest LlamaSharp is able to run Llava models 1.5 and 1.6, but Llava Next is not supported ?
Support for many types of modern vision models (Mini CPM) is unfortunately not implemented ...
Are there any plans to support those in the near future ?

This model here, for example: https://huggingface.co/KBlueLeaf/llama3-llava-next-8b-gguf - doesn't 'see' anything,
even though it's supposed to provide an improved understanding and higher resolution than 1.5 and 1.6 llava models...

[SignalRT]

@micoraweb it should be supported by LlamaSharp if llama.cpp support the format.

Have you tried with llama.cpp before to test it with LlamaSharp?

Making a little research It seems that ollama (that depends also on llama.cpp) manage this model. This is the prompt template that they are using:

{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}{{ if .Prompt }}<|im_start|>user
{{ .Prompt }}<|im_end|>
{{ end }}<|im_start|>assistant

This is a llama3 based template, it's very different that the one we are using in LlamaSharp llava example (that is based on mistral 7B).

Could you share the prompt that you are using?

[micoraweb]

Thank you for your reply!
My prompt is based on the example code from LlamaSharp's documentation and I'm doing something like this:

if (!prompt.Contains("<image>"))
{
    prompt = $"<image>\n{prompt}";
}

I suspect that the isn't being passed to the model correctly?

[SignalRT]

That prompt is adjusted to the model used in the example.

To use a llama3 based model you need to use a prompt based in llama3.