feature: local inference using yzma

[deadprogram]

Description

This looks like a very cool project! I was thinking it would be neat to be able to perform all or most of the inference on local models using yzma so the final result could be delivered in a single executable.

What do you all think?

[kujtimiihoxha]

Hey, this looks like a really good use case for Crush https://github.com/charmbracelet/crush, maybe worth a discussion to add native local model support to crush.

[deadprogram]

@kujtimiihoxha that is exactly what led me to this repo. 😸

[kujtimiihoxha]

Sweet, ok so from what I can see we would need to implement tool call parsing on our end right? I am not super familiar with it but I think vllm and other tools that support openai compatible servers implement templates like
https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_mistral.jinja

Other examples https://github.com/vllm-project/vllm/blob/main/examples/

I think some of that is here too https://github.com/ollama/ollama/tree/7dd4862a8997ce6f01018a76ad41bd4d8dbe217b/template

I suppose these templates depend on how the model was trained 🤔

We could create a fantasy provider similar to the other that does the install for llama.cpp behind the scene and loads the model when you call LanguageModel("name")...

Than I suppose the templates could live in the provider and when the LanguageModel sends messages to the model it would format them using that template.

This is just me thinking out loud.

[kujtimiihoxha]

Looks like the templates are in huggingface too https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF?chat_template=default

[joeblew999]

I’m also using https://github.com/hybridgroup/yzma to do local AI

So if there is anything movement on integration into fantasy it would be amazing useful as they a system can be designed to use some local and some remote AI in a combination