Structured output support

[Abdulrahman392011]

structured output is important. I use it a lot in my code. If you may post some templates as to how to use it with the liteRT, it will help a lot. Currently i run it with ollama in python in an ubuntu termux terminal. It works just fine except it doesn't support gpu inference or cpu offload in the future (looking forward to that) so i wanted to use liteRT API instead of ollama but structured output is critical for the code to run well.

Structured output is a way of reducing the model's hallucinations. It makes the model significantly more useful. Google ollama Structured output and read the blog post they have. Basically they form part of the response and force the model to fill in the blanks with a certain type (int, str, list, bool and so on) . This makes the model able to do things like answering mcq questions and extract information, it makes the model more useful.

It's especially apparent in smaller models when tasked with complex tasks.

[pkgoogle]

Hello, so this might belong in a different library... have you checked out https://ai.google.dev/edge/mediapipe/solutions/guide or https://github.com/google-ai-edge/gallery. Wondering if any of those will satisfy your needs instead?

[Abdulrahman392011]

I am sorry, I went through the links and there is no mention of structured outputs or anything with similar function.

this is a lot of work I am guessing, since you need to implement it from scratch.
but I think it will be useful for programmers. you can't really build an app with complex objective without structured output. I tried before and it needs a much bigger model to be able to do the same thing a smaller model with structured outputs can do.

do a google search for ollama structured output and follow the instructions in the blog post to try it yourself before making any conclusions about whether it's worth the effort or not.

ollama has an open source code and they implement it in python so just look at the code and see what can be done in the liteRT framework to have the same function.

[pkgoogle]

Ok no worries, just wanted to make sure we're not reinventing any wheels. We'll take a look.