Does this project support Vector DB search and send to (local) LLM to generate answers?

[orchardc]

The project looks really good and thanks for sharing it.

I'm using Python + LangChain, I have a question : Does this project support Vector DB search and send to (local) LLM to generate answers? It would be great to see a how-do example.

[SignalRT]

You can get the Vector DB support with Semantic Kernel. I'm working on a pipeline to process documents to be able to chat about the documentation referencing the sources, and it works. In my experience I get far worst results that making the same with OpenAI.

The current state of llama.cpp seems not to be the right one to have a great RAG system. The best open model we could use, seems to be BGE or something similar. Is seems that in llama.cpp they are working on supporting this models, but it's not yet done:

ggerganov/llama.cpp#3667

Right now it's possible to make what you are asking, but you will get worst results due to impossibility to support the best models to support Massive Text Embedding:

https://huggingface.co/spaces/mteb/leaderboard

[AshD]

Would the Mistral 7B v0.2 Instruct be a good choice here?

Thanks,
Ash