Invalid input type issue while running Vanna with local LLM-mistral7B-instruct

[ravvleen]

I am trying to run vanna with my locally hosted Mistral7B-instruct and Qdrant DB.

For LLM, I have tried using the OpenAI compatible endpoint using openai-chat object and with vllm endpoint through a CustomLLM class. I get the following issue while executing vn.train(sql="SELECT * FROM model_table;") while calling through an OpenAI compatible endpoint. Similar with vn.ask()

Error calling local LLM: Error code: 400 - {'object': 'error', 'message': "6 validation errors for ValidatorIterator\n0.typed-dict.text\n  Field required [type=missing, input_value={'role': 'system', 'conte... name in the question.'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/missing\n0.typed-dict.type\n  Field required [type=missing, input_value={'role': 'system', 'conte... name in the question.'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/missing\n0.typed-dict.image_url\n  Field required [type=missing, input_value={'role': 'system', 'conte... name in the question.'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/missing\n0.typed-dict.type\n  Field required [type=missing, input_value={'role': 'system', 'conte... name in the question.'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/missing\n0.typed-dict.input_audio\n  Field required [type=missing, input_value={'role': 'system', 'conte... name in the question.'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/missing\n0.typed-dict.type\n  Field required [type=missing, input_value={'role': 'system', 'conte... name in the question.'}, input_type=dict]\n    For further information visit https://errors.pydantic.dev/2.9/v/missing", 'type': 'BadRequestError', 'param': None, 'code': 400}
Question generated with sql: Error: Could not get a response from the local LLM.

This is my calling function:

class MyVanna(Qdrant_VectorStore, MyLocalOpenAILLM):
    def __init__(self, config=None):
        Qdrant_VectorStore.__init__(self, config=config)
        MyLocalOpenAILLM.__init__(self, config=config)

qdrant = QdrantClient( url="...", port=None, prefix=None, api_key="....")
vn = MyVanna(config={ 'client': qdrant, 
    "api_key": "....",  
    "base_url": ".....",  # local LLM's base URL
    "model_name": "....." #model path
})