[Solved] Cannot be used with vLLM, connection error "400"

[appleappstore102210]

When you try to open a Letta services locally and according to the official documentation https://docs.letta.com/guides/server/providers/vllm steps to connect to the vLLM as a backend, you may meet problem, cannot use and an error prompt "400". Here's an unofficial solution:

• Local environment for testing:

  • Letta deployed in docker
  • Qwen-Coder-3B deployed in the vLLM service
  • Windows 11 24H2 with WSL2 Ubuntu

• Solution steps:

  1. If your Letta is running in Docker, press Ctrl+C to stop the logs. Then, use the command docker start [your-container] to start this container, and enter the container's terminal by typing docker exec -it [your-container] bash.
  2. Open the file letta/llm_api/llm_api_tools.py, locate the following code segment (around line 146-147):

  else:
      functional_call = "required"
  

  Change it to:

  elif model_settings.vllm_api_base != None:
      function_call = "none"
      model_settings.openai_api_key='sk-placeholder'
  else:
      function_call = "required"
  

  3. Restart your container (or just restart the service), using docker restart [your-container].
  4. The issue should now be resolved.

Or, you can directly use the modified version provided in the attachment to replace letta/llm_api/llm_api_tools.py.
llm_api_tools.py

Since I do not have the local deployment of the required development environment, I have not submitted a Pull Request, but I hope this can help you.