Feature Request: Pluggable transport/serialization for embeddings (Mistralai-embed)

[flefevre]

LiteLLM supports embedding models via HTTP (OpenAI-compatible APIs), including Mistralai embeddings.

However, in some production setups—particularly with the latest Mistralai-embed model (mistral-embed-2312)—embeddings are served over non-HTTP backends (e.g., NATS) with binary serialization (pickle) and pre-tokenized inputs. LiteLLM currently assumes JSON over HTTP, which makes integration difficult without a separate adapter.

Proposal

Add a pluggable transport + serialization interface for embeddings, allowing:

Custom backends (NATS, gRPC, RPC, etc.)

Binary payloads and pre-tokenized inputs

Reuse of LiteLLM’s model abstraction, routing, and observability

Example config:

embedding_backend:
  type: custom
  transport: nats
  serializer: pickle
  subject: mistral-embed-2312

This would simplify integration for high-performance internal pipelines using mistral-embed-2312.

Willingness to Contribute

Happy to help with design discussion.