MLServer

An open source inference server to serve your machine learning models.

⚠️ This is a Work in Progress.

Overview

MLServer aims to provide an easy way to start serving your machine learning models through a REST and gRPC interface, fully compliant with KFServing's V2 Dataplane spec.

You can read more about the goals of this project on the inital design document.

Usage

You can install the mlserver package running:

pip install mlserver

Note that to use any of the optional inference runtimes, you'll need to install the relevant package. For example, to serve a scikit-learn model, you would need to install the mlserver-sklearn package:

pip install mlserver-sklearn

For further information on how to use MLServer, you can check any of the available examples.

Inference Runtimes

Inference runtimes allow you to define how your model should be used within MLServer. Out of the box, MLServer comes with a set of pre-packaged runtimes which let you interact with a subset of common ML frameworks. This allows you to start serving models saved in these frameworks straight away.

To avoid bringing in dependencies for frameworks that you don't need to use, these runtimes are implemented as independent optional packages. This mechanism also allows you to rollout your [own custom runtimes]( very easily.

To pick which runtime you want to use for your model, you just need to make sure that the right package is installed, and then point to the correct runtime class in your model-settings.json file.

The included runtimes are:

Framework	Package Name	Implementation Class	Example	Source Code
Scikit-Learn	mlserver-sklearn	mlserver_sklearn.SKLearnModel	Scikit-Learn example	./runtimes/sklearn
XGBoost	mlserver-xgboost	mlserver_xgboost.XGBoostModel	XGBoost example	./runtimes/xgboost
Spark MLlib	mlserver-mllib	mlserver_mllib.MLlibModel	Coming Soon	./runtimes/mllib
LightGBM	mlserver-lightgbm	mlserver_lightgbm.LightGBMModel	Coming Soon	./runtimes/lightgbm

Examples

On the list below, you can find a few examples on how you can leverage mlserver to start serving your machine learning models.

• Serving a scikit-learn model
• Serving a xgboost model
• Serving a lightgbm model
• Serving a custom model
• Multi-Model Serving with multiple frameworks
• Loading / unloading models from a model repository

Developer Guide

Versioning

Both the main mlserver package and the inference runtimes packages try to follow the same versioning schema. To bump the version across all of them, you can use the ./hack/update-version.sh script. For example:

./hack/update-version.sh 0.2.0.dev1