How to create same service with different models trained with different corpuses

[mxab]

Hi,
what is the best approach to publish and version a Service that is packed with models that have the same shape but are trained on different corpuses

something like this:

models = [model_with_corpus1, model_with_corpus2, ...]
for model in models:
    my_service = MyService()
    my_service.pack("model", model)
    my_service.save()

all models will be used in parallel, therefore the a "latest" model does not exists more like several latest ?

[parano]

Hi @mxab - by all models will be used in parallel - do you mean you want to deploy all those models in one model server? How many models do you have in this use case? BentoMl supports packing multiple models into one service for serving, although you will need to make sure all models can fit in memory.

For example, you can do something like this:

models = [model_with_corpus1, model_with_corpus2, ...]
bento_artifacts = [PickleArtifact(f"model_{i}" for i in range(len(models)))]

@artifacts(bento_artifacts)
class MyService(BentoService)
     @api(input=JsonInput())
     def predict(self, json_array):
           results = []
           for i in range(len(self.artifacts)):
              results.append(self.artifacts[f"model_{i}"].predict(json_array))
           return results

my_service = MyService()
for index, model in models.enumerate():
    my_service.pack(f"model_{index}", model)

my_service.save()

[mxab]

That would be one idea, but I guess this is to static for my use case.
Each trained model represents the corpus of one customer in my case. Therfore they have to be a bit dynamic in terms of newly added customers or deleted customers meaning I would have to train a model on a certain model class or remove it later

[parano]

@mxab we have seen similar use cases from other users - what worked really well is to use one BentoService class to pack&save one service bundle per model and deploy all those models individually on platforms such as knative. If you could share a bit more context about your project we can probably give more specific advice.

[mxab]

The use case would be:
We have several customers.
Each customer creates/update/deletes documents.
We want to configure a model and train it on each customer's documents.
The customer's documents change from time to time, therefore we have to retrain its model
Also sometimes we want to update the model configuration itself and therefore need to retrain all the models of all customers again
If a new customer joins we need to train its model with the current configuration. If the customer leaves we need to delete all models trained for it.

I was also think that somthing like Knative would make sense, unfortunatelly our applications are not running on K8s.
Maybe we can create small managing app that schedules the newly build containers and takes care of the routing or so.

Still I think we would require some Service qualifier for each customer, otherwise it currently force us to do something like

class ModelServiceCustomer1(BentoService):
...
class ModelServiceCustomer2(BentoService):
...
... and so on

Also can the Yatai server be configured with a webhook so that it might notify other systems of new models?

[elhe26]

Hey @mxab , did you find a solution?