Dependency updates & compatibility errors (re-training needed?)

[Joelius300]

I was running a model trained with mlflow 2.6.0 on mlserver 1.6.1. After upgrading mlserver to 1.7.0, I get compatibility errors as outlined below. I'm not sure where the root of issue is exactly, so this issue contains a lot of information from different angles. I apologize for the scattered thoughts and hope you can answer my questions regarding compatibility below. Thank you!

In mlserver v1.7.0, specifically this PR (#1951), mlserver has increased the version their proto gencode is built with from 4.25.1 to 5.27.2. This is a breaking change; the current version is not backwards compatible with any protobuf runtime <5.27.2. The protobuf cross-version runtime guarantee states that New Gencode + Old Runtime = Never Allowed.
This breaking change is causing issues with older protobuf runtimes as already reported: #2130

As outlined in that issue, I think the protobuf requirements should be changed/restricted from "*" to ">=5.27.2".

This also implies an increase of the minimum supported mlflow version. Mlflow 2.6.0 has a requirement of protobuf "<5", so it will pull the latest 4.x.x runtime. This runtime cannot work with the newer proto gencode of mlserver v1.7.0, where the proto version was 5.27.2. This issue persists up to mlflow v2.15.0, when they rebuild their own proto gencode and relax the constraint to "<6" (see PR mlflow/mlflow#12360). (As you can see in their PR, they don't have to increase the minimum proto version, because they build their protos in such a way, that it switches the version based on the currently running/imported protobuf version. mlserver does not do this resulting in a hard increase of the minimum protobuf version.)

Therefore, I think the mlflow dependency should also be changed/restricted from "*" to ">=2.15.0. Thanks to another compatibility bug found with mlflow (#2113), this has already been changed to ">=2.19.0", but not released.

I am running models trained with mlflow v2.6.0 and want to increase the mlserver version to 1.7. I expected this to work without a problem because the environments should be separated and dependencies should not infere. However, I get the protobuf error as described in #1951. It seems I would need to re-train the model using a newer mlflow version (specifically >=2.19.0) in order for it to run on mlserver 1.7.0.

This situation leads me to some questions:

1. In general, is it supported to run models trained with older mlflow versions on newer mlserver versions (given no major version changes in either)?
2. Is it correct that models trained with mlflow <2.19.0 need to be re-trained with >=2.19.0 to run with mlserver 1.7.0?
3. Are there any compatibility guarantees within a major like 1.x.x (only minor bumps) regarding trained/existing mlflow models?
4. Why/how do the dependencies of the outside mlserver environment interact with the isolated environment of the mlflow model? Is this expected? (see below)

Some additional info because I fear that I'm chasing the wrong thing:

I noticed in the stacktrace of the protobuf error that it starts inside of the isolated mlflow environment but ends in the outer mlserver environment. It looks like a runpy call is jumping the model (mlflow) environment border and ending up in the mlserver environment, and I'm quite confused by that. Is this something that is expected? If this "sandbox escape" was not there and the runpy call would only use the dependencies in the model (mlflow) environment, it would not be able to import the new mlserver proto gencode (the one built with proto 5.27.2) where it currently fails. I'm not sure what it would import, because mlserver is only specified as an extra dependency and has no upper limit in 2.6.0 (source), so mlserver wouldn't be there (?) unfortunately, I don't understand the interaction of mlflow and mlserver in this case well enough.

Here's that stacktrace (see env info below):

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/multiprocessing/spawn.py", line 122, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/app-root/.conda/envs/env/bin/mlserver", line 5, in <module>
    from mlserver.cli import main
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/__init__.py", line 2, in <module>
    from .server import MLServer
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/server.py", line 17, in <module>
    from .grpc import GRPCServer
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/grpc/__init__.py", line 1, in <module>
    from .server import GRPCServer
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/grpc/server.py", line 9, in <module>
    from .servicers import InferenceServicer
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/grpc/servicers.py", line 3, in <module>
    from . import dataplane_pb2 as pb
  File "/opt/app-root/.conda/envs/env/lib/python3.11/site-packages/mlserver/grpc/dataplane_pb2.py", line 9, in <module>
    from google.protobuf import runtime_version as _runtime_version
ImportError: cannot import name 'runtime_version' from 'google.protobuf' (/mnt/environments/3014daa3513bab3ce03cda028baa54034e3dff5de7e96c2e46ba23ee3154b0ff/lib/python3.11/site-packages/google/protobuf/__init__.py)

Environment info:

mlserver env (/opt/app-root/.conda/envs/env/)

Created manually with conda inside a container which then runs mlserver.

• python3.11
• mlflow==2.22.0
• mlserver==1.7.0 -> contains gencode that requires protobuf runtime >=5.27.2

model (mlflow) env (/mnt/environments/3014daa3513)

Created/managed by mlserver or mlflow (?) according to the conda.yaml file in the mlflow artifact

• python3.11
• mlflow==2.6.0
• implicitly protobuf<5

[Joelius300]

I'm now pretty confident my way of creating the environment does not work together with the custom conda environments and the way we modify the default docker image. I will update when I know more, but the questions regarding mlflow compatibility between model environment and the mlserver itself are still open; I don't think I understand that part fully.

Edit: It seems like this was not the problem.

[Joelius300]

The interweaving of the mlserver environment and the model environment are a big problem for us and we will need to re-evaluate whether we can keep using mlserver.

I will keep this issue open because of the questions that still stand. Notably, we would like to understand exactly how the environments of the mlserver and the mlflow model work together, especially in cases where key dependencies like mlflow have different versions. The requirements on differing python versions resp. pickle protocol version as outlined in the docs seem to be much less strict and in practice, problems arise in different places fairly quickly if some mismatch appears (ps. is the version matrix still up to date?).

We're not sure if our original goal to use mlserver to independently serve mlflow models in their own, isolated environments and therefore decouple the upgrade-lifecycle of model and model-serving-wrapper is supported by mlserver.

We would greatly appreciate any help in understanding the underlying process better and advice on how our goals can be achieved with your product. Thank you for developing mlserver.

[RobertSamoilescu]

Hi @Joelius300 ! Thank you for raising those issues. Before answering your questions above, I would really appreciate if you could tell me if those issues still persist in the 1.7.1.rc1. I am working towards a release, and I would like to address them, if possible.

[Joelius300]

@RobertSamoilescu Thank you for the response. I tested again with mlserver 1.7.1 and get the same errors. I couldn't find any difference between 1.7.0 and 1.7.1 in my tests.

[RobertSamoilescu]

Can you please provide a minimal script to generate the mlflow artifact along with the list of all installed packages you have in the Python env that you trained the model with? It would be very useful to replicate the error on my side.

I managed on my end to server a LR model trained with mlflow 2.6.0 with the latest release of mlserver and I only got the following warnings, which were expected:

WARNING mlflow.utils.requirements_utils: Detected one or more mismatches between the model's dependencies and the current Python environment:
 - mlflow (current: 2.19.0, required: mlflow==2.6.0)
 - cloudpickle (current: 3.1.0, required: cloudpickle==2.2.1)
 - scikit-learn (current: 1.5.2, required: scikit-learn==1.6.1)
 - scipy (current: 1.13.1, required: scipy==1.15.3)

[Joelius300]

@RobertSamoilescu That's exciting to hear! Are you using the seldonio/mlserver docker image? Because we built our own custom image to cache the built conda environment because creating, packing and unpacking all take a long time, so we want to do all of them only once.

In the original post I included "Environment info" at the bottom with the versions that aren't working together. We also tested many other things with different python versions, etc. Our latest attempt is using mlserver 1.7.1, mlserver-mlflow 1.7.1 and mlflow-skinny 2.22.0. This works for models trained with 2.22.0 (have that version in the conda.yaml file), but not for models trained with 2.6.0 (there we get the infamous protobuf error).

[RobertSamoilescu]

Yes, I'm using seldonio/mlserver docker image.

The fact that you mentioned you are building your own image might help. What I suspect it happens is that when you are building your docker image, an older version of protobuf is installed which is incompatible with the gencode files we generated. Since you are building the docker image yourself, after installing all the dependencies, could you please try to regenerate the gencode files by running the command make generate and let me know if the error still persists?

[Joelius300]

Ah well, we're installing the package into the image from PyPi instead of building it from source like in the official docker image. I tried to generate the protos after the installation, but a lot of files are missing to make it work if I understand correctly (the protos themselves as well as the bash scripts for doing so). I'm not sure we can achieve what you suggest without completely rewriting our Dockerfile.

I'm curious, is the protobuf version in your image dependent on anything outside of the mlserver package? I don't understand how you would resolve to a different version than we do. And if you resolve to the same version (5.x) it also wouldn't change the gencode version, right? The runtime code is the one in the model env apparently and with mlflow==2.6.0, that is protobuf 4.x (incompatible with the 5.x gencode).

As an aside, I played around with runtime.py of the mlserver-mlflow package to see if I can figure something out. I found that you can move all the mlflow imports into the functions to defer importing them, but unfortunately it does not fix the protobuf error. Protobuf is not imported yet when the MLflowRuntime is instantiated (unlike mlflow at the moment).

Before going on to test some more things, could you provide me with some info about what is supported by and expected from mlserver?

1. In general, is it supported to run models trained with older mlflow versions on newer mlserver versions (given no major version changes in either)?
2. Is it correct that models trained with mlflow <2.19.0 need to be re-trained with >=2.19.0 to run with mlserver 1.7.1? -> Already answered (No) as you said you got model 2.6.0 with server 2.19.0 running (source).
3. Are there any compatibility guarantees within a major like 1.x.x (only minor bumps) regarding trained/existing mlflow models? Can we expect that minor upgrades in mlserver should work with all existing models that were running on the previous version?
4. Why/how do the dependencies of the outside mlserver environment interact with the isolated environment of the mlflow model? The mixing of environments via the multiprocessing hacks makes it very hard for me to see through, maybe you'd be able to explain it.

Thank you for taking the time!

[RobertSamoilescu]

1. In general, you should be able to run models with older mlflow version on newer mlserver version. This seems to be a guarantee from mlflow (see here)
2. No, re-training is not necessary.
3. We don't provide any guarantees, but we try as much as we can to keep things compatible.
4. If needed, you can create MLServer images that have specific dependencies (at specific versions) preloaded. If you need to build a custom MLServer image, I recommend you following our docs. You should be able to easily modify the Dockerfile to generate compatible gencode files.
   Secondly, you may provide a custom conda environment together with a particular model (see here). In that case, all the models using the same custom environment are separated into a different "inference pool". This is a set of workers (processes) that all run the exact conda environment provided (including the python interpreter). There are some compatibility requirements so that the version python interpeter in the conda environment and the python interpreter of the base MLServer image are compatible in the format they use for serialising things, specifically writing and reading from the same multiprocessing.Queue.

[Joelius300]

Thank you for the answers and links.
I realize our use-case might not be in scope of mlserver. We are already providing custom environments for models with the "environment_path" option (docs). The custom environment for the model is built from the conda.yaml file that mlflow logs when using log_model and contains all dependencies needed for running the model (but does and should not contain mlserver). To avoid creating this env, packing and unpacking it every time the container is started, we have a custom docker image that handles the packaging and caches the tar'd env, as well as unpacking it into a shared volume so that is only needs to be done once for every model period, regardless of how many times it's started.
If I understand correctly, the doc you linked (building a custom mlserver image) is for the case when you have 1 model and want to bundle its dependencies with the mlserver image, in which case you also don't set "environment_path". In our case we have 1 or more models, all with their own environments (specified with "environment_path") but we still need one central mlserver installation that manages the inference part as well es the parallel inference.
I think those "compatibility requirements" you mention is what we are struggling with. We also thought that it was just the python version and the pickle protocol that need to match, but when we try it, there seem to be much more components that need to be compatible, e.g. the protobuf. Since we want to upgrade the mlserver version often but avoid retraining our models, we'd like to bump mlserver from 1.6 to 1.7, but when we try that it fails because of a protobuf incompatibility.

What could we do to get our setup to work? We would like to

• always use the latest compatible mlserver version
• have each model in their own, isolated environment (e.g. specified with "environment_path")
• have a custom docker image that can handle our internal caching mechanisms

Is this possible with mlserver?

[lc525]

MLServer, to simplify loading of direct model artifacts (such as those created by mlflow), has the concept of "runtimes". This is code that implements the MLModel "interface". (see here for mlflow).

In this way, the model that you specify as an end-user can be simplified considerably (i.e "just point to the model in standard mlflow format and MLServer can do the rest"). However, what MLServer actually runs as a "model" is always the class that inherits from "MLServer" (the runtime, be it one already provided together with MLServer or a custom one).

It's important to know that the code of the runtime must be able to run in the same environment as the one specified by the model (via settings like environment_tarball and environment_path).

When you directly load a Model that inherits from MLModel (e.g "custom runtime") you get to fully decide the dependencies for that model, and the environment specified via environment_tarball and environment_path is the one where the code runs.

When you provide a framework-specific model-artifact directly and rely on an existing runtime (the mlflow runtime) your environment needs to make sure that it caters for the dependencies of that runtime as well, because the code of the runtime will end up running in the worker processes that use the environment you've specified. In this case, I believe that the code of the mlflow runtime ends up trying to access the protobuf gencode via a chain of import paths (I haven't dug deep into exactly how that ends up being the case, but as a matter of principle that code is not strictly needed from a runtime perspective).

In the short term, I believe that you should still be able to use MLServer by writing a simple custom runtime that loads your model as a way around this. Ultimately, it means implementing just a couple of methods like "load", "predict" and "unload".

We can also take a look in minimising the dependencies that end up being loaded by our existing runtimes, because they create inadvertent, implicit dependencies in the custom environments, which as you have noticed may lead to package conflicts.

[Joelius300]

Okay, thank you. This snippet is probably what is most important in our case.

It's important to know that the code of the runtime must be able to run in the same environment as the one specified by the model (via settings like environment_tarball and environment_path).

So am I correct in saying that the core issue is that mlserver-mlflow runtime is not compatible with mlflow <2.19? If we add the mlserver-mlflow package to the model environment, we would probably see the incompatibility directly out of the gate and the build would be prevented (I'll try to confirm that see at the bottom). Generally, we didn't want the serving engine (mlserver) inside of the model environment, but in this case it seems that it's conceptually impossible as it will interact, even if it's not installed in the model env.

Also, regarding this part:

In this case, I believe that the code of the mlflow runtime ends up trying to access the protobuf gencode via a chain of import paths (I haven't dug deep into exactly how that ends up being the case, but as a matter of principle that code is not strictly needed from a runtime perspective).

I suspect the same thing. I even tried to fix it by moving imports around but I guess I would need to remove them instead of just moving them.

In the short term, I believe that you should still be able to use MLServer by writing a simple custom runtime that loads your model as a way around this. Ultimately, it means implementing just a couple of methods like "load", "predict" and "unload".

In this case, we would need to re-implement the mlserver-mlflow package, right?

We can also take a look in minimising the dependencies that end up being loaded by our existing runtimes, because they create inadvertent, implicit dependencies in the custom environments, which as you have noticed may lead to package conflicts.

That would be great! The current compatibility restrictions make life-cycling models hard.

Edit: I confirmed that installing mlserver-mlflow into the model env will result in an earlier fail. Again, it does not solve the problem, and we also don't want it, but it confirms what we suspected.

ERROR: Cannot install -r /mnt/models/... (line 5) and mlflow==2.6.0 because these package versions have conflicting dependencies.
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[lc525]

So am I correct in saying that the core issue is that mlserver-mlflow runtime is not compatible with mlflow <2.19? Generally, we didn't want the serving engine (mlserver) inside of the model environment, but in this case it seems that it's conceptually impossible as it will interact, even if it's not installed in the model env.

I believe you are correct. Conceptually, it's not the "serving engine" that you need inside the model environment, but code that knows how to take the mlflow model artifact and "run it" within MLServer (the runtime). It's just that the way the current runtimes are written, they import code from mlserver rather than both the runtime and mlserver importing a thin package with the required interfaces (which would be the solution to the problems you're seeing). If you would want to contribute a PR to this before we get to it, we'd certainly welcome it!

The mlserver-mlflow runtime is tested with mlflow versions >=2.19. However, from what we have tried, model artifacts trained with versions <2.19 should still load/run on mlflow versions >=2.19 (i.e if you were to update the version installed in your environment)

Also, regarding this part:

In this case, I believe that the code of the mlflow runtime ends up trying to access the protobuf gencode via a chain of import paths (I haven't dug deep into exactly how that ends up being the case, but as a matter of principle that code is not strictly needed from a runtime perspective).

I suspect the same thing. I even tried to fix it by moving imports around but I guess I would need to remove them instead of just moving them.

The runtime imports mlserver.X packages, but the __init__ of MLServer imports things like .server that bring in more dependencies than strictly needed.

In the short term, I believe that you should still be able to use MLServer by writing a simple custom runtime that loads your model as a way around this. Ultimately, it means implementing just a couple of methods like "load", "predict" and "unload".

In this case, we would need to re-implement the mlserver-mlflow package, right?

Yes, and for now you would need to copy the MLServer datatypes (mlserver/model.py and mlserver/settings.py and dependencies (mlserver/codecs/, mlserver/types/) rather than importing them from MLServer.

The mlflow-runtime is written fairly generally, but minimally (i.e for a PoC) you would need to implement/copy from the existing runtime just the the load and predict functions. You could also choose to write this specific to one model (i.e load a specific mlflow model artifact from the same directory) Further docs here.

We can also take a look in minimising the dependencies that end up being loaded by our existing runtimes, because they create inadvertent, implicit dependencies in the custom environments, which as you have noticed may lead to package conflicts.

That would be great! The current compatibility restrictions make life-cycling models hard.

I agree this is harder than it needs to be when using custom MLServer images and environments but without writing your own custom runtimes.

[Joelius300]

It's just that the way the current runtimes are written, they import code from mlserver rather than both the runtime and mlserver importing a thin package with the required interfaces (which would be the solution to the problems you're seeing).

I assume this is closely related to #1892 right?

If you would want to contribute a PR to this before we get to it, we'd certainly welcome it!

If we decide it's the best way to solve our issue, I might, but I'm not sure we will.

The mlserver-mlflow runtime is tested with mlflow versions >=2.19. However, from what we have tried, model artifacts trained with versions <2.19 should still load/run on mlflow versions >=2.19 (i.e if you were to update the version installed in your environment).

That's very possible, but then we are fiddling around with the model environment after it was trained, which we'd like to avoid.

We will also consider writing our own custom mlflow runtime, but again, I'm not sure what path the team will decide.

Thank you both for the detailed answers! I think it should be enough to devise a plan for us but I think that this (combination of) issue(s) is very important to mlserver itself and would like to keep this issue open if possible.