Unable to load .json and .ubj files from /proc

[balazsprehoda]

Issue

When trying to load .ubj and .json files, we get the following traceback:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/xgbserver/xgbserver/__main__.py", line 46, in <module>
    model.load()
  File "/xgbserver/xgbserver/model.py", line 58, in load
    self._booster = xgb.Booster(
                    ^^^^^^^^^^^^
  File "/prod_venv/lib/python3.11/site-packages/xgboost/core.py", line 1724, in __init__
    self.load_model(model_file)
  File "/prod_venv/lib/python3.11/site-packages/xgboost/core.py", line 2697, in load_model
    _check_call(_LIB.XGBoosterLoadModel(self.handle, c_str(fname)))
  File "/prod_venv/lib/python3.11/site-packages/xgboost/core.py", line 284, in _check_call
    raise XGBoostError(py_str(_LIB.XGBGetLastError()))
xgboost.core.XGBoostError: filesystem error: cannot make canonical path: No such file or directory [/mnt/models/xg.ubj]

We are using xgboost v2.1.1, and loading .bst files from /proc/$pid/root/models works fine. The problem appears to be with LoadSequentialFile:

auto path = std::filesystem::weakly_canonical(std::filesystem::u8path(uri));

I see that on master, all model types are read using this method, so a simple version upgrade is not going to fix this.

Context

We are using xgboost (v2.1.1) with kserve (v0.15.2), using kserve's "modelcar" approach. The xgboost server and the model itself are in different containers of a pod (with process namespace sharing enabled), and xgboost should read the model file through /proc/$pid/root/..., where $pid is the process id of the modelcar container. /mnt/models in the xgboost server container is a symlink pointing to /proc/$pid/root/models (the modelcar's filesystem).

Is there a solution for this that I'm not aware of? We would like to avoid copying the model itself to a shared volume since that would defeat the purpose of using modelcars (at least partially).

[hcho3]

The model loading method was written with the assumption that the model file was residing in a file system.

I think there are two ways to proceed.

Option 1. You can load models from an in-memory buffer. The load_model() method accepts a bytearray argument where the byte array represents the content of the model file; OR
Option 2. Update LoadSequentialFile to support /proc files. Would you be willing to contribute a fix?

[balazsprehoda]

@hcho3 Thank you for your insights. With the bytearray option available, I think it's reasonable to fix this on the client side. I opened a PR for this in the kserve repository.