Model Initialisation happening per request - JAVA GPU inferencing

[suryakant261]

We are observing high latency when running real-time batch inference on GPUs using the xgboost4j-gpu Java library. Our investigation suggests that a model initialization step may be occurring on every predict call, even when Booster objects are reused.
Please see the attached flame graph, which highlights the time spent in the model initialization phase during a predict call.

Code reference we are using for prediction.

try {
            DMatrix dmat = new DMatrix(features, nRows, nCols, Float.NaN);
            booster = boosters.borrowObject();  // taking a pre initialised Booster object from a pool
            Timer.Context context = MetricUtils.addTimer("xgb_true_time", this.getClass()).time();

            float[][] modelScores = booster.predict(dmat);
            context.stop();

        } catch (XGBoostError e) {
            e.printStackTrace();
        } finally {
            if (booster != null ) {
                boosters.returnObject(booster); // returning back to pool
            }

Question
Is this internal model initialization for each batch prediction an expected behavior for the GPU predictor, or could this indicate a potential bug or a misconfiguration on our part? Any guidance on how to avoid this per-batch setup cost would be greatly appreciated.

(I was referring to this section, and from the source code, I also understood that Init is called for every predict call , https://github.com/dmlc/xgboost/blob/release_2.0.0/src/predictor/gpu_predictor.cu#L890)

[trivialfis]

It's an issue we have been trying to address. The tree model is stored in the CPU and must be copied to the GPU for every prediction call. We are moving toward persistent GPU storage, but it will take some time before we get there.

[iyashu]

Thanks @trivialfis for the revert. As of today, we're using cpu for inferencing xgboost model for our application and moving to GPU degraded the overall performance of the application due to copying of model on every predict call.

Is there any alternative API or mechanism we can use to leverage GPUs for inference efficiently?

[trivialfis]

For fast inference, please visit the FIL project https://docs.rapids.ai/api/cuml/stable/fil/

[trivialfis]

I will work on this. Do you have any suggestions for how to expose the inplace predict method to Java?

The inplace predict skips the construction of the DMatrix, but requires input data to be on GPU if you want to run inference on GPU.

Alternatively, you are happy with the DMatrix path?

[suryakant261]

Thanks for picking this up @trivialfis

In place predict also seems to be loading model per request. how would using InplacePredict help avoid the model initialisation overhead per inference request
https://github.com/dmlc/xgboost/blob/release_2.0.0/src/predictor/gpu_predictor.cu#L971
d_model.Init(model, tree_begin, tree_end, m->DeviceIdx());

Is there a possibility where we can keep a set of models initialised on GPU and do these init operations only if it's first inference request for the model?

[trivialfis]

Yes. I will try to persist the model on GPU. But now that you are using the Java interface and looking to optimize the latency:

We are observing high latency when running real-time batch inference on GPUs using the xgboost4j-gpu Java library.

I think exposing the inplace predict would be a good extra, after the model initialization is sorted out.