OCR Performance question

[AI091]

Hi team, I see that in Docling v2.56, the default OCR model is changed to Rapid-OCR. What are the implications of this with GPU acceleration, is it well supported? I tried it earlier but it seemed onnxruntime didn't use the GPU properly.

[dolfim-ibm]

If you install onnxruntime-gpu it should work.

[AI091]

Thanks, trying it out

[AI091]

I am still getting very high CPU usage , I think its because of this RapidAI/RapidOCR#112

[AI091]

should I be using torch backend instead?

[AI091]

Running with openvino works properly, even slightly faster than EasyOCR with GPU accleration.

[dolfim-ibm]

Thanks for giving it a try. We will make sure to document it. (if you would like to make a docs contribution, you are welcome)

[AI091]

Thanks, will try to do it soon :)

[dolfim-ibm]

@AI091 which steps did you take to run the openvino version? Some extra dependency?

[AI091]

Yes, just installed openvino package and chose it as the backend in RapidOcrOptions

[hisan-ideamaker]

@AI091 's solution worked ! I got around 30-40% increase in speed just by selecting openvino. Tested it on colab t4 gpu. Adding code instructions here, for someone looking for the code:

first: pip install openvino
then:

pipeline_options = PdfPipelineOptions()
pipeline_options.ocr_options = RapidOcrOptions(
    backend="openvino",
)

worked like a charm :)

[dolfim-ibm]

@hisan-ideamaker @AI091 @geoHeil Are you saying that selecting openvino is going to use the GPU? Which GPU? We just tried with Nvidia GPUs and doesn't seem to do anything on it.

[AI091]

I tried it on laptop GPU (RTX 4060 mobile) , had lower cpu usage , higher GPU usage and lower time than when I used paddle or onnxruntime .

[AI091]

all attempts were inside docling however, didn't attempt to isolate and run OCR on its own

[hisan-ideamaker]

@dolfim-ibm hey, i'm not quite sure if its utilizing the gpu, but i did notice considerable increase in speed just by changing backend to openvino. Testing it with different pdfs around 20-40% if your tests didn't suggest that the gpu is being utilised, could it be that openvino backend is just faster than the onnx ?
I tested it on colab t4 gpu and m1 pro.

[dolfim-ibm]

Yes, I think this is the case (depending on your hardware, but I guess on Intel CPU it could have a significant speedup).

[tripflex]

@hisan-ideamaker yes my assumption is that openvino uses intel's avx and/or other instruction sets for CPU acceleration (although i have not tested this myself).

@dolfim-ibm can you please let us know when the PR for torch support will be added (mentioned PR coming soon), thanks, would love to test this out!