Model execution time for a 600x800 color image takes 7.5 seconds - Any ways to reduce ??

[gganes3]

Using db resnet-50 and crnnvgg16_bn models for prediction, is there any way to reduce execution time?

[fg-mindee]

Hi @gganes3 👋

This is quite an interesting topic 👍
There are many things at hand, I'll try to give the best overview I can:

• input size : the input size has a direct impact on latency. It's recommended to start with the size the model was trained on, but if needed, you can try to reduce the input size (that might have some consequences on performances, depending on how far you changed it to)
• hardware: from the latency you measured, I guess you are running this on CPU. You could switch to GPU for a better latency!
• backend : which backend are you running? Pytorch or TensorFlow?
• architecture choice: docTR provides several architectures, and depending on whether you run PyTorch or TensorFlow, the layers of the model will not have the same latency (the under-the-hood low-level implementations of layers is specific to the framework/backend). But you could check lighter architectures to get a better latency
• runtime: so this would require doing a few these manually (cannot be used as is) but if using a DL framework for training or development doesn't mean that one cannot use something else for inference. Pytorch models usually convert nicely to ONNX for instance, which is usually better for CPU in terms of latency. For TensorFlow there are a few options, but not supported officially as of today: you can check OpenVINO

Now that being said, could you share a minimal snippet showcasing how you performed inference please?
The default size that we use is (1024, 1024), and as you can see on our latency benchmarks (https://github.com/mindee/doctr/blob/main/references/detection/latency.csv, https://github.com/mindee/doctr/blob/main/references/recognition/latency.csv), the measured latency is way lower than this. This could be because you have an extremely dense document, and thus that the recognition model needs to run on many batches.

I hope this helps :)

[gganes3]

Yeah, same thoughts here. Not able to understand why. I am trying to move to the GPU however I am encountering an issue where it says Torch not compiled with CUDA enabled

  File "C:/Projects/AutoPixelSizeCalculation/AutoPixelSizeCalculation_ocrpredictor_test.py", line 48, in calculatepixelsize
    model = OCRPredictor(det_predictor, reco_predictor).cuda()
  File "C:\Users\gganesan166268\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 680, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "C:\Users\gganesan166268\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 570, in _apply
    module._apply(fn)
  File "C:\Users\gganesan166268\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 570, in _apply
    module._apply(fn)
  File "C:\Users\gganesan166268\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 570, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "C:\Users\gganesan166268\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 593, in _apply
    param_applied = fn(param)
  File "C:\Users\gganesan166268\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\nn\modules\module.py", line 680, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "C:\Users\gganesan166268\AppData\Local\Programs\Python\Python37\lib\site-packages\torch\cuda\__init__.py", line 208, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled

Is it that I have not installed the GPU version of cuda?

[fg-mindee]

It most likely means that you did not install the GPU version of PyTorch. Usually, it's because of some NVidia driver issue when you installed the package, and at that time it doesn't detect CUDA and picks the CPU-only build!

[fg-mindee]

did you manage to fix this @gganes3 ?

[gganes3]

No. I am still trying @fg-mindee. By the way, can you just let me know how I can retrain my detection model to size(256,1024). Also, what is the data that you use for training the detection model?

[fg-mindee]

First things first, could you try to investigate your CUDA situation? About non-square size for text detection and the data, it's a completely different topic so I suggest to discuss this elsewhere 😅