different between heatmap and ocr output

[RickyGunawan09]

first, I want to say thank you for this awesome repo that you guys make is really wonderful.

but I have some questions about the detection. I see some differences between heatmap and OCR output in huggingface space.

as you can see in the heatmap they can detect all the words, but in OCR output there is some word that is not detected,
how can I get the coordinates like in the heatmap? or how can I improve it so I can detect all the words?

thank you.

[felixdittrich92]

Hi @RickyGunawan09 👋,
thanks for reporting this 👍

Have you tested it with the last doctr release locally if the issue still exists ? (Because the HF Space demo runs on an older version of doctr)
If it would be possible could you share the original image ? So we could identify the problem on our own :)

[RickyGunawan09]

Hi @RickyGunawan09 👋, thanks for reporting this 👍

Have you tested it with the last doctr release locally if the issue still exists ? (Because the HF Space demo runs on an older version of doctr) If it would be possible could you share the original image ? So we could identify the problem on our own :)

Thank you for answering, I'm using doctr version 0.6.0a0 locally when I first encounter this issue, so then I tried in huggingface space and there is the same problem.
I'm using torch and for detection, I'm using db_resnet50

I cannot give you in public because there is some confidential part in the image, I see your linkedin but I cannot find your email so i try to connect to sent you the image @felixdittrich92 ,

but when tracing the code in class DBPostProcessor bin_thresh: float = 0.3 when I change the value from 0.3 to 0.2 or 0.1 I can get the bounding box.

and I encounter another problem or another different in local and huggingface space

when I run in local I can't get the BoundingBox but when in huggingface space I can get it.
I tried changing bin_thresh but this time it didn't help.

it is the same problem ?
thank you @felixdittrich92

[felixdittrich92]

Hi @RickyGunawan09 👋,

changes:

doctr/doctr/models/detection/differentiable_binarization/base.py

Line 36 in 3d7c812

bin_thresh: float = 0.3,

changed to 0.2 (all text in your examples are detected)
I have tested all 3 examples you send me on Linkedin and after this minor change it looks good to me
(Feel free to test it also with the PyTorch implementations which works a bit faster and maybe a bit better)
Don't compare it with the HF demo which runs on version 0.4.1 and we are currently on 0.6.0 (release) and 0.6.1 (dev) :)

from doctr.io import DocumentFile
from doctr.models import ocr_predictor


doc = DocumentFile.from_images('/home/felix/Desktop/Test_3.png')
predictor = ocr_predictor('db_resnet50', 'crnn_mobilenet_v3_small', pretrained=True)
result = predictor(doc)
result.show(doc)

[RickyGunawan09]

Hi @RickyGunawan09 👋,

changes:

doctr/doctr/models/detection/differentiable_binarization/base.py

Line 36 in 3d7c812

bin_thresh: float = 0.3,

changed to 0.2 (all text in your examples are detected) I have tested all 3 examples you send me on Linkedin and after this minor change it looks good to me (Feel free to test it also with the PyTorch implementations which works a bit faster and maybe a bit better) Don't compare it with the HF demo which runs on version 0.4.1 and we are currently on 0.6.0 (release) and 0.6.1 (dev) :)

from doctr.io import DocumentFile
from doctr.models import ocr_predictor


doc = DocumentFile.from_images('/home/felix/Desktop/Test_3.png')
predictor = ocr_predictor('db_resnet50', 'crnn_mobilenet_v3_small', pretrained=True)
result = predictor(doc)
result.show(doc)

Alright Felix I'll try to change it to bin_thresh=0.2 and thank you for your answer !!! :)

[felixdittrich92]

@frgfm do you think we should pad inputs by default in the preprocessor if inputs are smaller as default size and afterwards scale the coords back to original? It seems to lead to better results in each case 🤔

[frgfm]

I was thiking about this the other day. My point of view is that we have several options:

• training it on multi-scale (that won't fix the problem but improve robustness to scale)
• pad the input as you said
• rethink the architecture: passing an image twice as big will yield a final non-reduced feature map twice as big. And this is an issue. Ideally what we would like is having (H, W) input --> (a, b) feat map, and (2H, 2W) input --> (a, b).
• Another aspect: being able to assess how the input image should be considered (is it a crop or full image)

We can try padding but I think there are two problems:

• first, the receptive field might change the quality of predictions because it will see zeros rather than a scene
• second, this increases RAM & latency for nothing

[RickyGunawan09]

@felixdittrich92 I finally fix this using parameter preserve_aspect_ratio = True in ocrpredictor. if someone came across problem like this try using this param and see if this param fixes it or not. thank you 😄