Using TrOCR in DocTR

[antecessor]

Recently a model released in hugging face by Microsoft that do OCR using transformers. I am wondering can we do text detection using DocTR, then apply TrOCR for text recognition. If it is possible that code base improve to have this ability, that would be wonderful.

[fg-mindee]

Hi @antecessor 👋

Thanks for opening a topic about this! We're planning on integrating TrOCR at some point in docTR indeed. But regarding your question specifically, I'm not sure I understand?
What do you mean by "do text recognition using DocTR, then apply TrOCR for text extraction"?
docTR does end-to-end OCR (1 module for text detection/localization, then 1 module for text recognition), so they perform the same task. If you meant doing text detection with docTR, then using TrOCR for text recognition, that's entirely possible yes :)

[antecessor]

Sorry, I updated terms in my original question.

[fg-mindee]

No worries!

So yes, this is entirely possible:

from doctr.io import DocumentFile
from doctr.models import detection_predictor
predictor = detection_predictor(pretrained=True)
doc = DocumentFile.from_pdf("path/to/your/doc.pdf").as_images()
out = predictor(doc)

The result will be a list of N elements, where N is the number of pages in your source doc, and each element is a numpy ndarray of shape (*, 4). You can then do what you please with the localization to crop the images and send them through TrOCR for instance 👍

Let me know if this is unclear!

[fg-mindee]

Hi @antecessor 👋

Just checking: did my previous message answer your question?

Best

[antecessor]

Yes. Thank you.

[felixdittrich92]

@antecessor
@fg-mindee
I have do some tests with the TrOCR model and realized that the combination from ViT-Encoder (init with BeiT pretrained weights) and Roberta (large pretrained) as Decoder is powerful for printed and handwritten BUT it is also very slow ~600ms / text line on a V100 GPU ... wdyt ?
I try currently to train a smaller prototype of this a bit modified ( i use deit_distilled_base_patch_224 weights for the ViT Encoder) and BERT (multilingual pretrained) as Decoder. If i have some benchmarks i will add this here for the moment i have to take time to collect/generate more synthetic data.
The first test run was on only 20K data and has reached a CER-Score from ~0.25

For the moment i think the ViTSTR model would be a better fit :)
you can check out the original repo if you want : https://github.com/roatienza/deep-text-recognition-benchmark

[fg-mindee]

We'll definitely investigate the integration of TrOCR, but this discussion is more about whether docTR subcomponents can be used along with TrOCR :)

[kforcodeai]

Guys I created space similar to @fg-mindee DocTr space, or say a clone of it, which uses TrOCr for recognition and dbresnet50 for detection, check it out
https://huggingface.co/spaces/Kforcode/Doctr_plus_TrOCR
as mentioned by @felixdittrich92 its slow over cpu

[fg-mindee]

Thanks for that @K-for-Code 🙏
It seems that the URL isn't working anymore, have you changed it?

[Mohammed20201991]

Could you let me know if the URL doesn't exist? @kforcodeai

[felixdittrich92]

@K-for-Code
nice work thanks for sharing 😄
if you want you can share this also in Slack showcases 👍
Would be great if you can add also the TrOCR small models for test purposes (if it is available) !?
huggingface/transformers#14893