[discussion] Plugins docTR

[felixdittrich92]

Hi 👋,

with the growing number of features which are not directly related to text detection and recognition (like object detection / language detection / and following) the repository is also growing to provide an easier way to maintain we should maybe think about an kind of plugins which provide the functionality the users want:

To explain this a bit more in detail the idea i have in mind is something like:

from doctrplugins import ObjectDetector, LanguageDetector , TableRecognizer

# doctrplugins is an repository encapsulated from doctr but easy made to plug the things you need inside

obj_det = ObjectDetector( ... )
lang_det = LanguageDetector( ... )
table_rec = TableRecognizer( ... )

model = ocr_predictor(det_arch='db_resnet50', reco_arch='crnn_vgg16_bn', pretrained=True, plugins=[obj_det, lang_det, table_rec])

We could then adjust the output according to the plugins

Some improvements i would see:

• keep doctr light and focus on text detection / recognition models (TF / PT / (ONNX inference later))
• plugins which needs a DL solution can be provided directly via ONNX no need to do framework split
• users can decide what they need

This is just a rough idea, so what do you think? @frgfm @charlesmindee @odulcy-mindee @aminemindee

[frgfm]

Passing plugins to the predictor is a very good idea 👍
However, for the sake of having a complete API, I think we should keep those plugins inside docTR (it looks like you're suggesting to create another package).

We certainly need to simplify the codebase structure for upcoming new features though. As long as those plugins don't require extra dependencies, I think it can be kept inside docTR. And that raises a question of mine:

• while object detector and table recognizers are very similar to other features, language detector either requires a large word matching DB or semantic understanding. Those last two will most likely require extra dependencies.
• but back to the full features of docTR, as long as we restitute the text with some vocab setting, do we need to output the language as well? We could easily output the best vocab candidates (i.e. I train a model on all European characters, once I finish my analysis, I check which characters have been detected, which will rule out some subvocabs)

What do you think?

[felixdittrich92]

Hi @frgfm 👋,

About your language detection related points i think (if it works well) we have a lightweight (only numpy and regex) and well performing solution soon in #1023

But this discussion is a bit more global the 'services' like obj_detection and so on was only an example the main topic is to think about how we can make new features (which are not directly related to text detection and recognition) accessable without wasting up the library :)
(And overall without the need to provide it in TF and PT ... i have had in mind to see some features more like a kind of microservices which plays inside the backend you know what i mean ? 😅 )

Yeah i was thinking about another package/packages (one for each new external service which can plugged inside the predictor and will increase the output) but yes we could store it also inside doctr maybe in an plugins folder :)

The advantage outside of doctr would be:

• users can create there own services (only a few lines inside doctr to make it accessible to the predictor)
• we would keep doctr clean to focus on OCR (all the additional stuff would be outside)

A great example to show what the idea is (no python):
https://mrousavy.com/react-native-vision-camera/docs/guides/frame-processor-plugin-list

(community plugins and VisionCamera works like the backend for these services)

[odulcy-mindee]

@felixdittrich92 I share @frgfm's point of view: I think it should remain in the same repo. The problem of plugins is that you need to update each repo to keep them up-to-date. It would be easier to update thanks to CI.

Wouldn't it be possible to let ocr_predictor as it is and to create other submodules for ObjectDetector, LanguageDetector and so on, like so:

from doctr.object_detector import ObjectDetector
from doctr.language_detector import LanguageDetector
from doctr.table_recognizer import TableRecognizer

and to use ocr_predictor in each detector or do we need to update the output of ocr_predictor for such tasks ?

We can also build some around "detector theme", as it has been done in VidGear with "Gears".

[felixdittrich92]

Hi @odulcy-mindee yes than i think it would be the way to create a plugins folder where we can store each plugin (a bit like transforms) :)
Mh i think it makes the most sense to pass plugins directly into the pipeline (ocr_predictor) there are already all informations available which we need for different plugins (obj_det / table_rec - image / lang_det - text after ocr) and we could easily update the output with the additional information we get (for example add barcode detection with coords to DocumentBuilder as Artefact)