Supported Languages

[Juhong-Namgung]

Hi,

What languages are supported?
Based on the code below, it seems that only four languages are supported.
In practice, Korean documents do not seem to be recognized correctly.

# python3.10/site-packages/docling/pipeline/standard_model_pipeline.py

class StandardModelPipeline(BaseModelPipeline):
    _layout_model_path = "model_artifacts/layout/beehive_v0.0.5"
    _table_model_path = "model_artifacts/tableformer"

    def __init__(self, artifacts_path: Path, pipeline_options: PipelineOptions):
        super().__init__(artifacts_path, pipeline_options)

        self.model_pipe = [
            EasyOcrModel(
                config={
                    "lang": ["fr", "de", "es", "en"],
                    "enabled": pipeline_options.do_ocr,
                }
            ),
...

Thank you

[dolfim-ibm]

We should distinguish between 1) programmatic documents and 2) scanned documents.

In the first case, we are language independent, we have tested Asian languages with good success.
In the second case, we depend on the underlying OCR engine. At the moment we have binding for EasyOCR which has support for 80+ languages. On their website you find the language parameters to provide.

We are actually extending Docling with a simpler way to change OCR backend and customize the parameters. For the moment changing the config requires you to make a new ModelPipeline object.

[EdwardSJ151]

Shouldn't code like this be enough for it to switch EasyOCR to portuguese?

ocr_options = EasyOcrOptions(lang=['pt'])# , use_gpu=True)

pipeline_options = PdfPipelineOptions()
pipeline_options.do_ocr = True
pipeline_options.do_table_structure = True
pipeline_options.table_structure_options.do_cell_matching = True
pipeline_options.ocr_options = ocr_options

doc_converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(pipeline_options=pipeline_options)
    }
)

I double checked that portuguese in EasyOCR is indentified as 'pt' and this still didnt work, all letters with accents get jumbled