Setup and processing steps for VLM

[yogesh-n]

Question

...

What are the steps for using a VLM model for information extraction from a pdf. What should be done first to download and make available required models and set the path to the models. Is there any tool within the docling repository to download the required models and set the path variable ? How is a extraction pipelien defined when we have pre-existing fields as per a json structure to be extracted. The pdf is of mixed format where there are scanned areas of text and also plain textual areas.

[dosubot]

To use a Vision-Language Model (VLM) for information extraction from mixed-format PDFs in Docling, follow these steps:

1. Download and make models available:
Use the built-in tool to download required models. For example, to get the Granite Docling VLM and EasyOCR (if needed), run:

docling-tools models download granitedocling easyocr

Set the environment variable to point to your model directory:

export DOCLING_ARTIFACTS_PATH="/path/to/your/model-cache"

Docling will look for models in this path. If models are missing, Docling will raise an error rather than downloading them automatically, so make sure the path is correct and models are present. For production or containerized setups, you can mount a volume with pre-downloaded models and set the path accordingly. See more setup options here.

2. Mixed-format PDF handling:
Docling VLM pipelines process each PDF page as an image, so both scanned and digital text areas are handled automatically—no need for separate OCR steps. The VLM model extracts text and structure from the page images directly, making it robust for mixed content source.

3. Defining an extraction pipeline for JSON-structured fields:
Use the DocumentExtractor class and provide a template that defines the fields you want to extract. The template can be a JSON string, a Python dict, or a Pydantic model. For example:

from docling.document_extractor import DocumentExtractor

extractor = DocumentExtractor()
result = extractor.extract(
    source="your_file.pdf",
    template={
        "bill_no": "string",
        "total": "float"
    }
)
print(result.pages[0].extracted_data)

Or with a Pydantic model:

from pydantic import BaseModel, Field

class Invoice(BaseModel):
    bill_no: str = Field(examples=["A123"])
    total: float = Field(default=0.0)

result = extractor.extract(
    source="your_file.pdf",
    template=Invoice
)

The extraction result contains a list of pages, each with extracted data matching your template, the raw model output, and any errors example notebook.

4. CLI usage:
You can also use the CLI for batch processing:

docling convert /path/to/pdf_directory --from pdf --to doctags --vlm-model GRANITEDOCLING --output /path/to/output_dir

This will process all PDFs in the directory and output structured results source.

Troubleshooting:
If you see errors about missing models or misconfigured paths, double-check that your models are downloaded and that DOCLING_ARTIFACTS_PATH is set to the correct directory source.

Let me know if you need a more detailed example or help with advanced pipeline options!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[yogesh-n]

@dosu .. After following the above steps, when I try to run the extraction pipeline, I get the error - ValueError: Unrecognized model in C:\Users\ynars.cache\docling\models. Should have a model_type key in its config.json, or contain one of the following strings in its name: ...

[dosubot]

This error means Docling can't recognize the model in your models directory because it's missing a config.json file with a model_type key, or the folder structure doesn't match what Docling expects.

Here's how to fix it:

1. Make sure your models directory is set up correctly. Each model should be in its own subfolder named with double dashes (--) instead of slashes (/). For example, numind--NuExtract-2.0-2B for the NuExtract model.
2. Inside each model folder, you need a config.json file that includes a "model_type" key, plus all required files (like weights and processor configs).
3. Do not use the Hugging Face cache directory directly. Instead, use the CLI to download models into the correct structure:

   docling-tools models download --all -o C:\Users\ynars\.cache\docling\models

   This will create the expected layout and config files automatically.
4. Double-check that your environment variable is set to the parent directory containing all model subfolders:

   set DOCLING_SERVE_ARTIFACTS_PATH=C:\Users\ynars\.cache\docling\models
   

   (Restart your shell or environment after setting this.)

If you still see the error, check the contents of your models directory and the config.json file inside each model folder. If needed, you may need to rename folders to match the expected format (e.g., ds4sd--SmolDocling-256M-preview for SmolDocling) reference, reference, reference.

Let me know if you want to share your directory structure or config.json for a quick check!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}