SpaCy transformers support for LayoutLM models

[timothyjlaurent]

Hi all,

It would be great if we could use LayoutLM models as the transformer component in spacy pipelines. There are some differences in how the LayoutLM models work that make it incompatible with SpacyTransformers.

1. LayoutLM uses a "Processor" which does tokenization on both words and bounding boxes and Image processing. If not provided with bounding boxes it will also run OCR with tesseract
2. when calling the processor the image, boundingboxes, and words must be provided to generate embedding.

I've started to look at parts of SpaCyTransformers could be cloned and altered to to support these layoutlm type of models.

Currently, I'm adding the image and bounding boxes as extension attributes to the doc and then use a span extension to grab them when making spans:

def install_extensions():
    if not Doc.has_extension("image"):
        Doc.set_extension("image", default=None)
        Doc.set_extension("bbox", default=None)
        Span.set_extension("image", getter=lambda span: span.doc._.image)
        Span.set_extension("bbox", getter=lambda span: span.doc._.bbox[span.start: span.end])
        Doc.set_extension(DOC_EXT_ATTR, default=None)

And then creating the Docs directly from our internal LayoutLM model

def load_docs_from_layoutlm_inputs(nlp: Language, inputs: LayoutLMRawInputs) -> List[Doc]:
    install_extensions()
    docs = []

    for words, bbox, image, image_path, _id in zip(inputs.tokens, inputs.bbox, inputs.image, inputs.image_path, inputs.id):
        doc = Doc(nlp.vocab, words=words, user_data=dict(image_path=image_path, id=_id))
        doc._.image = image
        doc._.bbox = bbox
        docs.append(doc)

    return docs

Wonder if there are any thoughts or guidance on how to best support LayoutLM models in SpaCy.

[ljvmiranda921]

Hi @timothyjlaurent , the extension you're describing may be out of scope in spaCy, but we think that it's a great addition to the spaCy Universe. This can be in a form of a custom component that users can install and import.

Wonder if there are any thoughts or guidance on how to best support LayoutLM models in SpaCy.

• The Doc extensions will only store the bounding box and image information, but it still won't inform training. You might need to write your own version of the training loop to actually finetune the LayoutLM model.
• If you're interested in annotation, I did a small project exploring how to use LayoutLM with Prodigy. There might be a place there where custom extensions will be more helpful.