Key value pairs? Forms?

[ericfeunekes]

I'm curious if you have thoughts on how to use docling for things like completed government forms? E.g extract all key value pairs from a page.

It seems like a difficult problem because a table recognition model could easily get confused between tables and KV pairs. But I'm bringing it up because the extensibility of this library seems like it offers a great opportunity to build something like this that actually works, particularly with the JSON output.

I'm not aware of any specific model that can do it just yet, but even something like a moderately powerful VLM could be inserted in the pipeline somewhere to predict the KV pairs elements.

So have you thought about how to integrate this? How would you build it into the pipeline somewhere as a prediction, even using a powerful closed source model as a proof of concept initially?

[VickySekhon]

This is a great idea, I think that Docling could definitely tackle this issue by creating a specific pipeline targeting government forms as you mentioned. The technology already exists, I believe it's simply a matter of tailoring the output.

[ericfeunekes]

Yeah. It's not even necessarily government forms, but forms generally. So more like key value pair extraction.

I think the important part is differentiating between tables and KV pairs, and then figuring out what the output format would be.

Should this be an issue? It'd be a major help for some stuff I'm working on so happy to test some things out.

[PeterStaar-IBM]

@VickySekhon @ericfeunekes Yes, we are actively working on a form model for exactly this reason. Please stay tuned!

If you want to collaborate with us (eg share some of the forms you are interested in ), please let us know. We would be happy to do so.