Reference issue: Document Parsing capabilities performance improvement

[iirorahkonen]

Hi,

I'm using PdfPig through Kernel Memory (https://github.com/microsoft/kernel-memory) and ingesting PDF files for text embeddings. Only the text in the documents is relevant to my use cases, and I don't need the graphics. As I understand, I could technically read only the text from PDF files, but it seems that PdfPig doesn't support that.

The issue I face is a huge memory usage, about 4-6 gigs for a 15-megabyte file text and graphics. This is probably mostly caused by the huge list of operations - see the screenshot.

Does anyone have any insight on how I could get around the issue of the massive memory usage, skip parsing the graphics, or any information related to my assumptions of not needing the graphics?

[BobLd]

@iirorahkonen do you have the possibility to share the pdf document in question? I'd like to look into it as I'm currently trying to lower memory usage of PdfPig

You can look into the below on how to only process text command and avoid the rest (both file required):

• https://github.com/CalyPdf/Caly/blob/master/Caly.Pdf/PageFactories/TextLayerFactory.cs
• https://github.com/CalyPdf/Caly/blob/master/Caly.Pdf/TextLayer/TextLayerStreamProcessor.cs

EDIT: I have not used kernel-memory yet, but it looks like the above will have to be used as follow https://github.com/microsoft/kernel-memory/blob/32f5a599e6d32d81cafca83defe8ce14e9991ad5/examples/108-dotnet-custom-content-decoders/Program.cs#L10

[iirorahkonen]

Unfortunately, I cannot share the file as it's proprietary information. It is an A1 size export from Autocad. I can try to find a similar one if that helps.

Thank you for the tip on the page factories! I'll also make sure to test it out. I have already implemented a CustomPdfDecoder in Kernel Memory based on that example for debugging. I'll make sure to share any code for the implementation.

[BobLd]

@iirorahkonen no problem with not sharing. As it's from autocad, I expect a lot of path objects indeed. Have a look at what I did in Caly to only extract the text (links I provided before) - let me know if you need help there. I think you'll have good results. Don't hesitate to shares benchmarks too!

[EliotJones]

I'm thinking we should expose a Capabilities enum flag from parsing options. This would allow skipping slow performance areas when not needed by the client. These capabilities would be things like:

[Flags]
public enum PdfParsingCapabilities
{
    Text = 1 << 1,
    ExactGlyphBoundingBoxes = 1 << 2,
    Paths = 1 << 3,
    Images = 1 << 4,
    Colors = 1 << 5,
    AcroForms = 1 << 6,
    // etc
}

Then when we parse we can skip over any capabilities not requested by the caller, though we default to null which is equivalent to extracting everything we do today.

So, for example, if the user only opts in to Text and ExactGlyphBoundingBoxes we skip all processing to do with image, form, paths, colors, etc. I think this would be a fairly elegant pay-as-you-go approach to skipping performance bottlenecks.

[BobLd]

@EliotJones I like the idea. I did some benchmarking using a different approach and I believe we should also aim at skipping creating the operators too.

I have a proof of concept here https://github.com/CalyPdf/Caly/blob/master/Caly.Pdf/PageFactories/TextOnlyGraphicsStateOperationFactory.cs

I believe we should also add benchmarking

[EliotJones]

Thanks @BobLd, I'd like to address performance of creating objects for the operators in general as well, I'm not sure if there's anything in newer C# versions that would help optimize this. It's definitely an area overdue a refactor and thank you for making a start in #1062

[Numpsy]

I'm thinking we should expose a Capabilities enum flag from parsing options.

I can't remember if this was discussed somewhere in the past, but as flags are being discussed:
I have use cases where the only things I want to read from a PDF is the document info dictionary and/or the XMP, and don't need anything else, so a fastest path for that would be useful (I don't know how common a thing that use case is though).

(I think PdfSharp had such a flag in the API but that it wasn't actually implemented, though i haven't looked at that for some time)

[EliotJones]

@Numpsy I did actually include Page = 1 << 7 in the original draft post but then realised it kind of resulted in a bizarre API since everything else more-or-less is dependent on the Page flag being set. I think your use-case is addressed by not calling GetPages() since a lot of slower evaluation happens there, though there is probably a bunch of font and other resource evaluation you still incur the cost of for just opening the document.

Ideally the performance of the evaluation can be improved for all consumers by moving more of this stuff to lazy evaluation, or the NoPages setting would live separately to these flags.