Detecting page breaks in markdown output

[AdityaMannu1709]

Is there any way to detect page breaks in markdown output of a PDF?
For example, in PyMuPDF4LLM library 'n\n-----\n\n' were used in markdown output to detect page breaks.....Can see one of that output below -

[dolfim-ibm]

This is an interesting feature request. Let us put it in the pipeline after the upcoming release.
We are already using comments <!-- --> for tagging images. I think we could easily do the same to signal page breaks. And potentially, having the user specifying the preferred placeholder for it.

[AdityaMannu1709]

yes it would be great then

[AdityaMannu1709]

Just a more adding to this feature request , just like doctags if its possible to identify code tags, table and images captions in markdown also bcz currently they are part of normal markdown text flow.

[laxas]

It would be usefull to have the page number in that "page break comment".

[rmenziejr]

I manually did this by looking at the page elements count and exporting to markdown from the beginning to end of each page element. you could probably implement this more elegantly on the backend but here is example:

from itertools import accumulate
...
result = converter.convert()
page_element_count = [1+len(i.assembled.elements) for i in result.pages]
element_page_cutoff = list(accumulate([1] + page_element_count))
pages = [
        (
            result
            .document
            .export_to_markdown(
                from_element=s,
                to_element=e
            )
        )  
        for s,e in zip(element_page_cutoff[:-1], element_page_cutoff[1:])
]
full_text = print("\n\n<!-- new page -->\n\n".join(pages))

[pat-ben]

Page break is really important indeed!
Is there any visibility to release this feature?

(In PyMuPDF4LLM, the page break looks like this: "n\n-----\n" (2 empty lines, then 5 hyphens, then 1 empty line).

[rmenziejr]

from itertools import accumulate
...
result = converter.convert()
page_element_count = [1+len(i.assembled.elements) for i in result.pages]
element_page_cutoff = list(accumulate([1] + page_element_count))
pages = [
(
result
.document
.export_to_markdown(
from_element=s,
to_element=e
)
)
for s,e in zip(element_page_cutoff[:-1], element_page_cutoff[1:])
]
full_text = print("\n\n\n\n".join(pages))

[Tonaras]

Awesome work, Rmenzie! This works quite well when the input file is a PDF. I couldn't make this work with docx files that have page breaks though. I guess it's a limitation with Docling's current docx parser. Any clues are highly appreciated until the team implements proper page-break detection in an upcoming version (hopefully soon, fingers crossed).

[ghost]

Haven’t worked with docx yet but will be shortly so I will let you know if I find the modification

[simjak]

Is this implemented in the markdown exporte or a custom-made solution like above is still needed?
@dolfim-ibm

[789AMAL]

@dolfim-ibm , Please take a look at #762.

[Tonaras]

I refused to believe that this awesome library cannot properly detect page break tags in docx input files. I've spent endless hours to figure out a way but nothing works. This is an absolute necessity for so many use cases and in desperation I'm begging this great team to implement pagination as soon as possible! Ideally, a custom defined "page break"-tag should be inserted in all exporters, not just in markdown.

Warmest regards to all!

[dKC-7000001]

Well to add page breaks in the markdown you can achieve it simply adding a argument of page_break_placeholder while exporting to markdown.

All docling dependency versions

docling==2.28.2
docling-core==2.24.0
docling-ibm-models==3.4.0
docling-parse==4.0.0

from docling.backend.docling_parse_v2_backend import DoclingParseV2DocumentBackend
from docling.datamodel.pipeline_options import PdfPipelineOptions

source = "your_document_path"

converter = DocumentConverter(
    format_options={
        InputFormat.PDF: PdfFormatOption(
            backend=DoclingParseV2DocumentBackend,  
            # pipeline_options=PdfPipelineOptions(artifacts_path=model_dir)
        )
    }
)
result = converter.convert(source)
markdown_text = result.document.export_to_markdown(page_break_placeholder="<!-- page break -->")
print(markdown_text )

Documentation link :- https://docling-project.github.io/docling/reference/docling_document/#docling_core.types.doc.DoclingDocument.export_to_markdown

[rmenziejr]

This appears to be an updated feature? I do not recall this on the original docs when this thread began but haven't confirmed.

[vagenas]

Page breaks in Markdown export (#309) was added in docling-project/docling-core#213, released with docling-core v2.24.0.

[jai-uparkar]

@vagenas I tried the "page_break_placeholder" feature last week and it was not working:
#1260

[vagenas]

@jai-uparkar, I just provided an answer: #1260 (comment)

[molokanov50]

I tried the following code:

import os
from docling.document_converter import DocumentConverter

def run(input_dir, output_dir, converter):
    for f in os.listdir(input_dir):
        outFileName = f + '.md'
        if f.endswith('.pdf'):
            outFileName = f[0:len(f) - 4] + '.md'
        if os.path.exists(os.path.join(output_dir, outFileName)):
            print('existing path: ' + os.path.join(output_dir, outFileName))
            continue
        source = os.path.join(input_dir, f)
        result = converter.convert(source)
        my_md = result.document.export_to_markdown(page_break_placeholder='<!-- pbp -->')
        spl = my_md.split('<!-- pbp -->')

        p = 1
        head = '<!-- page 1 -->\n\n'



        with open(os.path.join(output_dir, outFileName), 'w') as outFile:
            outFile.write(head)
            if len(spl) > 0:
                outFile.write(spl[0])
                for i in range(1, len(spl)):
                    p += 1
                    outFile.write(f'<!-- page {p} -->')
                    outFile.write(spl[i])



conv = DocumentConverter()
run('/home/molokanov/my_input', '/home/molokanov/my_output', conv)

To say it briefly, I parse the resulting text and replace the common page break placeholder by an expression with page number.
I noticed that the pagination is incorrect. This fact is clearly visible on large documents. One of my input PDFs included 540 pages, and the result consists only of 538 pages. More than one-page difference between he input and output is present already on page 181.
Is the pagination with the use of page_break_placeholder intended as strict? Or approximate? How to make it strict, in the full accordance with the input PDF? Any ideas?

[rmenziejr]

@rmenziejr I've tried your code as well on my 540-page PDF. The resulting markdown also includes 540 pages, but the text in it is shorter than the input text. The rest of the text is just truncated. In the middle of the document, there is a 2-page difference between the original and the markdown, despite the page numbers being the same.

@molokanov50 If that is the case then the backend that reads the pdf is not capturing all the pages. You could try switching the pdf reader backend and see if that helps.
This appears to be an issue with the ability to read your pdf in general and not specific to the page break component specifically.

[molokanov50]

@rmenziejr I've tried the alternative reader backend and I'd like to share more precise observations.
On both default and DoclingParseV2DocumentBackend reader backends, the result contains 540 pages, that is equal to the number of pages in the source pdf. But actually, only 84% of the source text is contained in the markdown:

• on page 181 of the markdown, the actual text is from page 160 of the source;
• on page 540 of the markdown, the actual text is from page 454 of the source.

What should I try next, in order to eliminate this problem?

[molokanov50]

@rmenziejr I have one more important observation. When I use neither page_break_placeholder option nor your iterating over result.pages approach at all, the complete text from the input is included in the markdown, and when I use any of this approaches, then the markdown contains only 84% of the input text.

[rmenziejr]

I would submit an issue then. From that standpoint the text stored inside the page contents in the markdown document object is not aligning.

[molokanov50]

@rmenziejr I returned back to page_break_placeholder option since it operates more precisely than the iterating over result.pages. I've found an error. It skips empty pages in the source pdf and doesn't write two or more page breaks in a row. As far as I understand, there are no additional setups to this option currently. I've submitted this issue.