Plain text extraction and newlines

[billziss-gh]

Question
In what situations does trafilatura output a newline when doing plain text extraction?

Explanation
I am using trafilatura to extract plain text from HTML, which I process further with spacy. Spacy segments text into sentences using punctuation, which can be a problem because headlines often do not end with a period/fullstop.

So the following problem often happens. Trafilatura extracts:

My heading
My first sentence. My second sentence.
Some more text.

Spacy then sees the following sentences:

My heading My first sentence.
My second sentence.
Some more text.

An obvious fix would be to enforce a sentence break wherever there is a newline. However this assumes that newlines are only used to separate headings, paragraphs, etc. and cannot appear otherwise.