feat: add hybrid chunker

[vagenas]

Known points to fix & improve

Planned for this PR

• last chunk is disregarded due to bug in _merge_chunks_with_matching_metadata()
• introduced transformers as a docling-core dependency
• _split_by_doc_items() currently assumes DocItem has a .text i.e. assumes TextItem
• unit tests needed

TBD

• we allow too many tokens when using another delimiter like "####"
• table splitting currently text-based, can be optimized, e.g. based on row or cell
• duplication of serialization logic in new chunker implem and BaseChunker serialize(); former should reuse latter
• general refactoring of new chunker implementation needed
• we currently indeed split in the middle of sentences; clarify if issue or not
• use of external library semchunk; to be clarified
• minor: streaming is superficial

[vagenas]

Converting to draft until all points planned for this PR are addressed (see description)..

[vagenas]

_split_by_doc_items() currently assumes DocItem has a .text i.e. assumes TextItem

No direct impact after all, as the method seeks to split DocChunks with multiple DocItems (e.g. lists). Tables on the other hand are not part of such upstream DocChunks (from HierarchicalChunker) and are already handled without problems (any splitting needed is done based on text, i.e. by another method).

[vagenas]

Main changes with this PR:

1. Adds a new chunker, currently named TokenAwareChunker, which uses a hybrid approach, applying tokenization-aware refinements on top of document-layout-based (AKA hierarchical) chunking. More precisely:
   • it starts from the result of the hierarchical chunker and, based on the input tokenizer (typically to be aligned to the embedding model tokenizer), it:
   • does one pass where it splits chunks only when needed (i.e. oversized w.r.t. tokens), &
   • another pass where it merges chunks only when possible (i.e. undersized successive chunks with same headings & captions) — users can opt out of this step via param merge_peers (by default True)
2. Extends BaseChunker interface with a new method, currently named serialize(), which returns the enriched chunk serialization, by default being a concatenation of the relevant metadata (headings & captions) with the chunk's .text. This is the representation that should meet the token limitations applied by TokenAwareChunker.

👉 Any better naming ideas, particularly for the new chunker, are welcome. Some options in no particular order:

• HybridChunker
• TokenizedLayoutChunker
• TokenizedHierarchicalChunker (although it's technically not a subclass of HierarchicalChunker)
• AdaptiveChunker

[vagenas]

Finally went for HybridChunker as that appears to be the standard industry term for this approach, see e.g.:

• https://www.ibm.com/architectures/papers/rag-cookbook/chunking
• https://zilliz.com/learn/guide-to-chunking-strategies-for-rag