Milvus error: "length of varchar field text exceeds max length, row number: 126, length: 105761, max length: 65535"

[dodekapod]

Document : https://arxiv.org/pdf/2501.15188

(DocSerializer pid=722) INFO     | components.indexer.loaders.pdf_loaders.marker:aload_document:245 - Processed /app/data/2501.15188v2.pdf in 1376.83s
(DocSerializer pid=722) INFO     | components.indexer.loaders.serializer:serialize_document:88 - Document serialized successfully [file_id=2501.15188v2.pdf | partition=arXiv | task_id=dc746dc61b2c19230ba271539ce77c1efc70cfcf01000000]
(Indexer pid=718) INFO     | components.indexer.chunker.chunker:split_document:195 - Starting document chunking [file_id=2501.15188v2.pdf | partition=arXiv | task_id=dc746dc61b2c19230ba271539ce77c1efc70cfcf01000000]
(Indexer pid=718) INFO     | components.indexer.chunker.chunker:split_document:245 - Document chunking completed [file_id=2501.15188v2.pdf | partition=arXiv | task_id=dc746dc61b2c19230ba271539ce77c1efc70cfcf01000000]
(MilvusDB pid=720) ERROR    | components.indexer.vectordb.vectordb:async_add_documents:375 - Unexpected error while adding a document [collection=vdb_test | database=Milvus | error=<MilvusException: (code=1100, message=length of varchar field text exceeds max length, row number: 126, length: 105761, max length: 65535: invalid parameter)>]

[dodekapod]

This could be the same problem :

(MilvusDB pid=714) openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 2048 tokens. However, you requested 5278 tokens in the input for embedding generation. Please reduce the length of the input.", 'type': 'BadRequestError', 'param': None, 'code': 400}}

I use default chunking settings.

[EnjoyBacon7]

This looks like a chunking problem, which may have been fixed since the creation of this issue. Can you confirm this is still relevant today?

[dodekapod]

@Ahmath-Gadji