Chunking Testbed

[gotsysdba]

Look into having a “Chunking Testbed” or some way to generate a single report for evaluating the LLM generated questions via a vector store where chunks were created using different chunking strategies? Also, there are several embedding models that have built-in chunking – e.g. jina ai has late-chunking which was proven to be better.