Vector indexer support in GigaMap?

[hrstoyanov]

ES GigaMap already provides bitmap and lucene indexing. With AI these days, another type of indexing is required - vector embedding indexing. This is somewhat similar to Lucene, but allows for "search by similarity" and is used in RAG systems for LLMs.

There is the excellent JVector java library that can be accommodated. However, If this is of interest to the ES team, I would highly recommend looking into the latest technology - LEANN. It uses a clever trick/delayed vectorization that reduces storage requirements by 97% , because traditional engines (like JVector) require a lot of storage for multidimensional embedding (where the only option to reduce storage size is vector quantization with lose of accuracy) . It might be easier to fit into GigaMap.

This should be generic enough for many uses cases (text, image, genetic sequences, media files, etc.), and not just for AI RAG

[fh-ms]

We have been working on vector storage for ES/GigaMap for a while now. Currently, it's only used internally; it has not yet been decided how we will deliver it.

The LEANN stuff sounds very interesting. We will definitely check it out.