Creating VSS fails without warning with a large database

[DDoyle1066]

I created a database with text from wikipedia's database and the process failed when I went to create the hnsw index. I don't know if there is something wrong with how I set things up but I do not get an error message; I just get a message saying that the python process was killed. The code that can be used to reproduce the error is here. I can email the files that are needed to generate the database; it would take about 2 days to generate otherwise and the process does seem to work for smaller subsets of the data.

I am on MacOS Sequoia 15.4.1. Let me know if you need any other details.

[Maxxen]

Hello!

A current limitation is that the HNSW index must fit into RAM memory. Given that it takes multiple days to generate the dataset, without knowing too much how the hnsw index works, do you think that a hnsw index over your data could feasibly fit into memory? If not my guess for the sudden exit would be that the OS kills the process as it consumes too much memory.

[DDoyle1066]

I see. I can load the entire dataset via polars although my RAM does sit at 150/192GB so that is probably it. The number of non nulls on the embedding column is 50M. The long processing time comes from generating embeddings not from trillions of rows. Is there anyway to make the indexing process use a smaller ‘batch size’ for lack of a better term or does it need to keep both the full data and the hnsw index in memory at once?

[DDoyle1066]

I don't know if this is possible but at the very least I think it would be helpful to do a quick check to see if the data will likely throw an out of memory error so that it is a bit clearer what is happening