qdrant indexing

[maaly77]

Hi,
I am trying to index 1.5 M sentences with their vectors from a mistral model as well as normalizing the embeddings. I am indexing it on my server using your docker image. The indexing is taking so long 7 days now and I am not sure if it should take that long or am I doing something wrong.
Thanks

[generall]

Do you mean that mistral embeddings + vector indexing together is taking more than 7 days?

[maaly77]

yes, the indexing of 1.5M sentences vectorised by mistral embedding model is taking more than 7 days. I used https://python.langchain.com/docs/integrations/vectorstores/qdrant#on-premise-server-deployment

[generall]

do you have an estimation of which fraction of time takes vectorization and which indexing?

[maaly77]

it seems that I am running out of memory while indexing. I am using this langchain function to index 5 million sentences with their metadata. It worked with 10 sentences but once I started indexing the 5 million it takes really long time (2 weeks) and the script keep running while encountering memory issues. My server is 128 GB memory. I am currently using this script in a loop where texts are fraction of the 5 million, not sure if it will work?

qdrant = Qdrant.from_documents(
texts,
embeddings,
url=url,
api_key=None,
prefer_grpc=True,
collection_name="mycollection",
#force_recreate=True,
)

[timvisee]

You may want to disable indexing while doing bulk upserts, which could significantly increase overall performance and indexing time. Please see our documentation here: https://qdrant.tech/documentation/tutorials/bulk-upload/

You likely want to upload to disk directly too, which is also described on that page.

[maaly77]

I am using langchain to create and index vectors https://qdrant.tech/documentation/frameworks/langchain/ using mistral embeddings model (On-premise server deployment). So are you implying that this approach can't be used with large datasets? and I will need to create vectors first then upload then index ....??? There is no fast way to do that with langchain framework?

[maaly77]

I am not sure how to know the indexing time. I managed to vectorize those sentences in chromadb and they took half day and the total size was 6 GB. Here are screenshots of the structure I had for qdrant collection. Not sure where to look to answer your question

[minakshimathpal]

@maaly77 Did you manage to get a solution to this issue. I am struggling with the same issue. "https://qdrant.tech/documentation/tutorials/bulk-upload/" says to upload the vectors first and then update the collection but even after doing that "indexed_vector_count" show zero inside my collection. @timvisee Any views/input on this please

[maaly77]

But I managed to index vectors locally using langchain library
Collection name: x
Points count: 45431
Vectors count: None
Indexed vectors count: 40448
Segments count: 8
An error occurred: 'dict' object has no attribute 'size'

[timvisee]

Could you elaborate on how you connect to, and set up, Qdrant through LangChain?

[maaly77]

@timvisee I am using this approach https://qdrant.tech/documentation/frameworks/langchain/
On-premise server deployment

[kirianguiller]

@timvisee Qdrant local mode does not support indexes and is bare bones on purpose.

The documentation that suggests to enable indexing later is for use with Qdrant server.

You can get an instance free of charge over at our cloud: https://cloud.qdrant.io/

So it is practically difficult to use Qdrant local mode for database having more than millions of embeddings?

I

[maaly77]

it is very slow with langchain given we can't use the batch mode but it is doable. it also depends on the model size and context length.