[Bug]: Scalar Index Construction Issues.

[ZhangFlyer]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: v2.5.1
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.5.3
- OS(Ubuntu or CentOS): Ubuntu
- CPU/Memory: Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz/251G
- GPU: NVIDIA RTX A6000
- Others:

Current Behavior

I have designed nine queries to test filtered search on milvus, the scalar part of the operation is equal range and contain respectively, when I build indexes on the corresponding scalar columns, there is no change in the query results.
Is it that the indexes are not successfully constructed or the indexes themselves do not yield anything?

Before the index is built

query1 - latency: 0.0120 s, recall: 1.0000, param: 100
query2 - latency: 0.0169 s, recall: 0.9900, param: 100
query3 - latency: 0.3246 s, recall: 1.0000, param: 100
query4 - latency: 0.0096 s, recall: 1.0000, param: 100
query5 - latency: 0.0131 s, recall: 1.0000, param: 100
query6 - latency: 0.2734 s, recall: 1.0000, param: 100
query7 - latency: 0.0099 s, recall: 1.0000, param: 100
query8 - latency: 0.0178 s, recall: 1.0000, param: 100
query9 - latency: 0.3738 s, recall: 1.0000, param: 100

After the index is built

query1 - latency: 0.0109 s, recall: 1.0000, param: 100
query2 - latency: 0.0167 s, recall: 0.9900, param: 100
query3 - latency: 0.3500 s, recall: 1.0000, param: 100
query4 - latency: 0.0096 s, recall: 1.0000, param: 100
query5 - latency: 0.0136 s, recall: 1.0000, param: 100
query6 - latency: 0.2754 s, recall: 1.0000, param: 100
query7 - latency: 0.0092 s, recall: 1.0000, param: 100
query8 - latency: 0.0171 s, recall: 1.0000, param: 100
query9 - latency: 0.4698 s, recall: 1.0000, param: 100

Is there any way to confirm that the scalar index has been built successfully and can be used?

Expected Behavior

No response

Steps To Reproduce

Milvus Log

No response

Anything else?

No response

[yanliang567]

@ZhangFlyer

1. how many data did you insert into the collection? if it was less than 1024, it could be no index built in milvus.
2. please test on latest milvus v2.5.15, which has optimize a lot in scalar index
3. building index usually leads to be faster in searching, but it is not 100% depending on different data type and cardinality.

/assign @ZhangFlyer
/unassign

[ZhangFlyer]

@ZhangFlyer

1. how many data did you insert into the collection? if it was less than 1024, it could be no index built in milvus.
2. please test on latest milvus v2.5.15, which has optimize a lot in scalar index
3. building index usually leads to be faster in searching, but it is not 100% depending on different data type and cardinality.

/assign @ZhangFlyer /unassign

Thank you very much for your answer, the amount of data in my collection is 10M, so the first problem should not exist. Next I will test it on the latest version of milvus v2.5.15.

[xiaofan-luan]

1. if you care about latency, please make sure you are not using strong because it introduce another 200-400 ms latency in search.
2. can you share what all the queries is? especially for Q3, Q6 and Q9

[ZhangFlyer]

1. Thank you very much. I noticed this issue and set the consistency level to Bounded during the query process to ensure low latency.
2. I designed nine queries composed of different scalar operations and filtering rates. Specifically, Q1, Q2, and Q3 are equality operations; Q4, Q5, and Q6 are range operations; and Q7, Q8, and Q9 are contain operations. Additionally, the filtering rates for Q1, Q4, and Q7 are <50%, for Q2, Q5, and Q8 are 50%–99%, and for Q3, Q6, and Q9 are >99%.
   I noticed that the latency for Q3, Q6, and Q9 is particularly high, and I suspect this is due to the extremely high filtering rate causing the system to use a pre-filtering + brute-force search approach during filtering. I had hoped that the scalar index would accelerate the search process, but so far, I have not achieved the expected results.

[ZhangFlyer]

@ZhangFlyer

1. how many data did you insert into the collection? if it was less than 1024, it could be no index built in milvus.
2. please test on latest milvus v2.5.15, which has optimize a lot in scalar index
3. building index usually leads to be faster in searching, but it is not 100% depending on different data type and cardinality.

/assign @ZhangFlyer /unassign

I updated Milvus to version 2.5.15 and repeated the above test, but the scalar index still did not seem to have any effect.

Before the index is built

After the index is built

An additional problem arose: previously, the data type array<> supported the ARRAY_CONTAINS( , ) filter operation, but when I tried to execute this operation again in the new version, an error occurred.

So how can we determine whether the scalar index has been successfully constructed and is functioning properly?
Thank you very much for your reply.