-
Notifications
You must be signed in to change notification settings - Fork 638
Open
Description
Is there an existing issue for this?
- I have searched the existing issues
Current Behavior
I want to perform a weighted BM25 search and the milvus_client code is throwing an error
Currently using 2.5.4 python sdk and milvus standalone, I've posted the docker compose and python version below under "Software version"
This is my code for searching the indexes with BM25, that individually work but errors with hybrid_search:
index_params = MilvusClient.prepare_index_params()
index_params.add_index(
field_name="sparse",
index_type="SPARSE_INVERTED_INDEX",
metric_type="BM25"
)
index_params.add_index(
field_name="sparse2",
index_type="SPARSE_INVERTED_INDEX",
metric_type="BM25"
)
# Search parameters for sparse vector search
search_param_sparse = {
"data": ["Who started AI research?"], # Your search query
"anns_field": "sparse",
"param": {
"metric_type": "BM25",
"params": {}
},
"limit": 3
}
request_sparse = AnnSearchRequest(**search_param_sparse)
search_param_sparse2 = {
"data": ["Who started AI research?"], # Your search query
"anns_field": "sparse2",
"param": {
"metric_type": "BM25",
"params": {}
},
"limit": 3
}
request_sparse2 = AnnSearchRequest(**search_param_sparse2)
results = client.hybrid_search(
collection_name="demo",
reqs=[request_sparse, request_sparse2],
ranker=WeightedRanker(0.7, 0.3), # Give 70% weight to sparse search
limit=3
)
Error Trace:
Traceback (most recent call last):
File "<LOCAL-PATH>\test_milvus_bm25.py", line 145, in <module>
results = client.hybrid_search(
collection_name="demo",
...<2 lines>...
limit=3
)
File "<LOCAL-PATH>\.venv\Lib\site-packages\pymilvus\milvus_client\milvus_client.py", line 359, in hybrid_search
ret.append([hit.to_dict() for hit in hits])
^^^^
TypeError: 'SequenceIterator' object is not iterable
This is what I changed on milvus client code to make it work:
From:
try:
res = conn.hybrid_search(
collection_name,
reqs,
ranker,
limit=limit,
partition_names=partition_names,
output_fields=output_fields,
timeout=timeout,
**kwargs,
)
except Exception as ex:
logger.error("Failed to hybrid search collection: %s", collection_name)
raise ex from ex
ret = []
for hits in res:
ret.append([hit.to_dict() for hit in hits])
return ExtraList(ret, extra=construct_cost_extra(res.cost))
To:
try:
res = conn.hybrid_search(
collection_name,
reqs,
ranker,
limit=limit,
partition_names=partition_names,
output_fields=output_fields,
timeout=timeout,
**kwargs,
)
except Exception as ex:
logger.error("Failed to hybrid search collection: %s", collection_name)
raise ex from ex
ret = []
for hits in res:
ret.append([hit.to_dict() for hit in list(hits)]) #added list() method
return ExtraList(ret, extra=construct_cost_extra(res.cost))
And it now works without throwing the error.
Is any of my dependencies missmatching or is the hybrid search code on the client bricked for this version?
Steps To Reproduce
1. Perform a Hybrid Search (I wasn't able to perform it in any way I tried)
Software version
Milvus: 2.5.4
Server: ?
Client: Python SDK 2.5.4
Name: pymilvus
Version: 2.5.4
Summary: Python Sdk for Milvus
Docker Compose:
version: '3.5'
services:
etcd:
container_name: milvus-etcd
image: quay.io/coreos/etcd:v3.5.5
environment:
- ETCD_AUTO_COMPACTION_MODE=revision
- ETCD_AUTO_COMPACTION_RETENTION=1000
- ETCD_QUOTA_BACKEND_BYTES=4294967296
- ETCD_SNAPSHOT_COUNT=50000
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/etcd:/etcd
command: etcd -advertise-client-urls=http://127.0.0.1:2379 -listen-client-urls http://0.0.0.0:2379 --data-dir /etcd
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health"]
interval: 30s
timeout: 20s
retries: 3
minio:
container_name: milvus-minio
image: minio/minio:RELEASE.2023-03-20T20-16-18Z
environment:
MINIO_ACCESS_KEY: minioadmin
MINIO_SECRET_KEY: minioadmin
ports:
- "9001:9001"
- "9000:9000"
volumes:
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/minio:/minio_data
command: minio server /minio_data --console-address ":9001"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
interval: 30s
timeout: 20s
retries: 3
standalone:
container_name: milvus-standalone
image: milvusdb/milvus:v2.5.4
command: ["milvus", "run", "standalone"]
security_opt:
- seccomp:unconfined
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
volumes:
- ./milvus.yaml:/milvus/configs/milvus.yaml
- ${DOCKER_VOLUME_DIRECTORY:-.}/volumes/milvus:/var/lib/milvus
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
interval: 30s
start_period: 90s
timeout: 20s
retries: 3
ports:
- "19530:19530"
- "9091:9091"
depends_on:
- "etcd"
- "minio"
attu:
container_name: milvus-attu
image: zilliz/attu:v2.5.2
environment:
MILVUS_URL: standalone:19530
ports:
- "8009:3000"
depends_on:
- "standalone"
networks:
default:
name: milvus
Metadata
Metadata
Assignees
Labels
No labels