[Bug]: [benchmark][cluster] create partition raises error failed to store partitions: context deadline exceeded in concurrent DQL & multi-partition scene #42874
Description
Is there an existing issue for this?
- I have searched the existing issues
Environment
- Milvus version:master-20250619-59366297-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar
- SDK version(e.g. pymilvus v2.0.0rc2):2.6.0rc125
- OS(Ubuntu or CentOS):
- CPU/Memory:
- GPU:
- Others:Current Behavior
argo task: json-path-corn-1750381200
test case name: test_json_path_locust_dql_dml_partitions_streaming_cluster
server:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
json-path-corn-1750381200-6-etcd-0 1/1 Running 0 3h41m 10.104.24.125 4am-node29 <none> <none>
json-path-corn-1750381200-6-etcd-1 1/1 Running 0 3h41m 10.104.19.71 4am-node28 <none> <none>
json-path-corn-1750381200-6-etcd-2 1/1 Running 0 3h41m 10.104.30.45 4am-node38 <none> <none>
json-path-corn-1750381200-6-milvus-datanode-546f5bff69-kzf9z 1/1 Running 3 (3h40m ago) 3h41m 10.104.18.99 4am-node25 <none> <none>
json-path-corn-1750381200-6-milvus-datanode-546f5bff69-mdxrm 1/1 Running 3 (3h40m ago) 3h41m 10.104.15.106 4am-node20 <none> <none>
json-path-corn-1750381200-6-milvus-datanode-546f5bff69-p2pkt 1/1 Running 3 (3h40m ago) 3h41m 10.104.34.211 4am-node37 <none> <none>
json-path-corn-1750381200-6-milvus-mixcoord-7577d45bcf-dfzz5 1/1 Running 3 (3h40m ago) 3h41m 10.104.9.74 4am-node14 <none> <none>
json-path-corn-1750381200-6-milvus-proxy-69b5cc999f-9d2tr 1/1 Running 3 (3h40m ago) 3h41m 10.104.32.4 4am-node39 <none> <none>
json-path-corn-1750381200-6-milvus-querynode-7dc9bb47d6-h7c9b 1/1 Running 4 (3h39m ago) 3h41m 10.104.32.5 4am-node39 <none> <none>
json-path-corn-1750381200-6-milvus-streamingnode-7556cbb6fscvvn 1/1 Running 4 (3h39m ago) 3h41m 10.104.9.70 4am-node14 <none> <none>
json-path-corn-1750381200-6-minio-0 1/1 Running 0 3h41m 10.104.19.69 4am-node28 <none> <none>
json-path-corn-1750381200-6-minio-1 1/1 Running 0 3h41m 10.104.24.131 4am-node29 <none> <none>
json-path-corn-1750381200-6-minio-2 1/1 Running 0 3h41m 10.104.30.43 4am-node38 <none> <none>
json-path-corn-1750381200-6-minio-3 1/1 Running 0 3h41m 10.104.16.30 4am-node21 <none> <none>
json-path-corn-1750381200-6-pulsarv3-bookie-0 1/1 Running 0 3h41m 10.104.19.74 4am-node28 <none> <none>
json-path-corn-1750381200-6-pulsarv3-bookie-1 1/1 Running 0 3h41m 10.104.24.134 4am-node29 <none> <none>
json-path-corn-1750381200-6-pulsarv3-bookie-2 1/1 Running 0 3h41m 10.104.16.34 4am-node21 <none> <none>
json-path-corn-1750381200-6-pulsarv3-bookie-init-h2xp5 0/1 Completed 0 3h41m 10.104.19.42 4am-node28 <none> <none>
json-path-corn-1750381200-6-pulsarv3-broker-0 1/1 Running 0 3h41m 10.104.24.95 4am-node29 <none> <none>
json-path-corn-1750381200-6-pulsarv3-broker-1 1/1 Running 0 3h41m 10.104.19.44 4am-node28 <none> <none>
json-path-corn-1750381200-6-pulsarv3-proxy-0 1/1 Running 0 3h41m 10.104.24.104 4am-node29 <none> <none>
json-path-corn-1750381200-6-pulsarv3-proxy-1 1/1 Running 0 3h41m 10.104.19.45 4am-node28 <none> <none>
json-path-corn-1750381200-6-pulsarv3-pulsar-init-cvmft 0/1 Completed 0 3h41m 10.104.19.43 4am-node28 <none> <none>
json-path-corn-1750381200-6-pulsarv3-recovery-0 1/1 Running 0 3h41m 10.104.24.98 4am-node29 <none> <none>
json-path-corn-1750381200-6-pulsarv3-zookeeper-0 1/1 Running 0 3h41m 10.104.24.126 4am-node29 <none> <none>
json-path-corn-1750381200-6-pulsarv3-zookeeper-1 1/1 Running 0 3h41m 10.104.19.70 4am-node28 <none> <none>
json-path-corn-1750381200-6-pulsarv3-zookeeper-2 1/1 Running 0 3h41m 10.104.30.44 4am-node38 <none> <none>
client log:
[2025-06-20 03:23:11,259 - DEBUG - fouram]: (api_request) : [Partition] args: [<Collection>:
-------------
<name>: fouram_EV15iSUe
<description>:
<schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}, {'name': 'float_vector_1', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 768}}, {'name': 'json_1', 'description': '', 'type': <DataType.JSON: 23>, 'nullable': True}, {'name': 'json_2', 'description': '', 'type': <DataType.JSON: 23>, 'nullable': True}], 'enable_dynamic_field': True}
, 'scene_test_partition_JCrvO0kl', ''], kwargs: {'client_request_id': '6aec9f5b9c83409fb6e005ed719eccbe'}, [requestId: 6aec9f5b9c83409fb6e005ed719eccbe] (api_request.py:83)
2025-06-20 03:23:33,997 [ERROR][handler]: RPC error: [create_partition], <MilvusException: (code=10001, message=failed to store partitions: context deadline exceeded)>, <Time:{'RPC start': '2025-06-20 03:23:11.761217', 'RPC error': '2025-06-20 03:23:33.997433'}> (decorators.py:140)
[2025-06-20 03:23:33,998 - ERROR - fouram]: (api_response) : [Partition] <MilvusException: (code=10001, message=failed to store partitions: context deadline exceeded)>, [requestId: 6aec9f5b9c83409fb6e005ed719eccbe] (api_request.py:58)
Expected Behavior
No response
Steps To Reproduce
concurrent test and calculation of RT and QPS
:purpose: `primary key: INT64`, shards_num=2, enabled dynamic field, DQL & DML & partitions
2 fields of different vector types, json and dynamic fields
:test steps:
1. create collection with fields:
'float_vector': 128dim
'float_vector_1': 768dim
'id': primary key type is INT64
'json_1': scalar json field, random_range[0, 1000] & None value
'json_2': scalar json field, {'id': <all cast type>} & None value
'json_dynamic_1': dynamic field, random_range[0, 1000] & None value
'json_dynamic_2': dynamic field, {'id': <all cast type>} & None value
2. build indexes:
HNSW: 'float_vector'
IVF_SQ8: 'float_vector_1'
JsonPathIndex - 'DOUBLE': 'json_1', 'json_2["id"]', 'json_dynamic_1', 'json_dynamic_2["id"]'
3. insert 5 million data into 10 partitions
4. flush collection
5. build indexes again using the same params
6. load collection
7. concurrent request:
- search
- query
- hybrid_search
- scene_insert_partition
(partition: create->insert->flush->release->drop)
- scene_test_partition
(partition: create->insert->flush->index->load->search->release->search failed->drop) <- create partition raises error
- scene_test_partition_hybrid_search
(partition: create->insert->flush->index->load->hybrid_search->release->hybrid_search failed->drop)Milvus Log
No response
Anything else?
No response