Skip to content

[Bug]: The dataNode keeps restarting: SIGSEGV: segmentation violation and HybridSearch returned fewer fields than expected #43537

@ThreadDao

Description

@ThreadDao

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: master-20250723-74c08069-amd64
- Deployment mode(standalone or cluster): cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server

  • config
    common:
      enabledJsonKeyStats: true
      enabledOptimizeExpr: false
      storage:
        enablev2: true
    dataCoord:
      compaction:
        clustering:
          autoEnable: true
      enableActiveStandby: true
    indexCoord:
      enableActiveStandby: true
    log:
      level: debug
    queryCoord:
      enableActiveStandby: true
    rootCoord:
      enableActiveStandby: true

client test

  1. create a collection
{'auto_id': False,
 'description': '',
 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}},
            {'name': 'int32_1', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'float32_1', 'description': '', 'type': <DataType.FLOAT: 10>, 'is_clustering_key': True},
            {'name': 'varchar_1', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 100}},
            {'name': 'varchar_2', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 100, 'enable_match': True, 'enable_analyzer': True}},
            {'name': 'array_varchar_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 100, 'max_capacity': 10}, 'element_type': <DataType.VARCHAR: 21>},
            {'name': 'json_1', 'description': '', 'type': <DataType.JSON: 23>}, {'name': 'binary_vector', 'description': '', 'type': <DataType.BINARY_VECTOR: 100>, 'params': {'dim': 128}},
            {'name': 'float16_vector', 'description': '', 'type': <DataType.FLOAT16_VECTOR: 102>, 'params': {'dim': 128}}, {'name': 'bfloat16_vector', 'description': '', 'type': <DataType.BFLOAT16_VECTOR: 103>, 'params': {'dim': 128}}],
 'enable_dynamic_field': False} (base.py:329)
  1. create scalar index and vector index -> insert 2m -> flush -> index again -> load
  2. concurrent requests:
Image

results

  1. dataNode keeps restarting due to SIGV
    dn_5k554.log
Image 2. Part hybrid_search return fewer fields than expected ``` [2025-07-23 12:29:39,229 - ERROR - fouram]: [CheckFunc] Search `output_fields`:{'id'} != expected:{'float_vector', 'id', 'bfloat16_vector', 'binary_vector', 'float16_vector'} (func_check.py:313) ``` 3. queryNode panic (known issue #43372) [qn_d92r6.log](https://github.com/user-attachments/files/21399493/qn_d92r6.log) Image

Expected Behavior

No response

Steps To Reproduce

Milvus Log

zong-roll-0723-43313-a-2-milvus-datanode-77fdcff64b-5k554         0/1     CrashLoopBackOff   141 (4m55s ago)   15h     10.104.26.21    4am-node32   <none>           <none>
zong-roll-0723-43313-a-2-milvus-datanode-77fdcff64b-6r82b         1/1     Running            142 (5m36s ago)   15h     10.104.18.182   4am-node25   <none>           <none>
zong-roll-0723-43313-a-2-milvus-mixcoord-554cd968f4-4v6np         1/1     Running            0                 15h     10.104.18.180   4am-node25   <none>           <none>
zong-roll-0723-43313-a-2-milvus-proxy-694b9fc754-42szx            1/1     Running            0                 14h     10.104.26.22    4am-node32   <none>           <none>
zong-roll-0723-43313-a-2-milvus-querynode-0-fb68dbf4-4dfxr        1/1     Running            0                 15h     10.104.18.181   4am-node25   <none>           <none>
zong-roll-0723-43313-a-2-milvus-querynode-0-fb68dbf4-cf24r        1/1     Running            0                 15h     10.104.27.56    4am-node31   <none>           <none>
zong-roll-0723-43313-a-2-milvus-querynode-0-fb68dbf4-d92r6        1/1     Running            1 (14h ago)       15h     10.104.19.92    4am-node28   <none>           <none>
zong-roll-0723-43313-a-2-milvus-streamingnode-795c654dc9-87v5n    1/1     Running            0                 15h     10.104.18.178   4am-node25   <none>           <none>

Anything else?

No response

Metadata

Metadata

Assignees

Labels

kind/bugIssues or changes related a bugpriority/critical-urgentHighest priority. Must be actively worked on as someone's top priority right now.triage/acceptedIndicates an issue or PR is ready to be actively worked on.

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions