[Bug]: Load failed and search failed after milvus restarted

[zhuwenxing]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:2.6-20251119-f288f698-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

loaded failed

[2025-11-19T17:22:13.432Z]         # release and reload with changed replicas

[2025-11-19T17:22:13.432Z]         collection_w.release()

[2025-11-19T17:22:13.432Z]         replica_number = 1

[2025-11-19T17:22:13.432Z]         if replicas_loaded in [0, 1] and len(ms.query_nodes) >= 2:

[2025-11-19T17:22:13.432Z]             replica_number = 2

[2025-11-19T17:22:13.432Z] >       collection_w.load(replica_number=replica_number)

[2025-11-19T17:22:13.432Z] 

[2025-11-19T17:22:13.432Z] testcases/test_action_second_deployment.py:220: 

[2025-11-19T17:22:13.432Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

[2025-11-19T17:22:13.432Z] /usr/local/lib/python3.10/dist-packages/pymilvus/orm/collection.py:430: in load

[2025-11-19T17:22:13.432Z]     conn.load_collection(

[2025-11-19T17:22:13.432Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:254: in handler

[2025-11-19T17:22:13.432Z]     raise e from e

[2025-11-19T17:22:13.432Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:250: in handler

[2025-11-19T17:22:13.432Z]     return func(*args, **kwargs)

[2025-11-19T17:22:13.432Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:297: in handler

[2025-11-19T17:22:13.432Z]     return func(self, *args, **kwargs)

[2025-11-19T17:22:13.432Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:195: in handler

[2025-11-19T17:22:13.432Z]     raise e from e

[2025-11-19T17:22:13.432Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:165: in handler

[2025-11-19T17:22:13.432Z]     return func(*args, **kwargs)

[2025-11-19T17:22:13.432Z] /usr/local/lib/python3.10/dist-packages/pymilvus/client/grpc_handler.py:1335: in load_collection

[2025-11-19T17:22:13.432Z]     check_status(response)

[2025-11-19T17:22:13.432Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

[2025-11-19T17:22:13.432Z] 

[2025-11-19T17:22:13.432Z] status = error_code: UnexpectedError

[2025-11-19T17:22:13.432Z] reason: "call query coordinator LoadCollection: when load 2 replica count: service resourc...Collection: when load 2 replica count: service resource insufficient[currentStreamingNode=1][expectedStreamingNode=2]"

[2025-11-19T17:22:13.432Z] 

[2025-11-19T17:22:13.432Z] 

[2025-11-19T17:22:13.432Z]     def check_status(status: Status):

[2025-11-19T17:22:13.432Z]         if status.code != 0 or status.error_code != 0:

[2025-11-19T17:22:13.432Z] >           raise MilvusException(status.code, status.reason, status.error_code)

[2025-11-19T17:22:13.432Z] E           pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=call query coordinator LoadCollection: when load 2 replica count: service resource insufficient[currentStreamingNode=1][expectedStreamingNode=2])>

search failed

[2025-11-19T17:22:13.436Z] >       collection_w.search(vectors_to_search[:default_nq], default_search_field,

[2025-11-19T17:22:13.436Z]                             search_params, default_limit,

[2025-11-19T17:22:13.436Z]                             default_search_exp,

[2025-11-19T17:22:13.436Z]                             output_fields=[ct.default_int64_field_name],

[2025-11-19T17:22:13.436Z]                             check_task=check_task,

[2025-11-19T17:22:13.436Z]                             check_items={"nq": default_nq,

[2025-11-19T17:22:13.436Z]                                          "limit": default_limit})

[2025-11-19T17:22:13.436Z] 

[2025-11-19T17:22:13.436Z] testcases/test_action_second_deployment.py:151: 

[2025-11-19T17:22:13.436Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/orm/collection.py:810: in search

[2025-11-19T17:22:13.436Z]     resp = conn.search(

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:254: in handler

[2025-11-19T17:22:13.436Z]     raise e from e

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:250: in handler

[2025-11-19T17:22:13.436Z]     return func(*args, **kwargs)

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:297: in handler

[2025-11-19T17:22:13.436Z]     return func(self, *args, **kwargs)

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:195: in handler

[2025-11-19T17:22:13.436Z]     raise e from e

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/decorators.py:165: in handler

[2025-11-19T17:22:13.436Z]     return func(*args, **kwargs)

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/client/grpc_handler.py:975: in search

[2025-11-19T17:22:13.436Z]     return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/client/grpc_handler.py:910: in _execute_search

[2025-11-19T17:22:13.436Z]     raise e from e

[2025-11-19T17:22:13.436Z] /usr/local/lib/python3.10/dist-packages/pymilvus/client/grpc_handler.py:899: in _execute_search

[2025-11-19T17:22:13.436Z]     check_status(response.status)

[2025-11-19T17:22:13.436Z] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

[2025-11-19T17:22:13.436Z] 

[2025-11-19T17:22:13.436Z] status = error_code: UnexpectedError

[2025-11-19T17:22:13.436Z] reason: "failed to search: loaded collection do not found any channel in target, may be in...ction do not found any channel in target, may be in recovery: collection on recovering[collection=462309007702185671]"

[2025-11-19T17:22:13.436Z] 

[2025-11-19T17:22:13.436Z] 

[2025-11-19T17:22:13.436Z]     def check_status(status: Status):

[2025-11-19T17:22:13.436Z]         if status.code != 0 or status.error_code != 0:

[2025-11-19T17:22:13.436Z] >           raise MilvusException(status.code, status.reason, status.error_code)

[2025-11-19T17:22:13.436Z] E           pymilvus.exceptions.MilvusException: <MilvusException: (code=106, message=failed to search: loaded collection do not found any channel in target, may be in recovery: collection on recovering[collection=462309007702185671])>

sn and qn all works well and numbers all >=2

pulsar-cluster-reinstall-3337-milvus-datanode-655467594f-29vcq    1/1     Running                  1 (28m ago)     28m     10.104.25.92    4am-node30   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-datanode-655467594f-9p7xd    1/1     Running                  1 (28m ago)     28m     10.104.13.195   4am-node16   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-mixcoord-78d758cdc8-mh9jj    1/1     Running                  1 (28m ago)     28m     10.104.19.48    4am-node28   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-proxy-5566f84f46-kkrbz       1/1     Running                  1 (28m ago)     28m     10.104.19.49    4am-node28   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-querynode-f856cd968-8rq8c    1/1     Running                  2 (27m ago)     28m     10.104.18.22    4am-node25   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-querynode-f856cd968-j5vlp    1/1     Running                  1 (28m ago)     28m     10.104.14.197   4am-node18   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-querynode-f856cd968-pfhhp    1/1     Running                  1 (28m ago)     28m     10.104.19.50    4am-node28   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-streamingnode-5d684cf2z9xv   1/1     Running                  2 (18m ago)     28m     10.104.30.250   4am-node38   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-streamingnode-5d684cflmz9m   1/1     Running                  1 (28m ago)     28m     10.104.25.93    4am-node30   <none>           <none>

Expected Behavior

No response

Steps To Reproduce

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/deploy_test_for_release_cron/detail/deploy_test_for_release_cron/3337/pipeline

cluster: 4am
ns:chaos-testing
pods

pulsar-cluster-reinstall-3337-milvus-datanode-655467594f-29vcq    1/1     Running                  1 (28m ago)     28m     10.104.25.92    4am-node30   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-datanode-655467594f-9p7xd    1/1     Running                  1 (28m ago)     28m     10.104.13.195   4am-node16   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-mixcoord-78d758cdc8-mh9jj    1/1     Running                  1 (28m ago)     28m     10.104.19.48    4am-node28   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-proxy-5566f84f46-kkrbz       1/1     Running                  1 (28m ago)     28m     10.104.19.49    4am-node28   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-querynode-f856cd968-8rq8c    1/1     Running                  2 (27m ago)     28m     10.104.18.22    4am-node25   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-querynode-f856cd968-j5vlp    1/1     Running                  1 (28m ago)     28m     10.104.14.197   4am-node18   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-querynode-f856cd968-pfhhp    1/1     Running                  1 (28m ago)     28m     10.104.19.50    4am-node28   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-streamingnode-5d684cf2z9xv   1/1     Running                  2 (18m ago)     28m     10.104.30.250   4am-node38   <none>           <none>
 pulsar-cluster-reinstall-3337-milvus-streamingnode-5d684cflmz9m   1/1     Running                  1 (28m ago)     28m     10.104.25.93    4am-node30   <none>           <none>

Anything else?

No response

[zhuwenxing]

sn panic

[2025/11/19 17:03:28.135 +00:00] [DEBUG] [internal/connection.go:495] ["Write data: "] [component=pulsar] [remote_addr=pulsar://pulsar-cluster-reinstall-3337-pulsarv3-proxy:6650] [local_addr=10.104.30.250:43026]
[2025/11/19 17:03:28.138 +00:00] [DEBUG] [internal/connection.go:495] ["Write data: "] [component=pulsar] [remote_addr=pulsar://pulsar-cluster-reinstall-3337-pulsarv3-proxy:6650] [local_addr=10.104.30.250:43026]
[2025/11/19 17:03:28.146 +00:00] [DEBUG] [pipeline/embedding_node.go:134] ["pipeline embedding insert msg"] [collectionID=462309007701515271] [segmentID=462309623556475637] [insertRowNum=984] [timestampMin=462309767933067281] [timestampMax=462309767933067281]
Assert "init_flag_ == true"  => Mmap manager has not been init. at /workspace/source/internal/core/src/storage/MmapManager.h:71

[2025/11/19 17:03:28.147 +00:00] [WARN] [segments/segment.go:376] ["create segment failed"] [collectionID=462309007701515271] [partitionID=462309007701515272] [segmentID=462309623556475637] [segmentType=Growing] [level=L1] [error="Assert \"init_flag_ == true\"  => Mmap manager has not been init. at /workspace/source/internal/core/src/storage/MmapManager.h:71\n"]
[2025/11/19 17:03:28.147 +00:00] [ERROR] [delegator/delegator_data.go:114] ["failed to create new segment"] [collectionID=462309007701515271] [channel=by-dev-rootcoord-dml_14_462309007701515271v0] [replicaID=462309008008282119] [segmentID=462309623556475637] [error="Assert \"init_flag_ == true\"  => Mmap manager has not been init. at /workspace/source/internal/core/src/storage/MmapManager.h:71\n"] [stack="github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).ProcessInsert\n\t/workspace/source/internal/querynodev2/delegator/delegator_data.go:114\ngithub.com/milvus-io/milvus/internal/querynodev2/pipeline.(*insertNode).Operate\n\t/workspace/source/internal/querynodev2/pipeline/insert_node.go:114\ngithub.com/milvus-io/milvus/internal/util/pipeline.(*pipeline).process\n\t/workspace/source/internal/util/pipeline/pipeline.go:93\ngithub.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).work\n\t/workspace/source/internal/util/pipeline/stream_pipeline.go:74"]
panic: Assert "init_flag_ == true"  => Mmap manager has not been init. at /workspace/source/internal/core/src/storage/MmapManager.h:71
	

goroutine 5816 gp=0xc004a28540 m=1545 mp=0xc000175808 [running]:
panic({0x7662580?, 0xc002ec15c0?})
	/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.9.linux-amd64/src/runtime/panic.go:811 +0x168 fp=0xc00b3fba98 sp=0xc00b3fb9e8 pc=0x28589e8
github.com/milvus-io/milvus/internal/querynodev2/delegator.(*shardDelegator).ProcessInsert(0xc003d991e0, 0xc003014ba0)
	/workspace/source/internal/querynodev2/delegator/delegator_data.go:117 +0x10e5 fp=0xc00b3fbd78 sp=0xc00b3fba98 pc=0x67172e5
github.com/milvus-io/milvus/internal/querynodev2/pipeline.(*insertNode).Operate(0xc0046db640, {0x6ef6980, 0xc004e29260})
	/workspace/source/internal/querynodev2/pipeline/insert_node.go:114 +0x1fa fp=0xc00b3fbe58 sp=0xc00b3fbd78 pc=0x675275a
github.com/milvus-io/milvus/internal/util/pipeline.(*pipeline).process(0xc007c09b90?)
	/workspace/source/internal/util/pipeline/pipeline.go:93 +0x83 fp=0xc00b3fbea8 sp=0xc00b3fbe58 pc=0x674b083
github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).work(0xc0047150e0)
	/workspace/source/internal/util/pipeline/stream_pipeline.go:74 +0x105 fp=0xc00b3fbfc8 sp=0xc00b3fbea8 pc=0x674b225
github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).Start.func1.gowrap1()
	/workspace/source/internal/util/pipeline/stream_pipeline.go:130 +0x25 fp=0xc00b3fbfe0 sp=0xc00b3fbfc8 pc=0x674c125
runtime.goexit({})
	/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.9.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc00b3fbfe8 sp=0xc00b3fbfe0 pc=0x2862421
created by github.com/milvus-io/milvus/internal/util/pipeline.(*streamPipeline).Start.func1 in goroutine 5574
	/workspace/source/internal/util/pipeline/stream_pipeline.go:130 +0x8f

goroutine 1 gp=0xc000002380 m=nil [chan receive, 9 minutes]:
runtime.gopark(0x27eaafe7e24a221?, 0x7f0649670980?, 0x8?, 0x21?, 0x18?)
	/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.9.linux-amd64/src/runtime/proc.go:435 +0xce fp=0xc003b355d0 sp=0xc003b355b0 pc=0x28597ae
runtime.chanrecv(0xc0010fad90, 0x0, 0x1)
	/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.9.linux-amd64/src/runtime/chan.go:664 +0x445 fp=0xc003b35648 sp=0xc003b355d0 pc=0x27eca05
runtime.chanrecv1(0xb838370?, 0x78b62e0?)
	/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.9.linux-amd64/src/runtime/chan.go:506 +0x12 fp=0xc003b35670 sp=0xc003b35648 pc=0x27ec592
github.com/milvus-io/milvus/cmd/roles.(*MilvusRoles).Run(0xc000d11900)
	/workspace/source/cmd/roles/roles.go:554 +0x15a5 fp=0xc003b35ba8 sp=0xc003b35670 pc=0x680bfc5
github.com/milvus-io/milvus/cmd/milvus.(*run).execute(0xb89ae40, {0xc000072990, 0x3, 0x3}, 0xc001405c70)
	/workspace/source/cmd/milvus/run.go:48 +0x2ae fp=0xc003b35c68 sp=0xc003b35ba8 pc=0x6818f2e
github.com/milvus-io/milvus/cmd/milvus.RunMilvus({0xc000072990, 0x3, 0x3})
	/workspace/source/cmd/milvus/milvus.go:60 +0x1e2 fp=0xc003b35cd8 sp=0xc003b35c68 pc=0x6818be2
main.main()
	/workspace/source/cmd/main.go:102 +0x285 fp=0xc003b35f50 sp=0xc003b35cd8 pc=0x681d325
runtime.main()
	/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.9.linux-amd64/src/runtime/proc.go:283 +0x28b fp=0xc003b35fe0 sp=0xc003b35f50 pc=0x2821f6b
runtime.goexit({})
	/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.24.9.linux-amd64/src/runtime/asm_amd64.s:1700 +0x1 fp=0xc003b35fe8 sp=0xc003b35fe0 pc=0x2862421

[yanliang567]

/assign @chyezh

/unassign

[chyezh]

/assign @cqy123456

[chyezh]

seems that InitRemoteChunkManagerSingleton's error is not handled.

[2025/11/19 16:54:37.890 +00:00] [WARN] [initcore/init_core.go:628] ["InitRemoteChunkManagerSingleton failed, C Runtime Exception: [UnknownError] precheck chunk manager client failed, error:Error in ListObjects[errcode:503, exception:XMinioServerNotInitialized, errmessage:Unable to parse ExceptionName: XMinioServerNotInitialized Message: Server not initialized yet, please try again., params:params, bucket=milvus-bucket, prefix=justforconnectioncheck], configuration:[address=pulsar-cluster-reinstall-3337-minio:9000, bucket_name=milvus-bucket, root_path=file, storage_type=remote, cloud_provider=aws, iam_endpoint=, log_level=fatal, region=, useSSL=false, sslCACert=19, useIAM=false, useVirtualHost=false, requestTimeoutMs=10000, maxConnections=100, gcp_native_without_auth=false]\n"]