[Bug]: In read-write concurrent scenarios for standalone, there are many query/search/hybrid search failures, and numerous slow queries.

[zhuwenxing]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version:master-20250411-f9febe3b-amd64
- Deployment mode(standalone or cluster):standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

Although this issue was discovered during a chaos test, the request failures occurred before the fault and after the fault was resolved. During these periods, Milvus was healthy

Expected Behavior

No response

Steps To Reproduce

Milvus Log

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-cron/detail/chaos-test-cron/20881/pipeline

log:

artifacts-standalone-pod-kill-20881-server-logs.tar.gz

Anything else?

No response

[zhuwenxing]

cluster: 4am
ns: chaos-testing
pods

[2025-04-14T00:33:53.865Z] + kubectl get pods -o wide

[2025-04-14T00:33:53.866Z] + grep standalone-pod-kill-20881

[2025-04-14T00:33:54.122Z] standalone-pod-kill-20881-etcd-0                                  1/1     Running                  0                 112s    10.104.15.122   4am-node20   <none>           <none>

[2025-04-14T00:33:54.122Z] standalone-pod-kill-20881-etcd-1                                  1/1     Running                  0                 112s    10.104.18.232   4am-node25   <none>           <none>

[2025-04-14T00:33:54.122Z] standalone-pod-kill-20881-etcd-2                                  1/1     Running                  0                 112s    10.104.19.82    4am-node28   <none>           <none>

[2025-04-14T00:33:54.122Z] standalone-pod-kill-20881-milvus-standalone-656c6f9b46-758rp      1/1     Running                  1 (96s ago)       112s    10.104.23.179   4am-node27   <none>           <none>

[2025-04-14T00:33:54.122Z] standalone-pod-kill-20881-minio-55cb6947b6-vh97p                  1/1     Running                  0                 112s    10.104.15.123   4am-node20   <none>           <none>

[yanliang567]

/assign @chyezh
the one of difference here could be streaming node, please help to take a look

/unassign

[chyezh]

It seems that some operation block the insert operation into growing segment.
Make the tsafe can not be pushed forward, then the search/query is lag.

[2025/04/14 00:41:34.591 +00:00] [DEBUG] [querynodev2/handlers.go:403] ["start to search channel"] [traceID=92f48803552f12beab9d62f2ebfb1f79] [msgID=457334140277096450] [collectionID=457333996504425525] [channel=by-dev-rootcoord-dml_14_457333996504425525v1] [scope=All] [nq=5] [segmentIDs="[]"]
...
[2025/04/14 00:41:34.989 +00:00] [DEBUG] [metricsutil/wal_write.go:100] ["append message into wal"] [message_type=COMMIT_TXN] [bytes=84] [append_duration=258.104µs] [impl_append_duration=175.962µs] [presisted=true] [timetick_0="before: 31.176µs, after: 35.407µs, before_err: <nil>"] [segment-assign_0="before: 1.063µs, after: 244ns, before_err: <nil>"] [message_id=457333997746782414] [time_tick=457334140305145865] [txn_id=457333996505021316]
...
[2025/04/14 00:41:44.298 +00:00] [WARN] [delegator/delegator.go:367] ["delegator search failed to wait tsafe"] [traceID=92f48803552f12beab9d62f2ebfb1f79] [collectionID=457333996504425525] [channel=by-dev-rootcoord-dml_14_457333996504425525v1] [replicaID=457333996524666890] [error="context canceled"]
...
[2025/04/14 00:41:47.499 +00:00] [INFO] [delegator/delegator_data.go:175] ["insert into growing segment"] [collectionID=457333996504425525] [channel=by-dev-rootcoord-dml_14_457333996504425525v1] [replicaID=457333996524666890] [collectionID=457333996504425525] [segmentID=457333996504728296] [rowCount=1507] [maxTimestamp=457334140305145865]

The insert operation is too slow:

[chyezh]

The building operation of growing index is too slow.
/assign @foxspy please help to check.

I will verify the metrics of previous related test.

[zhuwenxing]

failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-cron/detail/chaos-test-cron/20881/pipeline
log:
artifacts-standalone-pod-kill-20881-server-logs (1).tar.gz
cluster: 4am
ns: chaos-testing
pods

[2025-04-14T01:04:59.008Z] + kubectl get pods -o wide

[2025-04-14T01:04:59.009Z] + grep standalone-pod-kill-20881

[2025-04-14T01:04:59.264Z] standalone-pod-kill-20881-etcd-0                                  1/1     Running                  0                  32m     10.104.15.122   4am-node20   <none>           <none>

[2025-04-14T01:04:59.264Z] standalone-pod-kill-20881-etcd-1                                  1/1     Running                  0                  32m     10.104.18.232   4am-node25   <none>           <none>

[2025-04-14T01:04:59.264Z] standalone-pod-kill-20881-etcd-2                                  1/1     Running                  0                  32m     10.104.19.82    4am-node28   <none>           <none>

[2025-04-14T01:04:59.264Z] standalone-pod-kill-20881-milvus-standalone-656c6f9b46-x44zv      1/1     Running                  3 (8m34s ago)      8m58s   10.104.23.191   4am-node27   <none>           <none>

[2025-04-14T01:04:59.264Z] standalone-pod-kill-20881-minio-55cb6947b6-vh97p                  1/1     Running                  0                  32m     10.104.15.123   4am-node20   <none>           <none>

there are some errores

                "[2025-04-14T01:22:15.631Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=fail to Query on QueryNode 14: worker(14) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyFilterBitsNode, plan node id: 37407] : Assert \"start_offset >= 0 && start_offset < row_nums_\"  => Retrieve string views with out-of-bound offset:6144, len:0, wrong at /workspace/source/internal/core/src/common/Chunk.cpp:29\r",
                "[2025-04-14T01:22:15.631Z] [2025-04-14 01:21:35 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=fail to Query on QueryNode 14: worker(14) query failed: N6milvus21ExecOperatorExceptionE :Operator::GetOutput failed for [Operator:PhyFilterBitsNode, plan node id: 37407] : Assert \"start_offset >= 0 && start_offset < row_nums_\"  => Retrieve string views with ou...... (api_request.py:36)\r",

[chyezh]

Here's a huge amount of working thread, may make the k8s throttling.