[Bug]: [woodpecker] Insert failed: quota exceeded[reason=time tick long delay

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: master-20250716-b5a19376-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka): woodpecker   
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server

• mq: woodpecker
• config

    common:
      enabledJSONKeyStats: true
      storage:
        enablev2: false
    dataCoord:
      compaction:
        clustering:
          autoEnable: true
      enableActiveStandby: true
    indexCoord:
      enableActiveStandby: true
    log:
      level: debug
    queryCoord:
      enableActiveStandby: true
    queryNode:
      enableSegmentPrune: true
      mmap:
        scalarField: true
        scalarIndex: true
        vectorField: true
        vectorIndex: true
    rootCoord:
      enableActiveStandby: true
    streaming:
      walWriteAheadBuffer:
        capacity: 1m
        keepalive: 0.5s

client test

1. create collection with schema

{'auto_id': False,
 'description': '',
 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}},
            {'name': 'int32_1', 'description': '', 'type': <DataType.INT32: 4>}, {'name': 'float32_1', 'description': '', 'type': <DataType.FLOAT: 10>},
            {'name': 'varchar_1', 'description': '', 'type': <DataType.VARCHAR: 21>, 'params': {'max_length': 100}},
            {'name': 'array_varchar_1', 'description': '', 'type': <DataType.ARRAY: 22>, 'params': {'max_length': 100, 'max_capacity': 10}, 'element_type': <DataType.VARCHAR: 21>},
            {'name': 'json_1', 'description': '', 'type': <DataType.JSON: 23>}],
 'enable_dynamic_field': False} (base.py:329)

2. create vector index
3. try to insert 2m data and failed:

[2025-07-16 07:16:17,466 -  INFO - fouram]: [Time] Collection.insert run in 0.6828s (api_request.py:50)
[2025-07-16 07:16:17,468 -  INFO - fouram]: [Base] Number of vectors in the collection(fouram_yluVVa5w): 382427 (base.py:535)
[2025-07-16 07:16:17,665 -  INFO - fouram]: [Base] Start inserting, ids: 1935000 - 1939999, data size: 2,000,000 (base.py:366)
2025-07-16 07:16:17,808 [ERROR][handler]: RPC error: [batch_insert], <MilvusException: (code=9, message=quota exceeded[reason=time tick long delay])>, <Time:{'RPC start': '2025-07-16 07:16:17.683173', 'RPC error': '2025-07-16 07:16:17.808390'}> (decorators.py:140)
[2025-07-16 07:16:17,810 - ERROR - fouram]: (api_response) : [Collection.insert] <MilvusException: (code=9, message=quota exceeded[reason=time tick long delay])>, [requestId: c3594a69fc1e4fb59c4247b3f354c5c1] (api_request.py:58)
[2025-07-16 07:16:17,810 - ERROR - fouram]: [CheckFunc] insert request check failed, response:<MilvusException: (code=9, message=quota exceeded[reason=time tick long delay])>

Expected Behavior

No response

Steps To Reproduce

Milvus Log

• https://argo-workflows.zilliz.cc/workflows/qa/zong-sn-stable-7?tab=workflow&nodeId=zong-sn-stable-7-950868751&nodePanelView=inputs-outputs
• pods:

sn-ddl-op-79-2695-etcd-0                                          1/1     Running           0               35m     10.104.24.51    4am-node29   <none>           <none>
sn-ddl-op-79-2695-etcd-1                                          1/1     Running           0               35m     10.104.21.120   4am-node24   <none>           <none>
sn-ddl-op-79-2695-etcd-2                                          1/1     Running           0               35m     10.104.19.138   4am-node28   <none>           <none>
sn-ddl-op-79-2695-milvus-datanode-54b9d874bc-6x56w                1/1     Running           0               34m     10.104.27.208   4am-node31   <none>           <none>
sn-ddl-op-79-2695-milvus-datanode-54b9d874bc-bx2tr                1/1     Running           0               34m     10.104.33.122   4am-node36   <none>           <none>
sn-ddl-op-79-2695-milvus-datanode-54b9d874bc-jb9b9                1/1     Running           0               34m     10.104.19.141   4am-node28   <none>           <none>
sn-ddl-op-79-2695-milvus-mixcoord-8f547c6b-4jlkt                  1/1     Running           0               34m     10.104.21.135   4am-node24   <none>           <none>
sn-ddl-op-79-2695-milvus-proxy-6c7b576477-g2rlg                   1/1     Running           0               34m     10.104.25.97    4am-node30   <none>           <none>
sn-ddl-op-79-2695-milvus-querynode-0-775f57d64b-lfwl8             1/1     Running           0               34m     10.104.9.244    4am-node14   <none>           <none>
sn-ddl-op-79-2695-milvus-querynode-0-775f57d64b-vn9p2             1/1     Running           0               34m     10.104.15.58    4am-node20   <none>           <none>
sn-ddl-op-79-2695-milvus-streamingnode-7b77799b9f-hkdwf           1/1     Running           0               34m     10.104.17.29    4am-node23   <none>           <none>
sn-ddl-op-79-2695-milvus-streamingnode-7b77799b9f-rsd76           1/1     Running           0               34m     10.104.27.209   4am-node31   <none>           <none>
sn-ddl-op-79-2695-minio-0                                         1/1     Running           0               35m     10.104.24.53    4am-node29   <none>           <none>
sn-ddl-op-79-2695-minio-1                                         1/1     Running           0               35m     10.104.21.122   4am-node24   <none>           <none>
sn-ddl-op-79-2695-minio-2                                         1/1     Running           0               35m     10.104.27.200   4am-node31   <none>           <none>
sn-ddl-op-79-2695-minio-3                                         1/1     Running           0               35m     10.104.18.173   4am-node25   <none>           <none>

Anything else?

No response

[tinswzy]

There is a bug in the fence segment. The fence block key used is not the last blockId + 1.

[tinswzy]

/assign @ThreadDao
/unassign