Fix block processing bottleneck issues for networks with sub 1-second block times #394
Description
Describe the bug
The process_new_blocks function has a bottleneck when processing networks with sub-1-second block times (e.g., Arbitrum Sepolia). The system starts generating duplicate and out-of-order block detection errors when processing high-frequency blocks (>250 blocks/minute).
The BlockTracker::record_block method checks block order against the last recorded block in its history. However, with concurrent record_block calls, blocks may be added to the tracker's internal queue in a different order than they're numbered, causing the tracker to see "out-of-order" blocks that are actually just arriving concurrently.
Steps to reproduce
- Set up Arbitrum Sepolia network with monitor
arbitrum_sepolia_stylus.json - Use the default network configuration with
block_time_ms: 250 - Run the monitor and observe logs for several minutes
- Errors should appear showing missed blocks and out-of-order detections
Application logs
It works fine when setting block time to 1 second:
2025-10-29T16:39:31.621934Z INFO openzeppelin_monitor::services::blockwatcher::service: Started block watcher for network: arbitrum_sepolia
2025-10-29T16:39:31.621970Z INFO openzeppelin_monitor: Service started. Press Ctrl+C to shutdown
2025-10-29T16:40:00.528261Z INFO process_new_blocks: openzeppelin_monitor::services::blockwatcher::service: Processing blocks:
Last processed block: 209736002
Latest confirmed block: 209739569
Start block: 209739496 (skipped 3493 blocks)
Confirmations required: 12
Max past blocks: 73 network: "arbitrum_sepolia"
2025-10-29T16:40:04.286409Z INFO process_new_blocks: openzeppelin_monitor::services::blockwatcher::service: Processed 74 blocks in 4048ms network: "arbitrum_sepolia"
It fails when setting block time to 250ms:
2025-10-29T16:41:00.743050Z INFO process_new_blocks: openzeppelin_monitor::services::blockwatcher::service: Processing blocks:
Last processed block: 209739569
Latest confirmed block: 209739806
Start block: 209739570
Confirmations required: 12
Max past blocks: 253 network: "arbitrum_sepolia"
2025-10-29T16:41:03.136006Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739695, trace_id: cb9dae3e-850d-4ca7-be3a-93e1946dd85c, timestamp: 2025-10-29T16:41:03.135989+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.136050Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739696, trace_id: 36def967-01b4-42df-83b1-f418a74bb2ec, timestamp: 2025-10-29T16:41:03.136048+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137356Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739802, trace_id: 8dd45e42-90e5-46df-850c-752fd45aa5fd, timestamp: 2025-10-29T16:41:03.137355+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137369Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739803, trace_id: 54029899-8be8-4453-b213-6fd8405e4b7b, timestamp: 2025-10-29T16:41:03.137368+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137380Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739804, trace_id: ef53278a-bf16-4f67-8ba0-edb05416faaa, timestamp: 2025-10-29T16:41:03.137379+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137392Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Missed block 209739805, trace_id: 419e462b-4f5d-448b-8a41-3a592825bf7c, timestamp: 2025-10-29T16:41:03.137391+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137426Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739805 after 209739806, trace_id: e7236141-66c4-45dd-81a7-c8a5d22e0b57, timestamp: 2025-10-29T16:41:03.137424+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137454Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739804 after 209739805, trace_id: 94edd2b6-5e30-41d7-bbec-a972d0d567b0, timestamp: 2025-10-29T16:41:03.137452+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137480Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739803 after 209739804, trace_id: 9aa0b817-0534-466e-b859-7561a76eae7c, timestamp: 2025-10-29T16:41:03.137478+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137516Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739802 after 209739803, trace_id: c7d20418-e443-4ef7-9beb-969b5480735a, timestamp: 2025-10-29T16:41:03.137515+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137544Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739801 after 209739802, trace_id: 3144eae0-cec6-4a1d-b57e-36357d022e14, timestamp: 2025-10-29T16:41:03.137543+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137571Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739800 after 209739801, trace_id: a34ce50e-5a80-4a75-b1d0-9993ddc81762, timestamp: 2025-10-29T16:41:03.137569+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137600Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739799 after 209739800, trace_id: 6a72a2ec-f94a-4af9-a0b4-d342296e81c1, timestamp: 2025-10-29T16:41:03.137599+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137627Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739798 after 209739799, trace_id: 268f87ac-a366-4cea-bd03-686ff32c1f0d, timestamp: 2025-10-29T16:41:03.137625+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137653Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739797 after 209739798, trace_id: 34e35891-b695-44a4-a207-1262df87431c, timestamp: 2025-10-29T16:41:03.137651+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137677Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739796 after 209739797, trace_id: 7f416697-9aa7-4322-a980-e582b7299e7e, timestamp: 2025-10-29T16:41:03.137676+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137707Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739795 after 209739796, trace_id: 59c6409c-8e43-48a1-83e4-fc1bbccfb6f6, timestamp: 2025-10-29T16:41:03.137705+00:00 network: "arbitrum_sepolia"
2025-10-29T16:41:03.137729Z ERROR process_new_blocks: openzeppelin_monitor::utils::logging::error: Error occurred, Out of order or duplicate block detected: received 209739794 after 209739795, trace_id: 7c8cf84b-c767-4bbb-adc6-b1c08e8fc8d7, timestamp: 2025-10-29T16:41:03.137727+00:00 network: "arbitrum_sepolia"
Platform(s)
Mac (Apple Silicon)
Deployment Type
Built from source
Any other context that we need to know specific to the bug?
Use a private RPC provider as you'll get throttled quickly on public ones.
Version Information
1.1.0
Monitor Configuration
{
"name": "Arbitrum Sepolia Stylus",
"paused": false,
"networks": [
"arbitrum_sepolia"
],
"addresses": [
{
"address": "0xDAF4A50D2281497737BC3F4B8cF5f4c2fA3C159c",
"contract_spec": [
{
"inputs": [],
"name": "retrieve",
"outputs": [
{
"internalType": "uint256",
"name": "",
"type": "uint256"
}
],
"stateMutability": "view",
"type": "function"
},
{
"inputs": [
{
"internalType": "uint256",
"name": "num",
"type": "uint256"
}
],
"name": "store",
"outputs": [],
"stateMutability": "nonpayable",
"type": "function"
}
]
}
],
"match_conditions": {
"functions": [
{
"signature": "store(uint256)",
"expression": null
}
],
"events": [],
"transactions": [
{
"status": "Success",
"expression": null
}
]
},
"trigger_conditions": [],
"triggers": [
"evm_large_transfer_usdc_slack"
]
}Network Type
EVM
Build Command
No response
Code of Conduct
- I agree to follow the Code of Conduct