RPC node can not start with error: mismatching bitmap lengths expectedBitmapLength 30 providedBitmapLength 32

[sing1ee]

Describe the bug
I start two rpc node with version v7603-v4.3.13-0-g0b342d76, and got errors:

{"level":"error","error":"mask.SetMask failed","caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:52","time":"2023-01-08T12:45:59.101088128+01:00","message":"[Explorer] readSignatureBitmapPayload failed"}
{"level":"warn","error":"mismatching bitmap lengths expectedBitmapLength 30 providedBitmapLength 32","caller":"/home/runner/work/harmony/harmony/harmony/internal/chain/sig.go:49","time":"2023-01-08T12:46:00.243516005+01:00","message":"mask.SetMask failed"}
{"level":"error","error":"mask.SetMask failed","caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:52","time":"2023-01-08T12:46:00.243535902+01:00","message":"[Explorer] readSignatureBitmapPayload failed"}
{"level":"warn","error":"mismatching bitmap lengths expectedBitmapLength 30 providedBitmapLength 32","caller":"/home/runner/work/harmony/harmony/harmony/internal/chain/sig.go:49","time":"2023-01-08T12:46:02.319129868+01:00","message":"mask.SetMask failed"}
{"level":"error","error":"mask.SetMask failed","caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:52","time":"2023-01-08T12:46:02.319166497+01:00","message":"[Explorer] readSignatureBitmapPayload failed"}
{"level":"warn","error":"mismatching bitmap lengths expectedBitmapLength 30 providedBitmapLength 32","caller":"/home/runner/work/harmony/harmony/harmony/internal/chain/sig.go:49","time":"2023-01-08T12:46:04.205382278+01:00","message":"mask.SetMask failed"}
{"level":"error","error":"mask.SetMask failed","caller":"/home/runner/work/harmony/harmony/harmony/node/node_explorer.go:52","time":"2023-01-08T12:46:04.205416512+01:00","message":"[Explorer] readSignatureBitmapPayload failed"}
{"level":"warn","error":"mismatching bitmap lengths expectedBitmapLength 30 providedBitmapLength 32","caller":"/home/runner/work/harmony/harmony/harmony/internal/chain/sig.go:49","time":"2023-01-08T12:46:06.181036256+01:00","message":"mask.SetMask failed"}

To Reproduce
Steps to reproduce the behavior:

1. Sync db
2. start harmony

Expected behavior
RPC node can start normaly

Screenshots
No.

Environment (please complete the following information):

• OS: Linux,Ubuntu20.04

[MaxMustermann2]

This error seems to occur when the network type and the zone mismatch in the configuration.

[nxet]

I'm also getting this issue, but the node has been running fine for over a week. At some point it crashed, and now it won't start again.

This error seems to occur when the network type and the zone mismatch in the configuration.

Can you please elaborate? How can this be fixed?
Also, is this comment still relevant considering my geth was running fine with the very same config?

harmony, version v7658-v2023.2.0-0-g01691fd3

thanks!

[nxet]

So, I started syncing again from a known working config and state from about 8 days ago.
After running fine for a couple hours, the node got up to block 41613066 and errored. The daemon process itself did not restart, instead geth got the legacy sync back up and running, but from this point on there's no way to sync any new block.

2023-05-18T17:06:23.498453+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44335036 shardID=1
2023-05-18T17:06:23.498591+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44517688 shardID=2
2023-05-18T17:06:23.498704+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44358024 shardID=3
2023-05-18T17:06:23.498771+02:00 INF root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:1817 > Inserted new block elapsed=84.536ms gas=81315 hash=0xeb6abeac413eb5eb3bd7eb31f6c5e964225f4e0bf9db6a218987cf92b91ef4e8 number=41613063 stakingTxs=0 txs=1 uncles=0
2023-05-18T17:06:23.498835+02:00 INF root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:1897 > Imported new chain segment age=1w10h16m blocks=1 cache="1.91 GiB" elapsed=84.631ms hash=0xeb6abeac413eb5eb3bd7eb31f6c5e964225f4e0bf9db6a218987cf92b91ef4e8 mgas=0.081315 mgasps=0.9608158920307379 number=41613063 txs=1
2023-05-18T17:06:23.498904+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/syncing.go:923 > [SYNC] UpdateBlockAndStatus: New Block Added to Blockchain ShardID=0 blockEpoch=1442 blockHeight=41613063 blockHex=0xeb6abeac413eb5eb3bd7eb31f6c5e964225f4e0bf9db6a218987cf92b91ef4e8
2023-05-18T17:06:23.498971+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/explorer/storage.go:242 > writing trace into explorer DB block hash=0xeb6abeac413eb5eb3bd7eb31f6c5e964225f4e0bf9db6a218987cf92b91ef4e8 module="explorer storage"
2023-05-18T17:06:23.594313+02:00 INF root/go/src/github.com/harmony-one/harmony/internal/chain/reward.go:295 > Commit sigs are ready
2023-05-18T17:06:23.594557+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44335037 shardID=1
2023-05-18T17:06:23.594672+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44517689 shardID=2
2023-05-18T17:06:23.594747+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44358025 shardID=3
2023-05-18T17:06:23.594840+02:00 INF root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:1817 > Inserted new block elapsed=18.633ms gas=209031 hash=0xcc8ff666dbadbd7591e563f7037626b44ca0f4623424d389f4d723ff6de7bf3b number=41613064 stakingTxs=0 txs=1 uncles=0
2023-05-18T17:06:23.594910+02:00 INF root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:1897 > Imported new chain segment age=1w10h16m blocks=1 cache="1.91 GiB" elapsed=18.720ms hash=0xcc8ff666dbadbd7591e563f7037626b44ca0f4623424d389f4d723ff6de7bf3b mgas=0.209031 mgasps=11.165649087383619 number=41613064 txs=1
2023-05-18T17:06:23.595006+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/syncing.go:923 > [SYNC] UpdateBlockAndStatus: New Block Added to Blockchain ShardID=0 blockEpoch=1442 blockHeight=41613064 blockHex=0xcc8ff666dbadbd7591e563f7037626b44ca0f4623424d389f4d723ff6de7bf3b
2023-05-18T17:06:23.595068+02:00 INF root/go/src/github.com/harmony-one/harmony/internal/chain/reward.go:295 > Commit sigs are ready
2023-05-18T17:06:23.595130+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44335038 shardID=1
2023-05-18T17:06:23.595191+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44517690 shardID=2
2023-05-18T17:06:23.595250+02:00 INF root/go/src/github.com/harmony-one/harmony/core/offchain.go:165 > [insertChain/crosslinks] Cross Link Added to Beaconchain blockNum=44358026 shardID=3
2023-05-18T17:06:23.595309+02:00 INF root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:1817 > Inserted new block elapsed=1.689ms gas=0 hash=0xd9acc9bd2a856eae11113db100b4450c2e90ecccfb986854c7a01a6dba1ede38 number=41613065 stakingTxs=0 txs=0 uncles=0
2023-05-18T17:06:23.595387+02:00 INF root/go/src/github.com/harmony-one/harmony/core/blockchain_impl.go:1897 > Imported new chain segment age=1w10h16m blocks=1 cache="1.91 GiB" elapsed=1.785ms hash=0xd9acc9bd2a856eae11113db100b4450c2e90ecccfb986854c7a01a6dba1ede38 mgas=0 mgasps=0 number=41613065 txs=0
2023-05-18T17:06:23.595431+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/syncing.go:923 > [SYNC] UpdateBlockAndStatus: New Block Added to Blockchain ShardID=0 blockEpoch=1442 blockHeight=41613065 blockHex=0xd9acc9bd2a856eae11113db100b4450c2e90ecccfb986854c7a01a6dba1ede38
2023-05-18T17:06:23.595492+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/explorer/storage.go:242 > writing trace into explorer DB block hash=0xd9acc9bd2a856eae11113db100b4450c2e90ecccfb986854c7a01a6dba1ede38 module="explorer storage"
2023-05-18T17:06:23.604685+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/explorer/storage.go:242 > writing trace into explorer DB block hash=0xcc8ff666dbadbd7591e563f7037626b44ca0f4623424d389f4d723ff6de7bf3b module="explorer storage"
2023-05-18T17:06:23.604856+02:00 INF root/go/src/github.com/harmony-one/harmony/internal/chain/reward.go:295 > Commit sigs are ready
# here the sync fails, block 41613066
2023-05-18T17:06:23.604981+02:00 ERR root/go/src/github.com/harmony-one/harmony/api/service/legacysync/syncing.go:911 > [SYNC] UpdateBlockAndStatus: Error adding new block to blockchain 41613066 0 error="can't load code hash 422c55ce5f5e1ece63fb9347a319ccffb476da0a4e422082ab6714fe1285b3c2: not found"
2023-05-18T17:06:23.605056+02:00 ERR root/go/src/github.com/harmony-one/harmony/api/service/legacysync/syncing.go:1116 > [SYNC] ProcessStateSync failed (isBeacon: false, ShardID: 0, otherHeight: 41928679, currentHeight: 41613043) error="can't load code hash 422c55ce5f5e1ece63fb9347a319ccffb476da0a4e422082ab6714fe1285b3c2: not found"
2023-05-18T17:06:23.605114+02:00 INF root/go/src/github.com/harmony-one/harmony/consensus/consensus_service.go:357 > [UpdateConsensusInformation] Updating..... mode=Listening myBlock=41501264 myViewID=41505246 phase=Announce
2023-05-18T17:06:23.605169+02:00 INF root/go/src/github.com/harmony-one/harmony/consensus/consensus_service.go:409 > [UpdateConsensusInformation] Successfully updated public keys mode=Listening myBlock=41501264 myViewID=41505246 numPubKeys=239 phase=Announce
2023-05-18T17:06:23.605250+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/explorer/storage.go:242 > writing trace into explorer DB block hash=0x5128fed109960363f54ecea66ea9c0a2fdec073a0926584845762a7181b0a253 module="explorer storage"
2023-05-18T17:06:23.605315+02:00 INF root/go/src/github.com/harmony-one/harmony/consensus/consensus_service.go:84 > My Committee updated mode=Listening myBlock=41501264 myViewID=41505246 phase=Announce
2023-05-18T17:06:23.708581+02:00 INF root/go/src/github.com/harmony-one/harmony/consensus/consensus_service.go:96 > My Leader info=ca23704be46ce9c4704681ac9c08ddc644f1858a5c28ce236e1b5d9dee67c1f5a28075b5ef089adeffa8a372c1762007 mode=Listening myBlock=41501264 myViewID=41505246 phase=Announce
2023-05-18T17:06:23.800462+02:00 INF root/go/src/github.com/harmony-one/harmony/consensus/view_change.go:567 > [ResetViewChangeState] Resetting view change state Phase=Announce mode=Listening myBlock=41501264 myViewID=41505246 phase=Announce
2023-05-18T17:06:23.800500+02:00 INF root/go/src/github.com/harmony-one/harmony/consensus/consensus_service.go:440 > [UpdateConsensusInformation] Most Recent LeaderPubKey Updated Based on BlockChain leaderPubKey=ca23704be46ce9c4704681ac9c08ddc644f1858a5c28ce236e1b5d9dee67c1f5a28075b5ef089adeffa8a372c1762007 mode=Normal myBlock=41501264 myViewID=41505246 phase=Announce
2023-05-18T17:06:23.800552+02:00 INF root/go/src/github.com/harmony-one/harmony/consensus/consensus_service.go:471 > [UpdateConsensusInformation] not in committee, Listening mode=Normal myBlock=41501264 myViewID=41505246 phase=Announce
# sync restarts (?)
2023-05-18T17:06:23.800627+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/syncing.go:1140 > legacy sync is executed
2023-05-18T17:06:23.804865+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/downloader/client.go:87 > [SYNC] peer connection closed peerAddress=95.217.207.240:6000 reason="close all connections"
2023-05-18T17:06:23.804967+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/downloader/client.go:87 > [SYNC] peer connection closed peerAddress=65.109.48.134:6000 reason="close all connections"
2023-05-18T17:06:23.805031+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/helpers.go:124 > [SYNC] Finished making connection to peers len=3 shardID=0
2023-05-18T17:06:23.914757+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: beacon block height too much higher than current height beyond tolerance item=QmVTE8iwJ33kiuW9EKQaS9iHHUpqJivaXwMAtf9YoW2Lch
2023-05-18T17:06:23.924781+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/helpers.go:37 > [SYNC] getMaxPeerHeight blockHeight=41928685 peerIP=65.109.48.134
2023-05-18T17:06:23.924900+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/helpers.go:37 > [SYNC] getMaxPeerHeight blockHeight=41928685 peerIP=95.217.207.240
2023-05-18T17:06:24.005205+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/helpers.go:37 > [SYNC] getMaxPeerHeight blockHeight=41928685 peerIP=95.217.197.41
# but no luck restarting the sync, stuck forever on mismatch
2023-05-18T17:06:24.704776+02:00 WRN root/go/src/github.com/harmony-one/harmony/internal/chain/sig.go:49 > mask.SetMask failed error="mismatching bitmap lengths expectedBitmapLength 30 providedBitmapLength 32"
2023-05-18T17:06:24.704901+02:00 ERR root/go/src/github.com/harmony-one/harmony/node/node_explorer.go:52 > [Explorer] readSignatureBitmapPayload failed error="mask.SetMask failed"
2023-05-18T17:06:24.704943+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: mask.SetMask failed item=null
2023-05-18T17:06:25.024551+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/helpers.go:37 > [SYNC] getMaxPeerHeight blockHeight=41928686 peerIP=65.109.48.134
2023-05-18T17:06:25.024658+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/helpers.go:37 > [SYNC] getMaxPeerHeight blockHeight=41928686 peerIP=95.217.207.240
2023-05-18T17:06:25.034644+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/helpers.go:37 > [SYNC] getMaxPeerHeight blockHeight=41928686 peerIP=95.217.197.41
2023-05-18T17:06:25.034764+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/syncing.go:1339 > [SYNC] Checking sync status OtherHeight1=41928685 OtherHeight2=41928686 currentHeight=41613065 lastHeight=41613065
2023-05-18T17:06:25.034813+02:00 INF root/go/src/github.com/harmony-one/harmony/api/service/legacysync/syncing.go:1080 > legacy sync is executing ...
2023-05-18T17:06:25.066284+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: beacon block height too much higher than current height beyond tolerance item=QmXd9FepWmxwTXB4ZoxqKuCRsURihdj1aAMehkJXYdAzfr
2023-05-18T17:06:25.066372+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: beacon block height too much higher than current height beyond tolerance item=QmcnqceNAurKzApqynGSwmKwD14hBgqAcLS7UmfVDJpbEq
2023-05-18T17:06:25.867521+02:00 INF root/go/src/github.com/harmony-one/harmony/p2p/host.go:550 > peer disconnected hostID=QmaianyzuVYBB39HcNXLSfrFwASqpXFyEZhj75xWM5wjTE node=QmdckjsSVmXpaKVrirEPAxM5Ek7mZqNUz7rsbj3Cei5NSm
2023-05-18T17:06:26.093493+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: beacon block height too much higher than current height beyond tolerance item=QmVTE8iwJ33kiuW9EKQaS9iHHUpqJivaXwMAtf9YoW2Lch
2023-05-18T17:06:26.724387+02:00 WRN root/go/src/github.com/harmony-one/harmony/internal/chain/sig.go:49 > mask.SetMask failed error="mismatching bitmap lengths expectedBitmapLength 30 providedBitmapLength 32"
2023-05-18T17:06:26.724496+02:00 ERR root/go/src/github.com/harmony-one/harmony/node/node_explorer.go:52 > [Explorer] readSignatureBitmapPayload failed error="mask.SetMask failed"
2023-05-18T17:06:26.724533+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: mask.SetMask failed item=null
2023-05-18T17:06:26.950591+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: beacon block height too much higher than current height beyond tolerance item=QmYHYgRBKQXwN3PcTfeFxUb5TKc4mi858uGMW4oWxHSsV4
2023-05-18T17:06:27.957869+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: beacon block height too much higher than current height beyond tolerance item=QmVTE8iwJ33kiuW9EKQaS9iHHUpqJivaXwMAtf9YoW2Lch
2023-05-18T17:06:28.633548+02:00 WRN root/go/src/github.com/harmony-one/harmony/internal/chain/sig.go:49 > mask.SetMask failed error="mismatching bitmap lengths expectedBitmapLength 30 providedBitmapLength 32"
2023-05-18T17:06:28.633660+02:00 ERR root/go/src/github.com/harmony-one/harmony/node/node_explorer.go:52 > [Explorer] readSignatureBitmapPayload failed error="mask.SetMask failed"
2023-05-18T17:06:28.633759+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: mask.SetMask failed item=null
2023-05-18T17:06:29.850277+02:00 INF root/go/src/github.com/harmony-one/harmony/node/node.go:986 > [p2p]: issue while handling incoming p2p message: beacon block height too much higher than current height beyond tolerance item=QmVTE8iwJ33kiuW9EKQaS9iHHUpqJivaXwMAtf9YoW2Lch

The config I'm using

Version = "2.5.14"

[BLSKeys]
KMSConfigSrcType = "shared"
KMSEnabled = false
PassSrcType = "auto"

[General]
DataDir = "/var/lib/harmony/"
IsArchival = false
IsBeaconArchival = false
IsBackup = false
IsOffline = false
NoStaking = true
NodeType = "explorer"
ShardID = 0
TraceEnable = true

[Network]
BootNodes = ['/dnsaddr/bootstrap.t.hmny.io']
NetworkType = "mainnet"

[DNSSync]
Client = true
Port = 6000
Server = true
ServerPort = 6000
Zone = "t.hmny.io"

[P2P]
MaxConnsPerIP = 10
MaxPeers = 250
WaitForEachPeerToConnect = false

[Sync]
Downloader = false
Enabled = true
StagedSync = false

As said before, I'm running harmony version v7658-v2023.2.0-0-g01691fd3, compiled with go go1.19.8 linux/amd64

[nxet]

I stand corrected, the node just started syncing again (after I manually restarted the daemon process, not sure if that's relevant since the previous process shut down correctly and dumped state to file).
The error still shows up though, and I'd really like to understand what I'm doing wrong with my config.