Bootnode has no peers in `listtopicpeer`

[araskachoi]

• OS: Linux 4.15.0-38-generic Change type blobs in message Collation from string to []byte #41-Ubuntu SMP Wed Oct 10 10:59:38 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• go version: go version go1.10.4 linux/amd64
• Environment: Docker (image from master branch)

Issue:
bootnode has nothing in listtopicpeer but it has peers in its listpeer. If every node is subscribed to a topic, they should have at least some peers in listeningshard from their peerlist.

Steps to Reproduce:

• Create 6 macvlan networks (not bridge) on docker so that each node will seperately connect to
• Create 1 boot node, and 5 client nodes

docker exec -d bootnode sharding-p2p-poc -seed=0 -port=5566 -rpcport=7788 -ip=10.1.0.2
docker exec -d node1 sharding-p2p-poc -seed=1 -port=5566 -rpcport=7788 -bootstrap -bootnodes=/ip4/10.1.0.2/tcp/5566/ipfs/QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7 -ip=10.1.0.6
docker exec -d node2 sharding-p2p-poc -seed=2 -port=5566 -rpcport=7788 -bootstrap -bootnodes=/ip4/10.1.0.2/tcp/5566/ipfs/QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7 -ip=10.1.0.10
docker exec -d node3 sharding-p2p-poc -seed=3 -port=5566 -rpcport=7788 -bootstrap -bootnodes=/ip4/10.1.0.2/tcp/5566/ipfs/QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7 -ip=10.1.0.14
docker exec -d node4 sharding-p2p-poc -seed=4 -port=5566 -rpcport=7788 -bootstrap -bootnodes=/ip4/10.1.0.2/tcp/5566/ipfs/QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7 -ip=10.1.0.18
docker exec -d node5 sharding-p2p-poc -seed=5 -port=5566 -rpcport=7788 -bootstrap -bootnodes=/ip4/10.1.0.2/tcp/5566/ipfs/QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7 -ip=10.1.0.22

• Subscribe to shard 1

docker exec -d bootnode sharding-p2p-poc -client -rpcport=7788 subshard 1
docker exec -d node1 sharding-p2p-poc -client -rpcport=7788 subshard 1
docker exec -d node2 sharding-p2p-poc -client -rpcport=7788 subshard 1
docker exec -d node3 sharding-p2p-poc -client -rpcport=7788 subshard 1
docker exec -d node4 sharding-p2p-poc -client -rpcport=7788 subshard 1
docker exec -d node5 sharding-p2p-poc -client -rpcport=7788 subshard 1

• list peers on each node

sharding-p2p-poc -client -rpcport=7788 listpeer

• list topic peers on each node

sharding-p2p-poc -client -rpcport=7788 listtopicpeer

[ChihChengLiang]

Hummm, if you are runing in a Docker container, the -ip flag should be 0.0.0.0 to be able to receive other peers' connection.

[araskachoi]

interesting... And this is only for the bootstrap node? or should it be for every node?

[ChihChengLiang]

should be for every node

[araskachoi]

changing all the ip's to 0.0.0.0 seemed to make it worse. None of the nodes have any peers anymore.

bootnode command:

sharding-p2p-poc -seed=0 -port=5566 -rpcport=7788 -ip=0.0.0.0

client node command:

sharding-p2p-poc -seed=2 -port=5566 -rpcport=7788 -bootstrap -bootnodes=/ip4/0.0.0.0/tcp/5566/ipfs/QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7 ip=0.0.0.0`

The output of every single node when calling both listpeer and listtopicpeer are empty:

/ # sharding-p2p-poc -client -rpcport=7788 listpeer
>> []
/ # sharding-p2p-poc -client -rpcport=7788 listtopicpeer
>> {"listeningShard":[],"shardCollations_1":[]}

*note: I had joined shard 1 for every node

Can you tell me if I'm implementing the 0.0.0.0 ip address improperly? thank you very much!

[mhchia]

@araskachoi
I wrote a testing function to reproduce it, but it seems there are peers in listtopicpeer. It will be great if you can try it as well. Just check out to the branch test/add-whiteblock-issue-reprod-func in my repo github.com/mhchia/sharding-p2p-poc and run python test/test_series.py, and you can see the output.
I'm not sure if the function aligns with your reproduction process. If there is anything wrong please just tell me! Thanks a lot.

Output produced in my environment:

<Node seed=1 peer_id=exAnfp>: peers=['QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW', 'QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN']
<Node seed=1 peer_id=exAnfp>: topic_peers={'listeningShard': ['Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW', 'QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN'], 'shardCollations_1': ['QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW']}
<Node seed=2 peer_id=d3wzD2>: peers=['QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX', 'QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa']
<Node seed=2 peer_id=d3wzD2>: topic_peers={'listeningShard': ['Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX'], 'shardCollations_1': ['Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX']}
<Node seed=3 peer_id=a3emmG>: peers=['Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW', 'QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX', 'QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7', 'QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN']
<Node seed=3 peer_id=a3emmG>: topic_peers={'listeningShard': ['QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX', 'QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW'], 'shardCollations_1': ['QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX', 'QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW']}
<Node seed=4 peer_id=aog3HV>: peers=['QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX', 'QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7', 'QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW']
<Node seed=4 peer_id=aog3HV>: topic_peers={'listeningShard': ['QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW', 'QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX'], 'shardCollations_1': ['QmZTrYq4AmDtc3K6eCgfQWjk5oF8zsdo764DwdnC6E68qa', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW', 'QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX']}
<Node seed=5 peer_id=ZTrYq4>: peers=['QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX', 'QmS5QmciTXXnCUCyxud5eWFenUMAmvAWSDa1c7dvdXRMZ7', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW']
<Node seed=5 peer_id=ZTrYq4>: topic_peers={'listeningShard': ['Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW', 'QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX', 'Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr'], 'shardCollations_1': ['Qmaog3HVNo5fTMKFqMScSUbdR7d961uabRtP8FcVypTdHN', 'Qma3emmGx7zTvTstNyppztsp7FBSmXZdCTAoPGnsjJ61cr', 'Qmd3wzD2HWA95ZAs214VxnckwkwM4GHJyC6whKUCNQhNvW', 'QmexAnfpHrhMmAC5UNQVS8iBuUUgDrMbMY17Cck2gKrqeX']}

[araskachoi]

Using the test_issues script, this is the output i get

Getting some strange behavior on my end using test_issues

<Node seed=0 peer_id=d8vXqR>: peers=['QmTkBrU7gzzAZo6B93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R', 'QmXtW5fXrrvmHWPhq3FLHdm4zKnC5FZdhTRynSQT57Yrmd', 'QmaPF7a84gB8Ld35T9mJG2kDPNgQ1v5DkZr3hvjeGaxuQN', 'QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tHX6DEe8VL']
<Node seed=0 peer_id=d8vXqR>: topic_peers={'listeningShard': ['QmTkBrU7gzzAZo6B93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R', 'QmXtW5fXrrvmHWPhq3FLHdm4zKnC5FZdhTRynSQT57Yrmd', 'QmaPF7a84gB8Ld35T9mJG2kDPNgQ1v5DkZr3hvjeGaxuQN', 'QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tHX6DEe8VL'], 'shardCollations_1': ['QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tHX6DEe8VL', 'QmTkBrU7gzzAZo6B
93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R', 'QmXtW5fXrrvmHWPhq3FLHdm4zKnC5FZdhTRynSQT57Yrmd', 'QmaPF7a84gB8Ld35T9mJG2kDPNgQ1v5DkZr3hvjeGaxuQN']}<Node seed=1 peer_id=XtW5fX>: peers=['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R']<Node seed=1 peer_id=XtW5fX>: topic_peers={'listeningShard': ['QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R', 'Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi'], 'shardCollations_1': ['Qmd8vXqRiFVkcXc
B1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R']}<Node seed=2 peer_id=Rd2F8C>: peers=['QmTkBrU7gzzAZo6B93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R', 'Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi']
<Node seed=2 peer_id=Rd2F8C>: topic_peers={'listeningShard': ['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmTkBrU7gzzAZo6B93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R'], 'shardCollations_1': ['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmTkBrU7gzzAZo6B93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R']}<Node seed=3 peer_id=TkBrU7>: peers=['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tHX6DEe8VL', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R']
<Node seed=3 peer_id=TkBrU7>: topic_peers={'listeningShard': ['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tHX6DEe8VL', 'QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskD
cpFtPc5R'], 'shardCollations_1': ['QmYfkGxaRn675YYywLu7CHR9iTjUa7A4tHbskDcpFtPc5R', 'Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tHX6DEe8VL']}
<Node seed=4 peer_id=YfkGxa>: peers=['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmTkBrU7gzzAZo6B93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tHX6DEe8VL', 'QmXtW5fXrrvmH
WPhq3FLHdm4zKnC5FZdhTRynSQT57Yrmd']
<Node seed=4 peer_id=YfkGxa>: topic_peers={'listeningShard': ['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi', 'QmTkBrU7gzzAZo6B93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tH
X6DEe8VL', 'QmXtW5fXrrvmHWPhq3FLHdm4zKnC5FZdhTRynSQT57Yrmd'], 'shardCollations_1': ['QmTkBrU7gzzAZo6B93oPc2hetvv8EhmVEZGdkpmdkg7Pw2', 'QmRd2F8C6Nk49zjLmNCHKRzPaMUggEicJMw6tHX6DEe8VL', 'QmXtW5fXrrvmHWPh
q3FLHdm4zKnC5FZdhTRynSQT57Yrmd', 'Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi']}
<Node seed=5 peer_id=aPF7a8>: peers=['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi']
<Node seed=5 peer_id=aPF7a8>: topic_peers={'listeningShard': ['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi'], 'shardCollations_1': ['Qmd8vXqRiFVkcXcB1nigjCDuUyHcFkja6fQcU9w2zMzwwi’]}

Some notes on the output:

• bootnode has a complete list of peers/topic_peers
• node1 has 2 peers in peerlist/topic_peers
• node2 has 3 peers in peerlist/topic_peers
• node3 has 3 peers in peerlist/topic_peers
• node4 has 4 peers in peerlist/topic_peers
• node5 has 1 peers in peerlist/topic_peers

@mhchia also had noted that on his run, he had an empty list for his topic_peers.

[ChihChengLiang]

Hi @araskachoi, not quite sure if your issue is resolved. The output of your test_issues run looks good.

[araskachoi]

@ChihChengLiang hmmm... So does that mean that the nodes will only incrementally get the list of peers? aka the first node to connect to the boot will only have 1 peer, the 2nd node will have 2 peers, 3rd node will have 3 nodes, ... etc.? I suppose this will be fixed if we can "connect to bootnode" after initiating the application with the bootstrap flag ( #114 ).

Does this accurately describe what is occurring with the application as well as the script?

[mhchia]

@araskachoi
We tried again the test_issues test case with our environment and turn out to get the normal output. We suspect the anomaly is probably related to different environments, e.g. the version of the sharding-p2p docker image. Could you try the following commands to build the docker image and run the script with it?

$ git remote add mhchia git@github.com:mhchia/sharding-p2p-poc.git
$ git fetch mhchia
$ git checkout test/add-whiteblock-issue-reprod-func
$ docker build -f docker/dev.Dockerfile -t ethresearch/sharding-p2p:dev .