Description
Hi,
I'm trying to run a multi-node simulation with correct time sync. I'm hoping I can achieve this with socket protocol without having to set up hla.
I also need to have very regular sensor updates, so I was hoping to use use_internal_syncer/morse_sync to do this, which appears to be built for this purpose. Ideally, I'd like to have all nodes syncing to the 'real' clock as enforced by morse_sync.
However, it looks like use_internal_syncer (and morse_sync) does not behave correctly in multi-node mode.
Consider the following situation:
env.configure_multinode( protocol='socket',
server_address='localhost',
server_port='65000',
distribution={
"node1": ['robot1'],
"node2": ['robot2'],
})
env.use_internal_syncer()In this case, morse_sync will apply to node1 and node1 only, because use_internal_syncer forces you to use port 6000. Only the very first node will get a hold of port 6000, and others will not be able to use it.
As a hacky workaround, I tried something like:
sync_ports = {'node1': 6000, 'node2': 6001}
env.configure_multinode( protocol='socket',
server_address='localhost',
server_port='65000',
distribution={
"node1": ['robot1'],
"node2": ['robot2'],
})
# exapnded version of env.use_internal_syncer()
env.configure_stream('socket', time_sync=True, sync_port=sync_ports[os.environ['MORSE_NODE']])This does something closer to my expectation, in that independent morse_sync's apply to each node. However, the nodes do not sync to each other, which isn't surprising because the morse_sync's will be out of sync themselves.
In my mind, we could change morse_sync to support syncing multiple socket endpoints, which might allow syncing multiple nodes. Also, use_internal_syncer should take multi-node situation into account. What do you guys think?
- *MORSE version: morse 1.4-183-gae45
- Blender version: 2.79.0
- Python version: 3.6.9