Does eCAL automatically do time synchronization / offset estimation between hosts?

[chengguizi]

While I was testing send and receive eCAL messages over TCP between two hosts, I realise eCAL Mon prompts me saying the time synchronisation is off.

This makes me wonder, does eCAL already have mechanism to automatically estimate the time offset between the two hosts?
I tried to search for documentation but only found this.
https://continental.github.io/ecal/_api/file_contrib_ecaltime_include_ecaltime.h.html#file-contrib-ecaltime-include-ecaltime-h

If eCAL does, how should I change its behaviour:

1. the type of clock to synchronize / estimate offset with, either real-time clock or monotonic (steady) clock
2. How frequent the synchronization communication happens

I also would like to know, if it is a pure software sync method, or it could also utilise Ethernet hw infra, like PTP protocol? Thanks!

[chengguizi]

I found something very interesting, but has little clue on if this is already implemented fully, or just a test.

https://github.com/eclipse-ecal/ecal/blob/master/contrib/ecaltime/linuxptp/src/config/ecaltime.ini

I would like to find the most pratical way to obtain the clock offset between two hosts, so in the receiving side, i could subtract away the offset from the message I am receiving!

[FlorianReimold]

I will take your Question as a motivation to finally write some documentation about time-synchronization. But let me already summarize it shortly here:

• eCAL does not have an internal time synchronization. What it does have is the ecaltime-simtime that keeps the eCAL::Time (more or less) in sync to the eCAL Player, if configured. This time will pause / jump / run slower & faster depending on the eCAL Player. It relies on only 1 eCAL Player to be around, as multiple eCAL Player instances would overwrite each other's time.
• eCAL has a couple of timesync plugins, that you can configure in the ecal.ini. Those plugins are meant to connect to third party timesync implementations
• By default, eCAL uses the system time of your OS. So if you have any time synchronization that synchronizes that time, you should be fine. Obviously, this can mean that your eCAL::Time jumps by an arbitrary delta, when your OS (or your time synchronization) adjusts that clock. So it may not be monotonic.
• There are very special use cases where you want to have a time synchronization that does not affect the system time. In that case you would have to use a timesync module that connects directly to that time sychronization method (like ecaltime-linuxptp). Most users will probably never come across such a use case. We use it in the past to support proprietary timesync implemnetations on Windows, as the Windows system clock is not very precise (traditionally ms precision only, but can be a little higher nowadays)
• eCAL has function like IsMaster and IsSynchronized to check the synchronization status of the timesync plugin. Currently there is no open source plugin that provides those values, so they are absolutely meaningless. eCAL does not know whether some other application (like linuxptp) synchronizes your system time
• If your requirements are not very high, use NTP. That is already implemented in any OS and relatively easy to use, especially if you are connected to the internet. For offline systems you will typically have to configure it yourself.
• For proper time synchronization, use PTP / gPTP. That goes up to ns precision given proper network hardware. For a proper configuration all network devices in the loop need to support hardware timestamping and agree on one of the PTP / gPTP dialects. On Linux we prefer Linuxptp (which you actually don't need the ecaltime-linuxptp plugin for, if you just let it synchronize your system clock). On Windows there is no proper PTP / gPTP implementation that I know of. This is where the proprietary implementation I wrote about steps in.
• Synchronizing the system time with LinuxPTP and using the default ecaltime-localtime is absolutely the way to go for 99.9% of all setups. It is more performant and easier to configure than using the ecaltime-linuxptp time module that accesses your Network hardware manually.
• In virtual machines you can usually enable time-sync with the Host OS in the VM settings.

How to check for obvious timesync errors:

• eCAL Mon and eCAL Rec can both check for obvious timesync errors.
• in eCAL Mon you can double-click any topic. When a message is received that has a timestamp that differs too much from the current time, it displays a warning sign.
  
• When doing a distributed recording, eCAL Rec will permanently poll timestamps from any connected client and check for obvious differences. For a recording, a synchronized time is extremely important, as the time is used to bring the frames from different machines in the correct order later. It will show you a warning sign too, if an obvious difference is detected:
  https://youtu.be/h3DUirOMfu8?t=160

[chengguizi]

Thanks! Now I know why I am seeing the warnings. So not detected by deliberate time syncing, but just for sanity check.

[chengguizi]

@FlorianReimold
Thanks, that give some good insights! Couple of follow up questions:

1. For ecaltime-localtime mode, what is the local time? Is it real-time or monotonic time? The reason I have been avoiding to use system time is that real-time can have jitter (i.e. not steady), due to syncing to the Internet time.
2. Would there be a documentation on ecaltime-linuxptp, on how to use it? As it would be my first time trying to setup a ptp system. The only thing I know is that the Ethernet Adaptor chip in my system supports PTP.

[FlorianReimold]

1. See edit of my original answer. It is the non-monotonic system time.
2. See edit of my original answer. Honestly, just don't use it. Just sync your system time. That is easier more performant, and easier to debug

[chengguizi]

A more specific question.

https://github.com/eclipse-ecal/ecal/blob/534bc59ea19ce7a4e43c21e257420b14b362d160/samples/cpp/misc/time/src/time.cpp

What does these API suppose to do? IsMaster and IsSynchronized all seems that there is some synchronisation / offset estimation between hosts in eCAL.

[FlorianReimold]

That API is for use with third party timesync modules. There currently is no (real-time) timesync module that provides meaningful data through that API. I think the ecaltime-simtime uses it though.

[chengguizi]

Got it! Then I should not use this feature at the moment then