Issues with the Implementation of the Syncplay Protocol

[Predidit]

Sorry to bother you. I'm attempting to implement the Syncplay protocol, but I'm running into some issues since the documentation on the official Syncplay website appears to be outdated.

I would like to ask for clarification regarding the structure of the ping message within the state message. Here’s what I’m currently doing:

1. In the first message, I set clientRtt to 0.
2. Fill clientLatencyCalculation with a floating point value derived by dividing the current Unix timestamp (in milliseconds) by 1000.
3. leave latencyCalculation unset.
   Upon receiving the response message, I record the clientLatencyCalculation from that message and then use it to fill in latencyCalculation in the next message I send.

Under this approach, I receive a serverRtt value over 2000 in the response message, which clearly indicates a problem. I’m very curious about what the correct approach should be. Additionally, could the current method be the cause of the issue where I’m unable to properly set the position? Every time I send a state message, the position field in the returned state message is always 0.

Any guidance would be greatly appreciated!

[Predidit]

The log is as follows:

 SyncPlay: client >> [1742721748.039]: {"Hello":{"username":"mortis","room":{"name":"32421321"},"version":"1.7.0","features":{"sharedPlaylists":true,"chat":true,"featureList":true,"readiness":true,"managedRooms":false}}}
 SyncPlay: client >> [1742721748.04]: {"Set":{"file":{"duration":14400.0,"name":"454684[4]","size":220514438},"user":{"mortis":{"room":{"name":"32421321"}}}}}
 SyncPlay: server << [1742721748.639]: {"Set": {"ready": {"username": "mortis", "isReady": null, "manuallyInitiated": false}}}
 SyncPlay: server << [1742721748.64]: {"Set": {"playlistChange": {"user": "angular23", "files": []}}}
 SyncPlay: server << [1742721748.64]: {"Set": {"playlistIndex": {"user": "angular23", "index": null}}}
 SyncPlay: server << [1742721748.64]: {"Hello": {"username": "mortis", "room": {"name": "32421321"}, "version": "1.7.0", "realversion": "1.7.3", "features": {"isolateRooms": true, "readiness": true, "managedRooms": true, "persistentRooms": false, "chat": true, "maxChatMessageLength": 150, "maxUsernameLength": 16, "maxRoomNameLength": 35, "maxFilenameLength": 250}, "motd": "You are using Syncplay 1.7.0 but a newer version is available from https://syncplay.pl\n\nHello mortis,\n\nThe Syncplay latest is available from http://syncplay.pl/\n\n"}}
 SyncPlay: server << [1742721748.64]: {"Set": {"user": {"mortis": {"room": {"name": "32421321"}, "file": {"duration": 14400.0, "name": "454684[4]", "size": 220514438}}}}}
 SyncPlay: server << [1742721748.911]: {"State": {"ping": {"latencyCalculation": 1742724284.7857127, "serverRtt": 0}, "playstate": {"position": 0, "paused": true, "doSeek": false, "setBy": "mortis"}}}
 SyncPlay: client >> [1742721749.041]: {"State":{"ping":{"clientRtt":0,"clientLatencyCalculation":1742721749.041,"playstate":{"position":787.746,"paused":false,"setBy":"mortis","doSeek":false}}}}
 SyncPlay: server << [1742721749.728]: {"State": {"ping": {"latencyCalculation": 1742724285.7859957, "serverRtt": 0, "clientLatencyCalculation": 1742721749.4357758}, "playstate": {"position": 0, "paused": true, "doSeek": false, "setBy": "mortis"}}}
 SyncPlay: client >> [1742721750.041]: {"State":{"ping":{"clientRtt":0,"clientLatencyCalculation":1742721750.041,"latencyCalculation":1742721749.4357758,"playstate":{"position":788.747,"paused":false,"setBy":"mortis","doSeek":false}}}}
 SyncPlay: server << [1742721750.73]: {"State": {"ping": {"latencyCalculation": 1742724286.7859974, "serverRtt": 2536.955236196518, "clientLatencyCalculation": 1742721750.435978}, "playstate": {"position": 0, "paused": true, "doSeek": false, "setBy": "mortis"}}}
 SyncPlay: client >> [1742721750.826]: {"State":{"ignoringOnTheFly":{"client":1},"ping":{"clientRtt":0,"clientLatencyCalculation":1742721750.826,"latencyCalculation":1742721750.435978,"playstate":{"position":789.748,"paused":true,"setBy":"mortis","doSeek":true}}}}
 SyncPlay: server << [1742721751.729]: {"State": {"ping": {"latencyCalculation": 1742724287.7865756, "serverRtt": 2536.740721464157, "clientLatencyCalculation": 1742721751.435875}, "playstate": {"position": 0, "paused": true, "doSeek": false, "setBy": "mortis"}, "ignoringOnTheFly": {"client": 1}}}
 SyncPlay: client >> [1742721751.729]: {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0,"clientLatencyCalculation":1742721751.729,"latencyCalculation":1742721750.435978,"playstate":{"position":963.383,"paused":false,"setBy":"mortis","doSeek":false}}}}
 SyncPlay: client >> [1742721752.041]: {"State":{"ping":{"clientRtt":0,"clientLatencyCalculation":1742721752.041,"latencyCalculation":1742721751.435875,"playstate":{"position":963.713,"paused":false,"setBy":"mortis","doSeek":false}}}}
 SyncPlay: client >> [1742721752.209]: {"State":{"ignoringOnTheFly":{"client":1},"ping":{"clientRtt":0,"clientLatencyCalculation":1742721752.209,"latencyCalculation":1742721751.435875,"playstate":{"position":963.713,"paused":true,"setBy":"mortis","doSeek":true}}}}
 SyncPlay: server << [1742721752.73]: {"State": {"ping": {"latencyCalculation": 1742724288.7869577, "serverRtt": 2537.1248989105225, "clientLatencyCalculation": 1742721752.435173}, "playstate": {"position": 0, "paused": true, "doSeek": false, "setBy": "mortis"}, "ignoringOnTheFly": {"client": 1}}}
 SyncPlay: client >> [1742721752.73]: {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0,"clientLatencyCalculation":1742721752.73,"latencyCalculation":1742721751.435875,"playstate":{"position":963.713,"paused":true,"setBy":"mortis","doSeek":false}}}}
 SyncPlay: client >> [1742721753.04]: {"State":{"ping":{"clientRtt":0,"clientLatencyCalculation":1742721753.04,"latencyCalculation":1742721752.435173,"playstate":{"position":963.963,"paused":true,"setBy":"mortis","doSeek":false}}}}
 SyncPlay: server << [1742721753.728]: {"State": {"ping": {"latencyCalculation": 1742724289.7860034, "serverRtt": 2536.9554674625397, "clientLatencyCalculation": 1742721753.4353552}, "playstate": {"position": 0, "paused": true, "doSeek": false, "setBy": "mortis"}}}
 SyncPlay: client >> [1742721754.041]: {"State":{"ping":{"clientRtt":0,"clientLatencyCalculation":1742721754.041,"latencyCalculation":1742721753.4353552,"playstate":{"position":963.963,"paused":true,"setBy":"mortis","doSeek":false}}}}
 SyncPlay: server << [1742721754.73]: {"State": {"ping": {"latencyCalculation": 1742724290.7868865, "serverRtt": 2536.9568836688995, "clientLatencyCalculation": 1742721754.4356406}, "playstate": {"position": 0, "paused": true, "doSeek": false, "setBy": "mortis"}}}
 SyncPlay: client >> [1742721755.035]: {"State":{"ping":{"clientRtt":0,"clientLatencyCalculation":1742721755.035,"latencyCalculation":1742721754.4356406,"playstate":{"position":963.963,"paused":true,"setBy":"mortis","doSeek":false}}}}

[yuroyami]

You can find the protocol implementation for the incoming and outcoming messages in JsonHandler.kt and JsonSender.kt respectively. They would give you a pretty good idea how data is handled from the server and how data is bundled and in what shape it is sent to the server.

Here's what I currently do to formulate the ping JSON object within the State message:

val ping = buildJsonObject {
                servertime?.let { put("latencyCalculation", it) }
                put("clientLatencyCalculation", clienttime)
                put("clientRtt", viewmodel!!.p.ping.value ?: 0)
            }

This means:
1- You generate a Unix epoch timestamp and divide it by 1000. That's your clientLatencyCalculation

2- You receive latencyCalculation from the server, record it. That's your latencyCalculation(you're basically sending it back)

3- Calculate ping in your own terms. I use the system's ping under the hood to calculate the clientRtt.

I'm not sure why the position isn't being acknowledged or set. Just ensure you're sending a float value in seconds, including a decimal. Hope this helps

[Predidit]

@yuroyami Thank you very much for your help.

I noticed that in the official implementation of syncplay, an algorithm is used that calculates clientRtt by using the serverRtt from the received messages and clientLatencyCalculation (used as the message timestamp).

The related implementation can be found at https://github.com/Syncplay/syncplay/blob/b96d43a8301e5f82185e0d9cd67e728d0600899f/syncplay/protocols.py#L782-L810

Could you explain the rationale behind not adopting this method, and what drawbacks this approach might have?

[yuroyami]

Could you explain the rationale behind not adopting this method

It’s been a while since I implemented that part of the protocol, but I know for sure that I didn’t fully stick to the original spec. I don’t recall the exact reasons, but I’d recommend using their implementation first, and if that doesn’t work out, try mine. I’ll likely adjust my implementation to align with theirs since the server is responsible for all the calculations and needs accurate data. I originally went with my approach because it simply worked, and synchronization was reliable down to a few milliseconds. That said, the original implementation offers the best synchronization fidelity and should be the default choice.

[Et0h]

That bit isn't my code, but my understanding is that the 'official' (spec) implementation was coded that way 12 years ago with the intention of accounting for both latency and jitter at the same time.

It would be interesting if someone compared that method to the one used in Syncplay mobile to see if they give different results and if so which approach is more reliable.

It is possible that the best depends on the use case.

For regular WiFi connections it shouldn't make much difference but it could be relevant for those using unreliable connections such as mobile/cellular internet.

I seem to recall we did various tests of it at the time and it seemed to work well enough, but as it was more than a decade ago we wouldn't have been able to see how well it worked using modern mobile internet in different connection reliability scenarios.

[Predidit]

Thank you everyone for your help. The physical distance between me and the syncplay server is very far, and my network latency is consistently around 350ms. In this situation, I feel that the original implementation is better. I can't tell whether this is due to the advantage of the aforementioned latency algorithm, or because the official syncplay implementation reserves time (around 150ms) for the player's seek. I will conduct more rigorous testing later. Once I fully understand the protocol, the porting should be straightforward. If the original implementation proves to be better, I will draft a PR.

[Predidit]

I encountered some issues with my understanding of the ignoringOnTheFly field. In fact, the official documentation does not mention anything about configuring this field. My approach is straightforward: when initiating an ignoringOnTheFly request, I simply populate the client field with 1 and leave the server field unset. Then, upon receiving an ignoringOnTheFly request, I send a new one where I set the server field to 1 and leave the client field unset.

That leads to a problem: I can use this approach to set the playback progress when I am alone in the room, but when there are other members, they will also receive the ignoringOnTheFly requests that I initiate. However, according to the logic above, it appears that they will be unable to confirm these ignoringOnTheFly requests. Instead, the requests to confirm receipt of the ignoringOnTheFly requests are being erroneously sent to other members.

The log is as follows:

user1:

client >> {"State":{"ignoringOnTheFly":{"client":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742886845.607},"playstate":{"position":375.208,"paused":true,"doSeek":null}}}
server << {"State": {"ping": {"latencyCalculation": 1742889383.1235085, "serverRtt": 0.3436472415924072, "clientLatencyCalculation": 1742886845.6071508}, "playstate": {"position": 375.208, "paused": true, "doSeek": null, "setBy": "user1"}, "ignoringOnTheFly": {"server": 1, "client": 1}}}
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742886846.162,"latencyCalculation":1742889383.1235085},"playstate":{"position":375.208,"paused":true,"doSeek":null}}}
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742886847.205,"latencyCalculation":1742889384.2283847},"playstate":{"position":377.35537022524875,"paused":false,"doSeek":null}}}
server << {"State": {"ping": {"latencyCalculation": 1742889385.4063613, "serverRtt": 0.3578779697418213, "clientLatencyCalculation": 1742886847.8901274}, "playstate": {"position": 378.552, "paused": true, "doSeek": null, "setBy": "user1"}, "ignoringOnTheFly": {"server": 1}}}
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742886848.224,"latencyCalculation":1742889385.4063613},"playstate":{"position":378.552,"paused":true,"doSeek":null}}}
server << {"State": {"ping": {"latencyCalculation": 1742889386.239264, "serverRtt": 0.33347320556640625}, "playstate": {"position": 378.8480545059489, "paused": false, "doSeek": null, "setBy": "user2"}, "ignoringOnTheFly": {"server": 1}}}
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742886849.206,"latencyCalculation":1742889386.239264},"playstate":{"position":378.8480545059489,"paused":false,"doSeek":null}}}
server << {"State": {"ping": {"latencyCalculation": 1742889387.401657, "serverRtt": 0.35239577293395996, "clientLatencyCalculation": 1742886849.88417}, "playstate": {"position": 378.586, "paused": true, "doSeek": null, "setBy": "user1"}, "ignoringOnTheFly": {"server": 1}}}
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742886850.219,"latencyCalculation":1742889387.401657},"playstate":{"position":378.586,"paused":true,"doSeek":null}}}
server << {"State": {"ping": {"latencyCalculation": 1742889388.2448738, "serverRtt": 0.33289599418640137}, "playstate": {"position": 380.8497362440022, "paused": false, "doSeek": null, "setBy": "user2"}, "ignoringOnTheFly": {"server": 1}}}
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742886851.211,"latencyCalculation":1742889388.2448738},"playstate":{"position":380.8497362440022,"paused":false,"doSeek":null}}}

user2:

**server << {"State": {"ping": {"latencyCalculation": 1742889699.299359, "serverRtt": 0.5485236644744873, "clientLatencyCalculation": 1742887161.777736}, "playstate": {"position": 29.487, "paused": true, "doSeek": null, "setBy": "user1"}, "ignoringOnTheFly": {"server": 1}}}
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742887162.11,"latencyCalculation":1742889699.299359},"playstate":{"position":29.487,"paused":true,"doSeek":null}}}
server << {"State": {"ping": {"latencyCalculation": 1742889700.2540603, "serverRtt": 0.5400562286376953, "clientLatencyCalculation": 1742887162.7324827}, "playstate": {"position": 30.329943655084108, "paused": false, "doSeek": null, "setBy": "user2"}, "ignoringOnTheFly": {"server": 1}}}**
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742887163.064,"latencyCalculation":1742889700.2540603},"playstate":{"position":30.329943655084108,"paused":false,"doSeek":null}}}
server << {"State": {"ping": {"latencyCalculation": 1742889701.6888425, "serverRtt": 0.5403661727905273, "clientLatencyCalculation": 1742887164.1667578}, "playstate": {"position": 30.155, "paused": true, "doSeek": null, "setBy": "user1"}, "ignoringOnTheFly": {"server": 1}}}
client >> {"State":{"ignoringOnTheFly":{"server":1},"ping":{"clientRtt":0.0,"clientLatencyCalculation":1742887164.5,"latencyCalculation":1742889701.6888425},"playstate":{"position":30.155,"paused":true,"doSeek":null}}}

The key details of the issue should appear in the first three lines of user2's log.

[Et0h]

I encountered some issues with my understanding of the ignoringOnTheFly field. In fact, the official documentation does not mention anything about configuring this field. My approach is straightforward: when initiating an ignoringOnTheFly request, I simply populate the client field with 1 and leave the server field unset. Then, upon receiving an ignoringOnTheFly request, I send a new one where I set the server field to 1 and leave the client field unset.

You are correct that https://syncplay.pl/about/protocol/ explains the general concept of ignoringOnTheFly but not what the value represents.

For relevant code see
https://github.com/Syncplay/syncplay/blob/b96d43a8301e5f82185e0d9cd67e728d0600899f/syncplay/protocols.py#L295-319 and https://github.com/Syncplay/syncplay/blob/b96d43a8301e5f82185e0d9cd67e728d0600899f/syncplay/protocols.py#L273-293

Here is a (hopefully accurate) explanation of how it works...

TL;DR

ignoringOnTheFly uses counters in state messages ("client" and "server") to signal a transient "ignore" mode after relevant state changes (e.g., seeking). Counters are incremented when a state change initiating ignoring occurs and reset (sometimes via acknowledgment) to end it, allowing resumption of normal updates.

Overall Dynamics and Purpose

• Transient Ignoring State:

Instead of an "ignore period" defined by a time interval, the mechanism uses counters to signal a transient state. These counters indicate that a state change has occurred, and that updates which might conflict should be temporarily disregarded. Incrementing the counter---as opposed to setting it to a fixed value---serves to track that one or more state changes requiring temporary ignoring have been initiated. This helps the system manage overlapping or consecutive state changes, ensuring that the need to ignore conflicting updates persists until the transient state is resolved.

• Coordinated Update Handling:

By embedding these counter values in state messages, both the client and server are aware of each other's transient state. This coordination helps prevent oscillations or conflicting updates (for example, preventing the client from reverting to an outdated playback position). The counters help maintain awareness of the ongoing need to ignore updates, especially if several changes occur rapidly.

• Resumption of Normal Processing:

Normal state update processing resumes when the transient ignoring state is cleared. This occurs when the ignoringOnTheFly counters are reset. The counters are reset through different mechanisms, including:

• A form of acknowledgment: In some situations, a counter is reset if the other side reports a matching value.
• Proactive clearing: Counters can also be reset based on events like receiving a state update.

Once the counters are reset, it indicates that the conditions requiring temporary ignoring have been resolved, and normal state updates are processed again.

Client-Side Behaviour

• Counters in Use:
  The client maintains two counters:

  • clientIgnoringOnTheFly: Incremented when the client makes a local state change (e.g., a seek). This signals that the client is temporarily in a state where incoming conflicting state updates should not override its change.

  • serverIgnoringOnTheFly: Set based on the value received from the server, reflecting the server's transient ignoring state.

• Sending State Updates:
  When the client sends a state update via its sendState method:

  • If a local state change has occurred (indicated by a true stateChange flag), the client increments its clientIgnoringOnTheFly counter.

  • The outgoing state message includes an "ignoringOnTheFly" field containing:

    • A "client" key with the current clientIgnoringOnTheFly counter value.

    • A "server" key with the server's serverIgnoringOnTheFly counter value.

• Processing Incoming State Updates:
  Upon receiving a state message:

  • If the message includes a "server" key, the client updates its serverIgnoringOnTheFly counter and resets its own clientIgnoringOnTheFly counter.

  • If the message contains a "client" key and its value matches the client's counter, the client resets its clientIgnoringOnTheFly counter.

This approach ensures that the client only temporarily suspends processing of potentially conflicting state changes, resuming normal updates once the transient state (as indicated by the counters) is cleared.

Server-Side Behaviour

• Counters in Use:
  The server also uses two counters:

  • serverIgnoringOnTheFly: Typically incremented (especially during forced updates) to indicate that the server is in a transient ignoring state.

  • clientIgnoringOnTheFly: Tracks the transient ignoring state as reported by the client in incoming state messages.

• Sending State Updates:
  In the server's sendState method:

  • When a forced update is sent, the server increments its serverIgnoringOnTheFly counter.

  • The outgoing state message includes an "ignoringOnTheFly" field with:

    • "server" set to the current serverIgnoringOnTheFly counter.

    • "client" set to the current clientIgnoringOnTheFly counter (after which the server resets its record of the client's counter).

• Handling Incoming Updates:
  When processing a state message from the client:

  • If the message contains an "ignoringOnTheFly" field:

    • The "server" key is compared with the server's own counter; if they match, the server resets its serverIgnoringOnTheFly counter.

    • The "client" key is used to update the server's record of the client's transient ignoring state.

This mechanism allows the server to temporarily stop processing client updates that could conflict with a recent forced update until the transient ignoring state is resolved.

[Predidit]

@Et0h

Thank you for the clarification. I mistakenly used a simplistic locking mechanism on the client side, which caused the issue. Your help has brought me closer to the official implementation. I nearly successfully implemented a Dart version of the syncplay protocol, and porting from Dart to Java should be very straightforward.

Due to the extremely high network latency I experience with the official server, I have observed some interesting phenomena. When there is only one user in the room and there are no unresolved ignoringOnTheFly messages, I noticed that the duration field in the State received from the server steadily lags behind my local playback progress over time. Specifically, with each update message, the duration field falls behind by roughly one clientRtt compared to my local progress.

While I fully understand that some deviation between the server's duration field and the local playback progress is expected, this deviation should not consistently widen over time. It appears that the server does not update the duration field simply based on the passage of time. Could it be that the ping structure sent by the client influences its update? Or is it possibly a bug in the server's implementation?

[Predidit]

@Et0h

Edit: I have resolved the issue. I'm really sorry for bothering you—it turned out to be related to my implementation of messageAge. It looks like I've completed everything except for TLS. Thank you for your help. Since this was my first attempt at implementing the socket API, I encountered a lot of errors. I really appreciate your patience and assistance.

The following is unedited message:

The issue seems to be occurring in https://github.com/Syncplay/syncplay/blob/b96d43a8301e5f82185e0d9cd67e728d0600899f/syncplay/protocols.py#L273-293. When there's no need to handle "ignoringOnTheFly", it looks like we're simply returning the "position" field from the received message, which is exactly how my current implementation works.

However, in "Test Case 2 (Bob seeks to 5 minutes)" on https://syncplay.pl/about/protocol/, I noticed that we don't directly return the received "position"; instead, there is an offset applied, which is confusing to me.

[yuroyami]

@Predidit Regarding TLS, I am not sure what Flutter uses as engine when it comes to socket API but if it somehow uses Netty under the hood, you can check my Netty implementation to see how I promote a plain non-secure connection to a TLS connection. The Syncplay protocol relies on "Opportunistic TLS" which goes this way:

The connection at first is plain and non-secure, where the client asks the server if it supports TLS (this precedes the step where you send a Hello). If the server says yes, then the client will upgrade to TLS by adding the SSL/TLS handler to the beginning of the pipeline.

[Et0h]

@Predidit For more details on how Syncplay's TLS works see:

• TLS not working. #6
• Syncplay unofficial client and official server mutual TLS authentification Syncplay/syncplay#657
• https://github.com/Syncplay/syncplay/wiki/TLS-support