Store a client session's Pings and Overall Delays for later analysis

[pcar75]

Bonjour,
I would like to suggest the development of a facility on the client side to store the Ping times and Global delays into a file (text or CSV) while connected so it would be possible latter to analyse those results of server and clients setups, with tools such as a spreadsheet application for data or graph comparison.

Hints :

• Add a path-filename to store the data into.
• Add an On/Off button to activate or Clear button to deactivate the path-filename (à la Server -Recoding fashion)
• Add a time-stamp every X refresh stored
• Specify a maximum time to store data so files don't grow indefinitely
• If the client GUI refresh rates (about 0,5 seconds or less in my experience) are not appropriate (taxing I/O , performance) for storing data, specify a slower refresh rate for storing or possibly store to a memory structure (ex named pipe, other) and write to path-filename when maximum time, Off/Clear button or Disconnect/Exit happens.

Context :
We are currently running a series of tests designed to tweak performance and quality of our sessions, and it is difficult to play music and try to average the values of Pings and Overall Delays to help gauge the benefits.

Our current setup is :

• Drummer with Jamulus server on laptop Windows 7 Pro (64b) and SSD drive, Behringer Q802USB, old native drive
• Guitar with Jamulus client on tower Windows 10 Pro (64b) with SSD and HD drive, Behringer Q802USBm, old native driver
  Our internet speed are similar : ping 10-13 ms, jitter 2-4 ms, 33+ Mpbs download and 10+ Mpbs upload.

Besides testing for best hours/days to do the session due to the vagaries of internet loads regionally, we are ...

• Testing whether the -F and/or -T server arguments offer better results
• Testing whether my Windows 10 standard configuration or SafeModeNetworked configuration is better
• Testing whether a USB3.0 key JamulusOS client is better than a Windows one.
• Testing different buffers size and timing configuration
• Mixes of those tests

Thanks for considering this suggestion.
Merci

[drummer1154]

Salut,

we are a 3 piece band, all living in Munich, but we have 3 differents ISPs. We suffer from the "slow-down" effect.

Our current setup is:

• dr (me), Roland TD-20 via SPDIF, Lexicon Omega; iMac (Early 2008), Catalina; ISP=T-Com (VDSL2 17a G.Vector (ITU G.993.5), 50/10Mbit/s)
• b/g, interface BOSS GT-1B/BOSS GT-1; iMac (Retina 5K, 27-inch, 2017), Big Sur; ISP=VF/Kabel Deutschland (200/12Mbit/s)
• tp+key, Universal Audio Arrow; iMac (2020), Catalina; ISP=m-net (Surf&Fon-Flat 100 fiber (in reality VDSL2 30a (ITU G.993.2) DS-Lite), 100/40Mbit/s)

What we have tried:

• Public Servers (Central or others with low ping RTT), result: Overall Delay 40ms+ for b => hard to concentrate
• Private server on my local network: 64bit Win8.1 i3-3225 3.3GHz 16GB, result: Overall Delay 50ms+ for b => slow down

Meanwhile I learned about buffer bloat (https://www.bufferbloat.net/projects/bloat/wiki/). We used http://www.dslreports.com/speedtest to check the additional delay introduced by the buffer bloat. Results:

• dr: http://www.dslreports.com/speedtest/66452201
• b/g: http://www.dslreports.com/speedtest/66453320
• tp+key: http://www.dslreports.com/speedtest/66460300

I believe that the high additional delay for uploads on my place and for downloads on the bass player's place lead to the observed difficulties. Next try will be to set up a Private Jamulus Server on the m-net environment. Edit 2020-12-08: A Private Server on m-net environment is not possible because m-net uses DS-Lite (Dual-Stack Lite) and provides only public IPv6 addresses.

I believe that ping RTT alone is not sufficient to judge whether or not the overall delay will be small enough. Buffer bloat results from other users would be helpful.

Cheers
Helmuth

Edit: Links repaired, setup info updated (again).

[gene96817]

The problem with buffering is really caused by the choices network operators make in balancing their traffic. It is mostly just polite language for overloaded routers. We can get into a cat-and-mouse game to shape our traffic to minimize buffering at a router. This kind of game gets harder if the problematic router is several hops away. Unfortunately, managing the buffering at a router should not be a user issue.

I am experiencing this kind of problem in several areas in my city. Some neighborhoods have low latency (<20ms) access to my server. Other users have an additional 40+ ms. It is even worst if the path crosses more than one network operator. I am still gathering data to see if this is an old neighborhood vs new neighborhood issue or a poor community vs wealthy community issue.

I am not in favor of putting network diagnostic tools into Jamulus (to play buffer bloat games). It would be appropriate to have tools to report/record when we have poor network service. This would be helpful in discussing service issues with the network operator. (N.B. these problems are also causing poor video conferencing experience. However, most videoconferencing users just accept the problems as if it is a problem with the weather.)

[pcar75]

Bonjour,

Helmuth :

• Thanks for bringing attention to this part of possible latency problem, I will research this.

gene96817 :

• Thanks for further comments on this problem, However It is not a case of adding "network diagnostic tools", the tools are already there, i.e. the rapidly refreshed indicators like led lights and values of Jitter Buffer [auto] in the Client window and the Audio Stream Rate, Ping Time and Overall Delay in the Client>Settings window. The idea is simply to record what is already available i.e. mostly the Ping Time and Overall Delay ... provided , of course, that this processing does not cause undue impacts to Jamulus prime function.
  I too am not in favor of 'feature bloating' this great software, but I will do anything I can to reduce the latency and jitter, and better the sound and timing quality of our music.

[gene96817]

@pcar75 Thanks for the reply. Two things that you probably already know.

** the challenge with indicator lights, when we are trying to quickly indicate a change in status, we need a fast-on of the new status and a slow-off. It is easy to end up with fast changes that is hard to read.

** regarding buffer backup, if we detect that condition quickly, it is helpful to stop sending packets to let the backup clear. Yes, that will cause noise in the audio but that noise will (should?) be better for musicians than to have the sound be more and more delayed. (I don't know our internal machinery so I can only point out this basic principle.)

[drummer1154]

I don't know our internal machinery so I can only point out this basic principle

Same to me. Lack of documentation is currently a very weak point for me. As musicians and engineers we are used to reading block diagrams and signal flows. And at the moment the only Jamulus block diagram I am aware of is in "Figure 2: Simplified Jamulus block diagram" of Volker's (otherwise very well structured and comprehensive) case study from 2015 (http://llcon.sourceforge.net/PerformingBandRehearsalsontheInternetWithJamulus.pdf). After reading the paper, I was (mistakenly) thinking that there is only one mix for all clients, which as of today is obviously wrong as there is a "Personal Mix" per client.

If possible I would like to contribute to creating a more current diagram which should contain audio as well as control flows; if Volker reads this maybe he can direct me how to achieve this (and maybe also a bit more streamlined description in the wiki).

Edit 2020-12-06: I was not aware of Issue #64 - the diagrams there are exactly what I was missing. They need to be taken up into the wiki (if they are not already - sorry for my inability to check everything :-)

bringing attention to this part of possible latency problem, I will research this

Highly appreciated. I would really be happy if - within this community - we would be able to drill down latency problems as far as possible. In our trio, we have only 3 different views and access methods to research the problem. But in the community there are tons of other possibilities. Maybe this issue is not the right place to discuss everything related to latency. But perhaps we can make a start. My approach is to provide to the community the info which I have, hoping that there are other colleagues who can add their knowledge. In our case I hope there are other musicians on cable network (VF/KDG) who can give their input.

most videoconferencing users just accept the problems as if it is a problem with the weather

... which for me is no wonder because in verbal communication you can easily stop and ask for repetition or clarification - impossible when playing music in real time.

Last not least my appeal to the responsible persons: Please do not overload the Jamulus SW and GUI - the more functions (maybe only relevant for a small amount of users) are introduced, the higher the risk for bugs due to the amount of platforms to be supported. As already said there are tools outside this platform to investigate problems.

Cheers
Helmuth

[WolfganP]

If possible I would like to contribute to creating a more current diagram which should contain audio as well as control flows; if Volker reads this maybe he can direct me how to achieve this (and maybe also a bit more streamlined description in the wiki).

There was a previous discussion on diagrams at #64 but the tension between detailed techie flows and easy to understand pics for non techs was never resolved for them to be included in the wiki/documentation.

[drummer1154]

Reading #758 I need to say that for our trio with the given ISP situation (which we cannot currently change) JamKazam was not working well for us. For us latency, not bandwidth, is the bottleneck! For me the only solution appears to be using distributed computing power (i.e. audio compression/decompression) to overcome the latency problems and therefore I support the Jamulus approach.

@WolfganP Thanx to referring me to #64, I will study it. As an electrical engineer/drummer/sound engineer/software developer and project manager (retired!) I am convinced that the techie/non-techie conflict can be solved if the doc is giving the relevant information to everyone - most likely one simple sheet is not enough - linking should make it happen.

[gene96817]

I found the diagrams in #64 useful for a very very high-level view of how the pieces relate to each other. It doesn't have enough detail to give insights for troubleshooting problems. If the coders could provide the very techie flow, I am sure there are others (as in this thread) who could help provide higher-level abstractions for the technical users.

[gene96817]

@pcar75 This discussion about bufferbloat has been sitting in my mind. While the referenced article is pragmatic, it is the wrong way to deal with buffer backups (which they call the bloating of a buffer). I would be interested in a discussion about buffer backups in the client-server path and creating a mechanism for quenching the buffer back up. If this is interesting, we could design an enhancement to the data stream to detect and quench the backup. Assuming the Internet path is only momentarily backed up, this would accelerate recovery to minimize the interruption of the audio stream. (This is the best I can contribute to this topic. I am a protocol guy, not a coder.)

[ann0see]

I am convinced that the techie/non-techie conflict can be solved if the doc is giving the relevant information to everyone - most likely one simple sheet is not enough

We could add a more technical page on jamulus.io maybe in the upcoming knowledge base. In fact, I think we lack technical documentation.

Please open an issue on https://github.com/jamulussoftware/jamuluswebsite/issues

[drummer1154]

@ann0see: To whom do you address this? If to me, OK, I am willing to open an Issue, please help me with the title, thx. And what about the "upcoming knowledge base"? What is supposed to be to found there with respect to the current wiki?

[ann0see]

If to me, OK, I am willing to open an Issue

Yes, you should open an issue with the feature request template. Fill the template and describe that you would like to have more detailed, technical information documented somewhere for detailed troubleshooting purposes (at least that's what I read from skimming through your comments).

This information would not necessarily be for the end user but for people who want or need to understand the technical details. Therefore we'd not include it in the main wiki but on an external page in an upcoming blog like section on jamulus.io

Last night a group of us had a difficult time because of delay. We changed our own settings to make matters the best possible, and we played because that is what musicians do (regardless of our superb data analysis, networking and interprocess communications skills).
Jamulus does not require Data capture or Network protocol or Statistical analysis functions (maybe someone can write a plugin for WireShark and write R language scripts instead).