Performance Issues (Cosmos 5)

[spstack]

Hello, we're evaluating using cosmos and really liking the new web interface. However, we're running into some pretty serious performance issues with it once telemetry starts flowing. I know performance issues may be tough to debug through a forum like this, but I was hoping maybe you had some insight into what I could try to resolve this. I've tried a bunch of things that I'll detail more below, but the basic problem statement is that we're running COSMOS inside an ubuntu VM, and once we get a bunch of telemetry flowing (about 250 different packets per second), the system pretty much grinds to a halt. The telemetry viewer sometimes updates telemetry every ~10 seconds inconsistently, pages take a while to load (sometimes they don't at all), the processor is heavily loaded to the point where it's not quite usable.

System Info

• Ubuntu 20.04 VM with 16GB memory allocated (32GB in system)
• Core i7 with 6 cores allocated to the VM w/ no execution cap
• Use Host I/O cache setting is checked which allows for pretty good disk performance (couple hundred MB/s when I checked w/ hdparm)
• Using docker and running cosmos v5.0.2 release
• Plugin for our target uses the included tcpip_client_interface with the LENGTH protocol for tlm.

Redis?
I kind of suspect that it may be related to running Redis inside the VM, but I'm not totally sure. I tried a few suggestions from this page: https://redis.io/docs/reference/optimization/latency/, but it seems to suggest that intrinsic latency of Redis is just bad in VMs and there's not much you can do about it? I did try the "AOF + fsync never" suggestion by adding appendonly yes to redis.conf in cosmos and relaunching. Didn't seem to have an impact.

Too many telemetry defs?
In the plugin that we've created we have about 450 pages defined across 41 files. I don't think that's too excessive, but not sure what amount of telemetry cosmos has been tested with sooo maybe it is haha. I've noticed that when I open the packet viewer tool, it does take about 5 seconds to even load the available pages even with no telemetry flowing, so maybe it's a slowness parsing/accessing the tlm database? Really a shot in the dark here, but wondering if you've tested with this amount of telemetry.

Virtual Memory Config
I saw that in the rasberry pi setup in the docs there's some steps to configure the linux kernel:

sudo sysctl -w vm.max_map_count=262144
sudo sysctl -w vm.overcommit_memory=1

from: https://cosmosc2.com/docs/v5/raspberrypi
I tried this, and unfortunately didn't seem to alleviate the problem.

Anything else you think I can try, or have you observed any performance problems in testing? That's also totally fine if you don't, I can continue to investigate, but I thought I'd ask.

Thanks!

[ghost]

If you run off master, I just made a change right after the 5.0.2 release to set "append_only no" in redis.conf. That made a huge difference on the Raspberry pi. The AOF persistence in Redis was causing 10+ seconds block to Redis,

After that I would love to know some details about your target. Key details are: number of packets per second, total bytes per second, number of tlm items with conversions per second.

[ghost]

Is this a Windows host with a Ubuntu VM? I'd like to know a little more about your hardware / software setup. We run on WSL2 with Ubuntu 20 and have pretty good performance. I did recently have to add a C:\Users\jmthomas\.wslconfig file with the following:

[wsl2]
memory=8GB
swap=0

to avoid WSL2 (shown as Vmmem in Task Manager) from taking all the available memory in my laptop.

[spstack]

Yep it's a windows host running an Ubuntu VM through Virtualbox. I am running on a laptop, but it does have an i7-8850H and 32GB of RAM, so it's fairly capable (Dell Precision 7730 Laptop specifically). Anything specifically you want to know on the config? Hmm good to know that it performs well with WSL. The resource manager in the VM shows that cosmos isn't using close to 16GB of memory, so don't think it's a memory/paging issue.

I'm planning on eventually trying this out on an actual server that'll be much more powerful, but it'll still be in a virtual linux environment. Maybe the increased core performance will help there and it won't be much of an issue.

[spstack]

Hey there, little update after all of the work you guys have done in recent updates. The performance has improved quite a bit for me in version 5.0.4, but it's still a bit laggy when lots of telemetry is flowing for a while. The site itself is very snappy and responsive, but updates to telemetry pages in the telemetry viewer and packet viewer take about 5-10 seconds (sometimes longer) to update instead of the configured once per second. I'm still using the same settings vm.max_map_count and vm.overcommit_memory listed in the description which do help. Greatly improved overall though.

One issue I do seem to be having that was not present in v5.0.1 w.r.t. performance is that plugins now take about 5 minutes to load instead of <1 minute in v5.0.1. Is that expected?

[jmthomas]

Plugins should load pretty quickly. I suspect you have s.add_runtime_dependency 'cosmos', '~> 5.0' in your plugin.gemspec. I recently fixed that in the plugin template: 75367d0. Remove it and it should load much faster.

[spstack]

Yep you're absolutely right that fixed it. Thanks for the quick response!

[jmthomas]

Please try the 5.0.5 release as we made a bunch of performance enhancements. Additional enhancements will be coming (probably 5.0.6) to support very large DBs (10s of thousands of commands).