OTNS usage for anticipating performances

[echargy]

I’ve been experimenting with OTNS, and I noticed that its performance, particularly in terms of ping latency, doesn’t quite match what we observe on real hardware implementations (ping latency seems approx 10 times better with otns than real implem).

Is this something expected ?
Can we modify ONTS to introduce processing latencies to better reflect real-world behavior?

Thanks for your help

[jwhui]

I'm curious - can you share the latency numbers you are seeing for both real devices and OTNS?

[echargy]

Sorry I didn't share it earlier. For an FTD -> FTD only test (ping @ 1hop, pinging the mle-id), with the simulator :

• Ping size 1200B : 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 28/37.020/52 ms.
• Ping size 280B : 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 6/12.850/18 ms.
• Ping size 16B : 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 1/3.600/7 ms.

Whereas in real world environment (cannot really say which one publicly), we measure worse performance. Luckily, they are confirmed with published results online !

If we take a look at SiLabs performance paper (https://www.silabs.com/documents/public/application-notes/an1408-openthread-mesh-network-performance.pdf), you can see on Figure 4-2, page 11, that :

• For 16B, the avg latency at one hop is ~25ms
• For 280B, the avg latency at one hop is ~95ms

If now we take a look at Espressif test report (https://documentation.espressif.com/thread_network_performance_test_report.pdf), you can see in Table 3-2. Unicast Round-Trip Latency Test Results, page 10, that :

• For 10B, the min/avg/max values at one hop are 11/14/28ms
• For 650B, the min/avg/max values at one hop are 90/101/144ms
• For 1200B, the min/avg/max values at one hop are 166/181/205ms

[jwhui]

Tagging @EskoDijk here to provide some thoughts.

[EskoDijk]

@echargy There are 2 main things that make the OTNS results differ from real devices:

1. No simulation of processing delay (as you've mentioned) - this will especially be noticable for protocols that are crypto-heavy.
2. By default, there's no interference sources other than OTNS nodes themselves. They need to be manually added if desired.

For 2., it means the channel is almost always clear - no delays due to CCA. In reality, the channel is never perfectly clear.
OTNS has a command 'add wifi' to add a simulated "Wi-Fi like" interferer. It can also be tuned on the percentage of channel interference. This is one way to better reflect real-world conditions. There are more options, by changing radio model parameters (radioparam command - needs more docs!).

For 1., maybe a simple tunable radio node parameter can be added that emulates node processing delays (very roughly). This could be based upon PCAP captures of real nodes, maybe, to learn typical processing delays.

Here my OTNS results, using the latest version (Sep 15) and OT node version OPENTHREAD/thread-reference-20250612-218-geba5bdc43 , for a network of 2 Routers in total:

Ping size 10B: 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 5/6.860/12 ms.
Ping size 280B: 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 38/42.200/47 ms
Ping size 650B: 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 80/84.030/90 ms.
Ping size 1200B: 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 149/154.900/163 ms.

@echargy It looks like you used another OTNS version, seeing the difference in results?

When adding a "wifi" interferer node set to 90% interference, the ping result changes a bit, for example:

Ping size 280B: 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 39/48.830/67 ms.
Ping size 650B: 100 packets transmitted, 100 packets received. Packet loss = 0.0%. Round-trip min/avg/max = 81/96.930/122 ms.