app_tx_st30p stop/start does not work

[DianaEs2]

Describe the bug
The bug happens when a Tx session is restarted without stopping the MTL. Tx session transmits audio (or video) successfully on the first run, but after stopping and restarting the same session, no audio is received.

To Reproduce
Update RxTxApp to restart tx_st30p_session every 10 seconds.
Restarting a session performed as st_app_tx_st30p_sessions_uinit()->st_app_tx_st30p_sessions_init()
You can use the following diff and apply it to MTL v25.02
update.patch

Expected behavior
It is expected that Tx session can be stopped and restarted.

Configuration and command-line
The application runs with tx_rx_1a.json
tx_rx_1a.json
There are no extra parameters in the command line.
"program": "${workspaceFolder}/tests/tools/RxTxApp/build/RxTxApp",
"args": ["--config_file", "${workspaceFolder}/config/tx_rx_1a.json"]

Status report
mtl_system_status_20250422160134.zip

Logs
The modified RxTxApp produces the following log
mtl_log_level_info.txt
Please find line 205, where Rx session stopped receiving the stream on the second run.

HW specification (please complete the following information):

• MTL Version: 25.02
• OS: Ubuntu 22.04 LTS

Screenshots

Additional context
This problem is not observed with Rx session.

[DianaEs2]

Hi, are there any updates on this case? I can provide more information if required. Thanks!

[Sakoram]

Hi, sorry, I had no time to look into this. I will do it in the next few days

[Sakoram]

Hi, I took a look at the issue.
First question, why do you force the same IP address to be assigned to both interfaces? This will cause problems if the interfaces are on the same network. I suggest using different IP addresses.

I was able to reproduce epoch drops seen in your log: TX_AUDIO_SESSION(0,0): epoch drop 1576046462
The problem seems to occur only right after the session is restarted.
When I increase the reset period to 30 seconds, or when I use the MTL init paremeter dump_period_s = 1 I see drops only in the first log after reset, after that on my setup everything looks good. ( my guess is that after reset hardware queues are full, so it takes some time after "new" session can start sending ). It should be assumed that it will take some time to reinitialize the session, and the data stream will be interrupted.

In your log at line 296 I see that some data has been send, TX_AUDIO_SESSION(0,0:app_tx_st30p_0): fps 94.709426 frames 144, pkts 1437:0 inflight 0:0 that suggest that the TX session restarts, but maybe on your setup it takes even longer then on mine.

Please check what happens when the reset period is increased to, say, 50 seconds. The main question is whether the communication fails to be reinitialized at all or simply takes a long time. ( the normal initialization is much quicker, but I just want to get multiple statistics prints )

[DianaEs2]

Thanks for checking this issue. Below are the test results with recommended updates.

The main issue (loss of signal) is still reproducible.

Here is the code update:
update_v2.patch

1. Enable the second IP for the receiver. The new config json
   tx_rx_1a.json
2. Change the logging period to 1 second
3. Restart tx_session only once in hope that it may come back later.
   mtl_test_v2.log

Just in case, pcap for the test.
audio_test_v2.pcapng.gz

[DianaEs2]

The bug may depend on the host machine where the test is running. We are using HPE ProLiant DL380 Gen10 and DL345 Gen11. The bug is reproducible on Gen11 but not on Gen10. Or, more accurately would be to say that the bug is reproducible much more often on Gen11. I am providing a log from a successful test on Gen10 for comparison.

Hardware details for Gen10.
mtl_system_status_20250501145622.zip

A similar test was performed with a single restart of tx_session.
mtl_test_v3.log
tx_rx_1a.json

The analyzer shows signal recover as expected

Here is pcap of the successful test
mtl_test_v3.pcapng.gz

[cwhite102]

Just to add another data point... we see this stop/start issue on video (st-20) too. When stopping and then starting the video Tx session, it sometimes gives corrupted data, sometimes no data, and sometimes good data. Like audio, this only seems to happen on the HP Gen11 systems, not the Gen10 boxes.

Here's an image showing a corrupted stop/start stream in a multi-viewer...
https://postimg.cc/ZCdbpfVt

(sorry - github is giving a CORS error when trying to upload the image directly. I threw it on a hosting site instead)

[Sakoram]

It seems like the most obvious difference between Gen10 and Gen11 is the CPU. I'm currently not able to test MTL with AMD EPYC, so I'm not able to reproduce the issue, and sadly not able to help you.

( I have no clue why CPU could cause the issue )

[Sakoram]

I just notice one important difference. I see that you use old (1.13.7) version of e810 ICE driver on gen 11. Please consider updating to 1.14.9

[cwhite102]

Unfortunately, updating to ICE v1.14.9 didn't help. (MTL 25.02)

Here's a video (for the st2110-20 case) which may help show the symptoms better.
All these 2110 Tx streams are coming from the same HPGen11 box, using MTL in different processes. (no mtl manager, we configure the lcores).
You can see the second MTL Tx session is stopped and started twice during the video. When it is restarted, you can see the MTL 2110 output behavior.
All these 2110 Tx feeds are being received by another piece of industry 2110 equipment and presented as a multiviewer display (in a browser) which I screen recorded.

2025-05-14_00-27-22.mp4

I very highly doubt this has anything to do with the AMD cpu. There's a bug in the MTL (or lower) layers somewhere. This system just make it show up readily.

The behaviour is the same for a single Tx process... this is just what I had running when I took the video and to show the other processes were fine.

[DianaEs2]

Sakoram, is there a way to assist you in debugging this issue? Perhaps, you can advise us on where to put additional logging. How do you typically debug such issues? Will it help if we provide SSH access to this Gen11 box? Any ideas are welcome.

[Sakoram]

Hi, write to us at MTL_support@intel.com we will try to organize something and potentially arrange debugging.