What are the limitation in encodings/protocols ?

[Doreapp]

Hello,
The USB camera I use on the drone propose an output in MJPEG with specs way better than the output his raw format (YUYV).
On casual LAN Wi-Fi network, a gstreamer pipeline using MJPEG camera output, and transmitting the data using RTP protocol works very well (with very low latency) but when using WifiBroadcast:

• If I use the RTP protocol, then some data is received, but almost one third of the packets are lost, and no video is displayed on the GS
• If I send directly MJPEG encoded video (with a low resolution so that the packet size isn't too high), I only manage to display the top line of the video (about 1/15th of the video)

However, when using H264 encoding, with RTP, the video streaming is ok, with both:

• Test video src (Using raw format NV12)
• Real video src, where I need to decode the MJPEG video and then re-encode it in H264

But the video delay is quite bad.

So, here come my questions: is there some limitation in the encodings supported? And if not, what I my doing wrong / what parameters should I edit ?

---

As notes, here are the GStreamer pipelines I am using:

• On casual network
  • Without RTP

  gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=640, height=480, framerate=30/1 ! udpsink host=192.168.254.100 port=5000
  

  gst-launch-1.0 -v udpsrc port=5000 ! image/jpeg, width=640, height=480, framerate=30/1 ! jpegdec ! videoconvert ! xvimagesink
  

  • With RTP

  gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=1280, height=960, framerate=30/1 ! rtpjpegpay ! udpsink host=192.168.254.100 port=5600
  

  gst-launch-1.0 -v udpsrc port=5600 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=JPEG, payload=26, a-framerate=(string)30.0" ! rtpjpegdepay ! image/jpeg, width=1280, height=960, framerate=30/1 ! jpegdec ! videoconvert ! xvimagesink
  

• Using WifiBroadcast
  • Testing H264 (saw from other discussions)

  gst-launch-1.0 -v videotestsrc ! videoconvert ! video/x-raw, format=NV12, width=640, height=480, framerate=30/1 ! x264enc bitrate=4000 tune=zerolatency ! rtph264pay config-interval=1 ! udpsink host=127.0.0.1 port=5602
  

  gst-launch-1.0 -v udpsrc port=5600 caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264" ! rtph264depay ! avdec_h264 ! xvimagesink
  

  • Using Camera input and JPEG decoding

  gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=640, height=480, framerate=30/1 ! jpegdec ! videoconvert ! x264enc bitrate=4000 tune=zerolatency ! rtph264pay config-interval=1 ! udpsink host=127.0.0.1 port=5602
  

  .. And the same receiver (GS) pipeline as above

---

Notes:

• On Drone I am using an Odroid xu4 running Ubuntu 20.04
• On GS, it is a Samsung laptop running Ubuntu 20.04
• Both cards are TL-WN722N, but I also tried with ALFA AWUS036NHA in RX
• I use WLAN channel 10, and it should not be polluted

---

Thank you in advance !

[svpcom]

There are two problems MJPEG:

1. With raw udpsink (without) RTP is produces udp packet with size >> card mtu (1500). UDP over IPv4 can split large packets into fragments, but WFB send them as-is and shrinks and/or drops large packets
2. With RTP MJPEG can work, but its bandwidth is greater that WFB wifi link bandwidth (in my test with laptop camera in 640x480 it is ~ 10mbit/s but WFB link limit for MCS#1 modulation is ~6mbit/s)

$ gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=640, height=480, framerate=30/1 ! rtpjpegpay ! udpsink host=127.0.0.1 port=5600
...
$ nc -l -u 5600 | pv > /dev/null
46MiB 0:00:07 [1007KiB/s] [                <=>                       ]

ODroid XU4 (as I remember) has onboard hw h264 encoder so you can try to use it to offload video encoding.

P.S. Try to use latest WFB version (or 21.03.2 release), because previous WFB releases had long-lived (since 20.12) FEC bug

[svpcom]

Also I don't recommend to use any Ralink cards (rt28xx diver) - they have broken injection in 5.x kernels. Use rtl8812au instead

[Doreapp]

Thank you for your answer!

• I'll try using h264 in an optimized way, if I find a nice pipeline I'll post it here, as it may help some folk
• Thank you for your precision on WFB versions and Wi-Fi cards, I appreciate

[Doreapp]

So, after some tests, it is indeed only working using H264.

My pipelines, as documentation:

• Drone

  gst-launch-1.0 -v v4l2src device=/dev/video0 io-mode=2 ! image/jpeg, width=640, height=480 ! jpegdec ! videoconvert ! x264enc bitrate=4000 tune=zerolatency ! rtph264pay config-interval=1 ! udpsink host=127.0.0.1 port=5602
  

• GS

  gst-launch-1.0 -v udpsrc port=5600 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=H264, payload=26, a-framerate=(string)30.0" ! rtph264depay ! avdec_h264 ! xvimagesink
  

Note: It is not limited in resolution, it also works with 1280x960 for example.

[svpcom]

In h264 pipeline video bitrate sets explicitly and doesn't depend from resolution (with bigger resolution quality will be lower if bitrate is same)
In MJPEG bitrate is implicit function from video resulution