latency/framedrops with NDI input

[Friismestemacher]

hey there,
i use raspberry ninja as a NDi to webRTC brigde by using the --pipeline statement and having a NDI gstreamer plugin. That works in general, however I get some ugly framedrops and latency issues and I am woundering if that is my pipeline that might be not ideal or if there is something in rasberry_ninja going on.

That is my pipeline:
webrtcbin name=sendrecv stun-server=stun://stun4.l.google.com:19302 bundle-policy=max-bundle ndisrc ndi-name="MACBOOK-PRO-2\ \(NDI1\)" url-address=192.168.0.224:5961 ! ndisrcdemux name=demux demux.video ! queue ! videoconvert ! queue max-size-buffers=1 ! openh264enc bitrate=2500000 name="encoder" complexity=0 ! video/x-h264,profile=constrained-baseline ! queue max-size-time=1000000000 max-size-bytes=10000000000 max-size-buffers=1000000 ! h264parse ! rtph264pay config-interval=-1 aggregate-mode=zero-latency ! application/x-rtp,media=video,encoding-name=H264,payload=96 ! queue ! sendrecv. demux.audio ! queue ! audioconvert ! audioresample quality=0 resample-method=0 ! opusenc bitrate-type=1 bitrate=64000 inband-fec=true ! rtpopuspay pt=100 ssrc=1 ! application/x-rtp,media=audio,encoding-name=OPUS,payload=100 ! queue ! sendrecv.

And this is how the output of rasberry_ninja looks like, with a new ping happening like almost every second:

PINGED
Packet loss:0
Trying to increase change bitrate...
2500
object of type `GstVP8Enc' does not have property `bitrate'
PONG: 1682786541.0604208
PINGED

Thats a lot for your help in the last time!

[steveseguin]

What platform / system are you using? a linux desktop or a raspberry pi? If doing software based encoding, you'll want to make sure you have enough CPU cycles, otherwise it can stall out. Raspberry Pis are quite sensitive.

What version of Gstreamer are you using? Newest versions seem to work a bit better with error controls than older versions, and really old versions really struggle with webRTC in general.

Does using Firefox for playback help?

You can try using --noqos I suppose to get rid of that one error message. I'm not entirely sure why its mentioning VP8 if h264 is selected.

I'm not sure what the NDI source is; raw video? Does it need any sort of buffer on its own before being encoded?

You can disable the pinging logic if you want. pingTimer on line 214 or so. client['timer'] = threading.Timer(3, pingTimer).start() -- you can comment this out and see if it helps.

Maybe try with the rtph264pay without aggregate-mode=zero-latency ; sometimes that can have it work more smoothly

Have you tested on a local system; local to local, or ethernet to ethernet, to avoid any wifi or network issues?

Let me know how it goes and I can try to replicate the setup here otherwise.