obs-webrtc: Add Simulcast Support

[Sean-Der]

Description

This PR adds supports for Simulcast to the obs-webrtc output. Simulcast allows for multiple quality levels to be sent over one track in WebRTC.

RFC 8853 is a technical description. For a less technical explanation Wowza, Dolby and LiveKit have all written articles on it.

This PR starts with adding a Spinbox when WHIP is selected. This allows the user to choose how many layers to send in total. The acceptable range (for now) is 1-4.

The Height/Width/Bitrate is scaled from the users global choice. So depending on the users choice.

• 1 Layers (
  • 100% Height/Width/Bitrate
• 2 Layers
  • 100% Height/Width/Bitrate,
  • 50% Height/Width/Bitrate
• 3 Layers
  • 100% Height/Width/Bitrate
  • 66% Height/Width/Bitrate
  • 33% Height/Width/Bitrate
• 4 Layers
  • 100% Height/Width/Bitrate
  • 75% Height/Width/Bitrate
  • 50% Height/Width/Bitrate
  • 25% Height/Width/Bitrate

A server may wish to only accept a subset of the layers offered. Today if the user rejects any layers we throw an error and say how many layers were accepted. The user can lower the amount of layers they are sending and try again. Services in their documentation will be expected to tell users how they want video encoded and how many layers they are willing to accept.

Motivation and Context

• Low Latency - Decoding/Re-encoding adds too much latency for interactive content
• Make self hosting easier - Running ffmpeg is costly and difficult to scale
• Streamers more control - If streamers can generate transcodes they control the quality
• E2E Encryption/Prevent Modification- Streamers can distribute keys to viewers and servers can't modify video anymore.

Some questions that have been asked by reviewers already.

What about Enhanced Broadcasting? - Simulcast is part of an existing standard that already sees a billion minutes per day so it is here to stay and is already deployed massively. I would like to see Enhanced Broadcasting in the JSON blob be able to specify the protocol. When a user starts a enhanced broadcasting session it should be able to switch between RTMP/WebRTC/$x. I think both flows can/should exist because they serve different purposes.

UI/UX Considerations - A single opt-in is non-intrusive and respects the user. It would be good to get feedback from all OBS WebRTC users (Dolby, LiveKit, Cloudflare etc...) that this is a sound decision though.

How Has This Been Tested?

This has been tested with the nvenc, qsv, and x264.

I have tested this against Twitch, Broadcast Box, LiveKit and Dolby.

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[tytan652]

Hi @tytan652 I see you gave a 👎 was that an accident?

No, I used that to avoid always adding an "off-topic" to PRs saying that the PR adds code complexity to the streaming codepath which adds more work to my plans for a Service Overhaul.

Do you have any feedback/things I can improve?

No, this is why I just reacted.

[czoli1976]

About time we get Simulcast for WHIP, holy cow.

[koolscooby]

This would be great. We'd like to have Twitch Stream Together users be able to use OBS and use Simulcast to send multiple qualities to improve the participant experience. It's a much better quality encoding & transmission experience compared to in-browser content contribution.

One UI nit / question: does checking the box force simulcast, or does it just enable it if the server/SDP exchange also indicates support for simulcast? If the latter, I'd recommend changing the UI label to Enable simulcast which makes it a bit clear that it is enabled on the client, but also requires the server to support it.

[koolscooby]

@koolscooby Fixed! Changed from Use -> Enabled

If the server rejects Simulcast it will only accept the high layer.

In the future the server should be able to drive more (how many layers, suggestions around bitrate/resolution). For a V1 I think it just being .5x and .25x is a good trade-off on complexity vs value.

This seems like a completely sensible tradeoff to start getting users testing the underlying simulcast functionality in OBS & on the server side in a bigger set of network scenarios.

edit: ... without creating an enormous amount UI code or complicated UX.

[murillo128]

Anything we can do to get this merged sooner (review/test/etc..)? From Dolby Millicast side we are really interested in this feature which would allow us to finally deprecate our custom fork.

[Fenrirthviti]

We're aware this is here, and will get to it in due time. More testing/documentation is always welcome to help things.

[davidzhao]

There are a lot of LiveKit users that are excited about the ability to simulcast with OBS. Let me know if we can help in getting this across the finish line

[Fenrirthviti]

Logging for posterity that this has an open RFC for OBS with unresolved feedback that has not been accepted.

While not impossible this is merged first, it's a little strange to submit a functional PR of a design that has not been accepted by the project or finalized yet. I appreciate the enthusiasm but "move fast, fix later" is at odds with how the project operates in general.

This is not a request for discussion on this point, I am posting for visibility of process.

[nils-ohlmeier]

We as Cloudflare are in support of adding Simulcast support to OBS. Having Simulcast support in OBS will enable new use cases for certain users and services.
I think the default values chosen in this PR for 1/2 and 1/4 resolution are used by a lot of video conferencing services in their Simulcast settings and make perfect sense as the initial values for this feature.

[kwindla]

I'd like to weigh in to say that we (Daily.co) have users who need this. In general, our experience is that simulcast is critical for expanding the use cases and kinds of users who can stream successfully. Two examples: 1) users who don't have control over their upstream bandwidth. Perhaps they are on a corporate network with unpredictable congestion. 2) users who want to live stream and send the stream out to multiple different protocols/channels in real-time without re-processing.

[Sean-Der]

I have had conversations with OBS Maintainers, WHIP Services and the WHIP spec author about Simulcast and have updated the PR from that feedback. We want to get an MVP into OBS Studio so we can build towards the next/future items and would appreciate another round of review now that I've refactored the implementation to follow the MVP.

(MVP) - Users are given a Spinbox to select how many Simulcast layers to send. If the remote server doesn't accept the configuration throw an error modal. A screenshot of this is in the PR description.

(Next) - Explore options to allow servers to configure/hint layers even before the user hits 'Start Streaming', but would not dynamically change while connected. Users wouldn't need to configure anything. The behavior would feel closer to Twitch's Enhanced Broadcasting.

(Future) Implement fully dynamic simulcast. OBS could enable/disable layers on the fly depending on CPU/GPU/Network usage, similar to how browsers implement WebRTC Simulcast.

[Bleuzen]

In this current form this is only really compatible with CBR mode (but the UI doesn't make that clear).
For example I can set the encoder to VBR mode and it only changes resolution and bitrate for each layer but the max bitrate stays the same for all layers which doesn't make sense.

So for this first version I suggest to either only allow users to enable Simulcast in CBR mode or add some special handling for other rate control modes (at least VBR makes sense because in my experience some browsers don't like the bitstream padding from CBR so VBR is more stable for WebRTC streams.)

For the future it would be cool to have entirely separate settings per layer. I'm not a fan of the calculated resolutions & bitrates here and would rather choose my own and also do things like having one layer with AV1 and another layer with H264. But that is something for the future / another PR I guess.

[Sean-Der]

@Bleuzen those are all great enhancements. In the future I would like to see

• Dynamic toggling of tracks (don’t generate if no one is viewing)
• Multi codec
• User selection of resolution/bitrate

It’s overwhelmingly complicated though. I hope we can keep this PR as is and just get it merged. If people find it useful I would love to do all this :)

[mpisat]

We are all waiting for this to be merged. It works really well with OvenMediaEngine.
I hope this gets prioritized soon.

[Sean-Der]

I just rebased and fixed merge conflicts, @mpisat should be good to go! Sorry that took so long :(

[mpisat]

Will download artifacts and test out. Thank you!

[naikrovek]

For the future it would be cool to have entirely separate settings per layer. I'm not a fan of the calculated resolutions & bitrates here and would rather choose my own and also do things like having one layer with AV1 and another layer with H264. But that is something for the future / another PR I guess.

I agree with this completely. More control over the layers would be ideal.

[dascope]

Awesome, we are also excited and waiting for this since we supported WHIP support in Digital Samba 611.

[Sean-Der]

Having that control would be amazing @naikrovek!

For this first version I discussed with @Warchamp7 and this was the UI/behavior that gave most people what they need/with as little complexity. Do you agree with that still @Warchamp7, I don't want to speak for you!

Any other changes that should happen, or this is good.

[IsaiahJTurner]

Excited for the current implementation to merge, this will definitely meet our needs to stream HIGH, MEDIUM, and LOW layers to LiveKit! Long term, additional customization sounds like a great improvement.

Also, found this PR after A LOT of digging to understand where the Simulcast option that is included in the documentation was at.

[mpisat]

Excited for the current implementation to merge, this will definitely meet our needs to stream HIGH, MEDIUM, and LOW layers to LiveKit! Long term, additional customization sounds like a great improvement.

Also, found this PR after A LOT of digging to understand where the Simulcast option that is included in the documentation was at.

Livekit Ingress doesn't support simulcast no-transcoding bypass with this OBS. I tested it, it recognizes the layers but layer switching doesn't work. LiveKit needs to modify code to make that happen. I tried to fix myself but I failed.

However, if this goes mainstream, and merged to OBS, they might fix it. It'd be amazing.
With current obs-simulcast implementation, OvenmediaEngine works pretty awesome (kudos to Ovenmedia team)

[davidzhao]

@mpisat we'll definitely update our implementation on our side. native WHIP is coming to the SFU soon.

[peerasan]

Awesome! good to know. I wondering why this feature isn't on OBS yet.

[Fenrirthviti]

This is currently being targeted for 32.1, but requires final review and approvals still on our end, which have not yet been completed.

[Sean-Der]

@peerasan You can use builds from GitHub actions, do those work for you? If you have any problems always here to help :)

[Warchamp7]

Couple minor things to fix, but otherwise this follows all of the suggestions and guidance that have been provided off-thread for an MVP on the UX side.

[Sean-Der]

Thank you so much @Warchamp7 I am so grateful for the review :) This has been the thing that I have thought 'This is what makes Broadcast Box really cool'.