Calls: Introduce simulcast API

[renandincer]

This is a first stab at introducing simulcast into the Cloudflare Calls SFU API.

Examples

Example 1: Using the /tracks/change endpoint to reuse a transceiver

This example shows how to reuse an existing transceiver (with mid "2") to receive a different track from a remote session.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/change" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tracks": {
      "video-track-1": {
        "location": "remote",
        "sessionId": "2a45361d5fd7cc14eface0587c276c94",
        "trackName": "new-video-track",
        "mid": "2"
      }
    }
  }'

Example 2: Using the /tracks/change endpoint with simulcast preferences

This example demonstrates how to reuse a transceiver while specifying simulcast preferences, requesting the high-quality stream ("h") and falling back to the least bandwidth option if that's not available.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/change" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tracks": {
      "simulcast-video": {
        "location": "remote",
        "sessionId": "2a45361d5fd7cc14eface0587c276c94",
        "trackName": "simulcast-camera",
        "mid": "3",
        "simulcast": {
          "preferredRid": "h",
          "preferredRidNotAvailable": "leastbandwidth"
        }
      }
    }
  }'

Example 3: Changing multiple tracks simultaneously

This example shows how to change multiple tracks in a single request, reusing different transceivers for different remote tracks.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/change" \
 -H "Authorization: Bearer YOUR_API_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
   "tracks": {
     "video-track": {
       "location": "remote",
       "sessionId": "2a45361d5fd7cc14eface0587c276c94",
       "trackName": "camera-feed",
       "mid": "2"
     },
     "audio-track": {
       "location": "remote",
       "sessionId": "2a45361d5fd7cc14eface0587c276c94",
       "trackName": "microphone",
       "mid": "0"
     }
   }
 }'

Example 4: Using the /tracks/new endpoint with simulcast

This example demonstrates adding a new local track with simulcast capabilities, specifying the SDP with simulcast attributes and indicating a preference for the high-quality stream.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/new" \
-H "Authorization: Bearer YOUR_API_TOKEN" \
-H "Content-Type: application/json" \
-d '{
  "sessionDescription": {
    "sdp": "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nc=IN IP4 127.0.0.1\r\nt=0 0\r\nm=video 4002 RTP/AVP 96\r\na=rtpmap:96 VP8/90000\r\na=simulcast:send f;h;q\r\na=rid:f send\r\na=rid:h send\r\na=rid:q send\r\n",
    "type": "offer"
  },
  "tracks": [
    {
      "location": "local",
      "trackName": "simulcast-camera-feed",
      "mid": "0",
      "simulcast": {
        "preferredRid": "h",
        "preferredRidNotAvailable": "leastbandwidth"
      }
    }
  ]
}'

Example 5: Using the /tracks/change endpoint with a different fallback strategy

This example shows how to request a specific quality level (in this case, the lowest quality "q") with a fallback strategy of "none", which means no fallback will be attempted if the preferred RID is not available.

curl -X POST "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/change" \
 -H "Authorization: Bearer YOUR_API_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
   "tracks": {
     "simulcast-video": {
       "location": "remote",
       "sessionId": "2a45361d5fd7cc14eface0587c276c94",
       "trackName": "simulcast-camera",
       "mid": "3",
       "simulcast": {
         "preferredRid": "q",
         "preferredRidNotAvailable": "none"
       }
     }
   }
 }'

Open Questions

The following case is not handled: "I REALLY want the RID I just told you, but if the RID is no longer being published, then downgrade. However I'm okay with packet loss with a higher rid I can't handle", because we are using preferredRidNotAvailable for both "RID stopped published" and also "you don't have enough bandwidth for this RID"

[github-actions]

Preview URL: https://3387b898.preview.developers.cloudflare.com
Preview Branch URL: https://introduce-simulcast-api.preview.developers.cloudflare.com

[nils-ohlmeier]

Some initial quick points:

Example 4 is not what we had been discussing. When you push a track the tracks object should have no simulcast in it at all. Conceptually I don't even know what a preferredRid from the publishers point of view would mean.
But this does raise an interesting question: should we respond with an error if someone makes an API call like that, or just ignore the simulcast part?

Should we use /track/change or /track/update?
I don't have a strong preference either way, just wanted to mention here in case someone cares about it.
But I would prefer if we use PUT for the /track/change (or /update) endpoint.

Regarding the open question "I REALLY want the RID I just told you, but if the RID is no longer being published, then downgrade. However I'm okay with packet loss with a higher rid I can't handle":

• the first part of your quote makes sense and the current proposal allows the client to express this (even more detailed by saying how to downgrade).
• The second sentence in quote doesn't really compute for me. Are you trying to say "Please upgrade me, rather downgrade"? If so I don't think we currently have a way to request an upgrade. The part which doesn't make sense to me is specifically the packet loss part. Packet loss on video tracks will result in retransmissions, which overall ends up using more bandwidth. Where you trying to say "However I'm okay with using more bandwidth with a higher RID"?. The important part is that if we talk about bandwidth: if the higher bandwidth RID doesn't fit through the downlink it really doesn't make sense to try to push it through. It will only result in frozen/stuttering video. I'll stop rambling now and let you clarify what you mean. :-)

[renandincer]

Example 4 is not what we had been discussing. When you push a track the tracks object should have no simulcast in it at all. Conceptually I don't even know what a preferredRid from the publishers point of view would mean.

This is an oversight from my side. What I mean was new remote track with simulcast capabilities, not local. When you pull a brand new track, there should be an option to specify simulcast options though. We can't just depend on the change endpoint to handle simulcast.

But this does raise an interesting question: should we respond with an error if someone makes an API call like that, or just ignore the simulcast part?

I'm leaning towards erroring because it's a blatant issue with the API definition, push with simulcast doesn't make sense.

Should we use /track/change or /track/update?
I don't have a strong preference either way, just wanted to mention here in case someone cares about it.
But I would prefer if we use PUT for the /track/change (or /update) endpoint.

I would also prefer PUT at /track or /tracks as well

The second sentence in quote doesn't really compute for me.

What if you wanted to ignore all bandwidth estimation changes but keep the changes when a rid disappears entirely?

[nils-ohlmeier]

Example 4 is not what we had been discussing. When you push a track the tracks object should have no simulcast in it at all. Conceptually I don't even know what a preferredRid from the publishers point of view would mean.

This is an oversight from my side. What I mean was new remote track with simulcast capabilities, not local. When you pull a brand new track, there should be an option to specify simulcast options though. We can't just depend on the change endpoint to handle simulcast.

Yes when you pull a new track the /track/new endpoint will support the simulcast object for sure.

But this does raise an interesting question: should we respond with an error if someone makes an API call like that, or just ignore the simulcast part?

I'm leaning towards erroring because it's a blatant issue with the API definition, push with simulcast doesn't make sense.

Okay.

Should we use /track/change or /track/update?
I don't have a strong preference either way, just wanted to mention here in case someone cares about it.
But I would prefer if we use PUT for the /track/change (or /update) endpoint.

I would also prefer PUT at /track or /tracks as well

Great. We already have PUT for /track/close, which makes sense to me because again this attempts to modify an existing resource.

The second sentence in quote doesn't really compute for me.

What if you wanted to ignore all bandwidth estimation changes but keep the changes when a rid disappears entirely?

Ahh. Yeah I think that scenario highlights your concern if we need two different parameters:

• one for not available (any more)
• one for how to respond to issues with the downlink of the pulling client
  Which probably makes this easier to comprehend, rather than trying to squeeze these two dimensions into on parameter.

[astroza]

Do you agree we have a consensus about the HTTP method and endpoint?

curl -X PUT "https://rtc.live.cloudflare.com/v1/apps/YOUR_APP_ID/sessions/e017a2629c754fedc1f7d8587e06d126/tracks/update"

[astroza]

Regarding the request body I lean towards having schema/TracksRequest for it OR a subset of it that does not include the sdp and autoDiscover fields.

TracksRequest/subset should be updated to have the simulcast field

The subset could be look like:

    TracksUpdate:
      type: object
      properties:
        tracks:
          type: array
          items:
            $ref: "#/components/schemas/TrackObject"

and TrackObject should have the recently added simulcast field

[astroza]

In relation to the simulcast field:

• I think there is no doubt about keeping preferredRid ✅
• preferredRidNotAvailable
  • The name may not suggest what it does
  • It should not be exclusive for preferredRid: Any other option should default to this field
    Example: {preferredRid: "h" }, { minWidth: 1024 }, { maxBandwidth: 1000000 } (the last 2 are potential fields, we are not defining them here)
    If they can't be meet we should use a backup criteria, so what could be that field name?
    • { fallback: "leastbandwidth" }
    • { backupCriteria: "auto" } // auto would send the best quality as possible
    • something else