|
| 1 | +- Feature Name: Stream Type |
| 2 | +- Start Date: 2023-03-10 |
| 3 | +- PR: [video-dev/media-ui-extensions#10](https://github.com/video-dev/media-ui-extensions/pull/10) |
| 4 | + |
| 5 | +# Summary |
| 6 | + |
| 7 | +This is a proposal to provide well defined behaviors and an API for distinguishing between "live" content vs. "on demand" (or pre-recorded). |
| 8 | + |
| 9 | +# Motivation |
| 10 | + |
| 11 | +The idea of different “stream types” has been around for a long time in various HTTP Adaptive Streaming (HAS) standards and its precursors in some manner - minimally distinguishing between “live” content and “video on demand” / "on demand" content. However, these categories aren’t consistently named or distinguished in the same way across the various specifications. Moreover, there is no corresponding API in the browser. Yet these categories directly inform how one expects users to consume and interact with the media, including what sort of UI or “chrome” should be made available for the user. By way of example, the built in controls/UI in Safari that show up for a live source are different than those that show up for a VOD playlist. This proposal aims to normalize the names and definitions of stream types (in a way that is extensible and evolvable over time) by way of how they are expected to be consumed and interacted with by a viewer/user. It also provides a concise and easy to understand differentiator for anyone implementing different UIs/controls/"chromes" for the various stream types. |
| 12 | + |
| 13 | +An additional goal of this proposal is to recommend for MSE-based players or “playback engines” to try to normalize their use of existing APIs to be as consistent as possible with the proposed inferred stream type Algorithm. |
| 14 | + |
| 15 | +# Guide-level Explanation |
| 16 | + |
| 17 | +## Definitions |
| 18 | + |
| 19 | +- **On Demand Media Stream** - Media content that is entirely available for playback at the same time. Also sometimes called "Video on Demand" or "VOD." |
| 20 | +- **Live Media Stream** - Media content that is made available for playback over time. |
| 21 | + - **NOTE** - This definition also applies to "DVR" or "sliding window" content, where the content is being published/provided in real-time but earlier content is also still available. |
| 22 | + |
| 23 | +## Usage |
| 24 | + |
| 25 | +The API provides a single property, `streamType`, that will identify whether the media content is `"live"` or `"on-demand"`. If no source is loaded or the stream type for the source has yet to be determined, the `streamType` will be `"unknown"` (to disambiguate from elements that have not implemented the Stream Type API). |
| 26 | + |
| 27 | +When the stream type is determined for a given media source, the extended media element will dispatch a `"streamtypechange` event. |
| 28 | + |
| 29 | +## Example: |
| 30 | + |
| 31 | +```js |
| 32 | +const mediaEl = document.querySelector('#extended-media'); |
| 33 | + |
| 34 | +// A "big play button" to show on initial player load/render |
| 35 | +const bigPlayButtonEl = document.querySelector('#big-play-button'); |
| 36 | +// Parent element containing the player's standard UI for both "live" and "on-demand" |
| 37 | +const playerUIEl = document.querySelector('#player-ui'); |
| 38 | +// Element in the UI for indicating playback of live content at the "live edge" |
| 39 | +const liveIndicatorEl = playerUIEl.querySelector('#live-indicator'); |
| 40 | +// Seek bar for the UI |
| 41 | +const seekBarEl = playerUIEl.querySelector('#seek-bar'); |
| 42 | + |
| 43 | +const updateUI = () => { |
| 44 | + // If the stream type is "unknown" (because no source has |
| 45 | + // been loaded, the previous source has been unloaded, or |
| 46 | + // the stream type is not yet derived from the source), only |
| 47 | + // show a "big play button". This also accounts for cases |
| 48 | + // where `preload="none"`. |
| 49 | + if (mediaEl.streamType === 'unknown') { |
| 50 | + playerUIEl.classList.add('hide'); |
| 51 | + bigPlayButtonEl.classList.remove('hide'); |
| 52 | + } |
| 53 | + // Otherwise, we know the stream type for the current source, |
| 54 | + // so show the appropropriate UI based on the stream type. |
| 55 | + else { |
| 56 | + // For "live", show the live indicator and hide the seek bar |
| 57 | + // NOTE: For "DVR" content ("live" content that is still intended |
| 58 | + // to be seekable), you may want to further differentiate here. |
| 59 | + if (mediaEl.streamType === 'live') { |
| 60 | + liveIndicatorEl.classList.remove('hide'); |
| 61 | + seekBarEl.classList.add('hide'); |
| 62 | + } |
| 63 | + // For "on-demand", do the opposite. |
| 64 | + // NOTE: In this example, if `mediaEl` has not implemented `streamType`, |
| 65 | + // it will show the same UI as "on-demand". |
| 66 | + else { |
| 67 | + seekBarEl.classList.remove('hide'); |
| 68 | + liveIndicatorEl.classList.add('hide'); |
| 69 | + } |
| 70 | + // Show the standard UI and hide the "big play button" |
| 71 | + bigPlayButtonEl.classList.add('hide'); |
| 72 | + playerUIEl.classList.remove('hide'); |
| 73 | + } |
| 74 | +}; |
| 75 | + |
| 76 | +// Update the UI whenever "streamtypechange" is dispatched. |
| 77 | +mediaEl.addEventListener('streamtypechange', updateUI); |
| 78 | +// Also update the UI whenever "emptied" is dispatched to account for unloading the media. |
| 79 | +mediaEl.addEventListener('emptied', updateUI); |
| 80 | +``` |
| 81 | + |
| 82 | +# Reference-level explanation |
| 83 | + |
| 84 | +## Property: `streamType` |
| 85 | + |
| 86 | +A string value that represents the type of media stream that _**MUST**_ be one of the values listed below. |
| 87 | + |
| 88 | +**NOTE**: Any computation of the `streamType` value that requires fetching of playlists, manifests, or the like, _**SHOULD**_ respect the load and `preload` state specified on the extended `HTMLMediaElement` instance. For example, if `preload="none"`, implementors should wait until e.g. a `"loadstart"` event is dispatched from the instance. |
| 89 | + |
| 90 | +### Possible values |
| 91 | + |
| 92 | +- `undefined` - Unimplemented, in which case UI implementors may rely on a "best guess" solution for modeling the stream type (See _§Recommended inferred value from native `HTMLMediaElement`_, below). |
| 93 | +- `"on-demand"` - The current loaded media source is an On Demand Stream. |
| 94 | +- `"live"` - The current loaded media source is a Live Stream (including potentially "DVR" or "sliding window" live streams). |
| 95 | +- `"unknown"` - No media source is set, the source is not yet loaded (e.g. because `preload="none"`), or the stream type has yet to be determined. |
| 96 | + |
| 97 | +### Recommended computation for RFC8216 (aka HLS) |
| 98 | + |
| 99 | +`streamType = (#EXT-X-PLAYLIST-TYPE === "VOD") ? "on-demand" : "live"` |
| 100 | + |
| 101 | +**Context**: |
| 102 | + |
| 103 | +> A Media Playlist has further constraints on its updates if it contains an EXT-X-PLAYLIST-TYPE tag. An EXT-X-PLAYLIST-TYPE tag with a value of VOD indicates that the Playlist file MUST NOT change. An EXT-X-PLAYLIST-TYPE tag with a value of EVENT indicates that the Server MUST NOT change or remove any part of the Playlist file, with the exception of EXT-X-PART tags and Media Metadata tags as described above; the Server MAY append lines to the Playlist. |
| 104 | +
|
| 105 | +\- From: https://datatracker.ietf.org/doc/html/draft-pantos-hls-rfc8216bis-12#section-6.2.1 |
| 106 | + |
| 107 | +#### NOTE: Why not `#EXT-X-ENDLIST` detection? |
| 108 | + |
| 109 | +The `#EXT-X-ENDLIST` tag only indicates that a particular playlist will no longer be updated with additional Media Segments or media segment parts (`#EXTINF` / `#EXT-X-PART-INF`). Using this value therefore conflates, e.g., an ended live stream with an on demand stream, and thus should not be relied upon for stream type detection. |
| 110 | + |
| 111 | +### Recommended computation for ISO/IEC 23009-1 (aka "MPEG-DASH") |
| 112 | + |
| 113 | +`streamType = (MPD@type === "dynamic") ? "live" : "on-demand` |
| 114 | + |
| 115 | +**Context**: |
| 116 | + |
| 117 | +MPD@type: |
| 118 | + |
| 119 | +> default: `static` |
| 120 | +> specifies the type of the Media Presentation. For static Media Presentations (`@type="static"`), all Segments are available between the `@availabilityStartTime` and the `@availabilityEndTime`. For dynamic Media Presentations (`@type="dynamic"`), Segments typically have different availability times. |
| 121 | +
|
| 122 | +\- From: _§5.3.1.2 Table 3 - Semantics of MPD element_ |
| 123 | + |
| 124 | +### Recommended inferred value from native `HTMLMediaElement` |
| 125 | + |
| 126 | +```js |
| 127 | +streamType = Number.isFinite(mediaEl.duration) |
| 128 | + ? 'on-demand' |
| 129 | + : mediaEl.duration === Number.POSITIVE_INFINITY |
| 130 | + ? 'live' |
| 131 | + : 'unknown'; |
| 132 | +``` |
| 133 | + |
| 134 | +**Context**: |
| 135 | + |
| 136 | +`media.duration`: |
| 137 | + |
| 138 | +> Returns the length of the media resource, in seconds, assuming that the start of the media resource is at time zero. <br/> > Returns NaN if the duration isn't available. <br/> > Returns Infinity for unbounded streams. |
| 139 | +
|
| 140 | +\- From: https://html.spec.whatwg.org/multipage/media.html#dom-media-duration-dev |
| 141 | + |
| 142 | +## Event: `"streamtypechange"` |
| 143 | + |
| 144 | +This event should be dispatched from an extended `HTMLMediaElement` instance whenever the stream type has been determined for a loaded media source. |
| 145 | + |
| 146 | +**NOTE**: For media unload cases, implementors _**SHOULD**_ ensure the `streamType` is (re)set to `"unknown"` after an unload begins but prior to any `"emptied"` external event handlers being invoked. This ensures that consumers of the API may reliably use the `"emptied"` event to monitor potential changes in `"streamType"`. |
| 147 | + |
| 148 | +# Rationale and alternatives |
| 149 | + |
| 150 | +This proposed solution provides a simple API with simple values for exposing information that is useful for UI development that distinguishes between live and on demand content, but is only guaranteed to be available by the code/entity responsible for parsing the media data. Based on the scope of currently proposed stream types, an alternative implementation could potentially rely on `duration` values (corresponding to the values used in _§Recommended inferred value from native `HTMLMediaElement`_, above). Below is a discussion on why this is not recommended. |
| 151 | + |
| 152 | +## Why not simply rely on `duration`? |
| 153 | + |
| 154 | +While there are elements of various specifications that suggest live content _should_, under most circumstances, have a `duration=Infinity` (See, e.g. https://html.spec.whatwg.org/multipage/media.html#dom-media-duration-dev, https://www.w3.org/TR/media-source-2/#dom-mediasource-setliveseekablerange, https://www.w3.org/TR/media-source-2/#dfn-duration-change), there are a few reasons to avoid this. |
| 155 | + |
| 156 | +1. Although https://html.spec.whatwg.org/multipage/media.html#dom-media-duration-dev strongly suggests that `duration` _**SHOULD**_ be `Infinity` for live streams, there is at least some ambiguity between "live" vs. "unbounded" (referencing the language from the specification), so conflating the two may be inappropriate. |
| 157 | +2. At least some "playback engines" _**MAY**_ set a finite duration for live streams, at least by default. While I do not know of any cases, this may also be true for some native playback environments. |
| 158 | +3. As with the case of `#EXT-X-ENDLIST` in RFC8216 (aka "HLS"), the `duration` of live streams that have ended will typically be set to a finite value, leading to the same problems as discussed above at end of stream. This is arguably the most compelling reason to avoid conflating the two. |
| 159 | +4. By avoiding unnecessary constraints on `duration`, we leave room for alternative media ui extension proposals that may want to augment this value for other, duration-specific reasons. |
| 160 | +5. By using a custom `streamType` property with well-defined enumerated values, we increase legibility for develoeprs and leave room for extending the set of stream types in subsequent proposals, if merited. |
| 161 | +6. By using a custom `streamType` property, it is possible to polyfill the `HTMLMediaElement` without _requiring_ the use of e.g. Web Components (though this is still recommended & encouraged). |
0 commit comments