Chrome does not recognise old keyframes (LL-HLS issue)

[kanongil]

It appears that Chrome does not recognise keyframes that are inserted before the current media playhead. With LL-HLS and the current buffer appending logic, this can cause a playback freeze until a new keyframe appears, as documented below.

The Chrome behaviour is not new, eg. #680, but requires specific circumstances to trigger (near playhead, non-independent segments and/or bad keyframe alignment across levels). However, with LL-HLS, it is likely to be triggered for normal playback, since the PART-HOLD-BACK is easily less than a segment duration.

Ideas to hack around the issue:

1. Detect when this will happen and do a seek to the current playhead once the new level data has been buffered. It could cause Chrome to re-evaluate the buffered contents.
2. Add a "crappy MSE mode", which won't allow a level switch until the new level has a future independent part (relative to the current playhead) – possibly excluding forced / emergency switches.

What version of Hls.js are you using?

1.0.0-rc4

What browser and OS are you using?

Chrome 88.0.4324.192
Big Sur 11.2.3 (arm)

Test stream:

https://stream.sob.m-dn.net/live/sb1-ll/index.m3u8

https://hls-js-bacd5e47-f895-4a6f-8b13-e8c02b5c6a71.netlify.app/demo/?src=https%3A%2F%2Fstream.sob.m-dn.net%2Flive%2Fsb1-ll%2Findex.m3u8&demoConfig=eyJlbmFibGVTdHJlYW1pbmciOnRydWUsImF1dG9SZWNvdmVyRXJyb3IiOnRydWUsInN0b3BPblN0YWxsIjpmYWxzZSwiZHVtcGZNUDQiOmZhbHNlLCJsZXZlbENhcHBpbmciOi0xLCJsaW1pdE1ldHJpY3MiOi0xfQ==

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client on https://hls-js.netlify.com/demo and not just on my page

• The issue occurs in the latest client on https://hls-js-dev.netlify.com/demo and not just on my page

• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Use Chrome or Edge
2. Force any level (eg. 2)
3. Ensure video is playing, and at 2-3s from live head.
4. Apply a manual next level switch (eg. to 3) right after playhead passes the first parts of a segment (using timeline view – see image).
5. Observe that video plays remainder of current level buffer, and then freezes until the playhead reaches the next fragment (several seconds) – even though the new level data has been buffered.

Expected behavior

Smooth transition - like in Firefox and Safari.

Actual behavior

Video freezes for several seconds.

Console output

Partial media-internals log:

00:00:16.013 | duration | 79.8
-- | -- | --
00:00:16.697 | duration | 80.433333
00:00:18.018 | duration | 81.633333
00:00:19.012 | duration | 82.8
00:00:19.852 | duration | 83.499999
00:00:20.902 | info | "video decoder config changed midstream, new config: codec: h264, profile: h264 main, level: not available, alpha_mode: is_opaque, coded size: [640,360], visible rect: [0,0,640,360], natural size: [640,360], has extra data: false, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED}"
00:00:20.902 | kIsVideoDecryptingDemuxerStream | false
00:00:20.902 | kVideoDecoderName | "MojoVideoDecoder"
00:00:20.902 | kIsPlatformVideoDecoder | true
00:00:20.902 | info | "Selected MojoVideoDecoder for video decoding, config: codec: h264, profile: h264 main, level: not available, alpha_mode: is_opaque, coded size: [640,360], visible rect: [0,0,640,360], natural size: [640,360], has extra data: false, encryption scheme: Unencrypted, rotation: 0°, flipped: 0, color space: {primaries:BT709, transfer:BT709, matrix:BT709, range:LIMITED}"
00:00:20.902 | debug | "Media append that overlapped current playback position may cause time gap in playing VIDEO stream because the next keyframe is 2933ms beyond last overlapped frame. Media may appear temporarily frozen."
00:00:20.930 | dimensions | "640x360"
00:00:20.930 | kResolution | "640x360"
00:00:20.902 | duration | 84.699999
00:00:20.930 | pipeline_buffering_state | {"for_suspended_start":false,"state":"BUFFERING_HAVE_ENOUGH"}
00:00:22.100 | duration | 85.899999
00:00:22.705 | duration | 86.433333
00:00:23.932 | duration | 87.633333

[kanongil]

Did a test on the viability of the first hack, and it appears that it can actually work, and might be the best solution.

The main complexity will be to detect when the new level has buffered the data for the current playhead. Any ideas?

[robwalch]

2. Add a "crappy MSE mode", which won't allow a level switch until the new level has a future independent part (relative to the current playhead) – possibly excluding forced / emergency switches.

This would make a lot of sense. Seeking is disruptive to both application logic and UX often noticeable to the user so maybe the first hack could be implemented as application logic using hls.js events rather than built in?

"crappy MSE mode" (which I don't thing is crappy if it prevents appends over the playback position) would be difficult (require a lot of changes) to implement for any independent part, but might not be so bad if only on upcoming segment boundaries (first part only) since the logic to switch on fragments is there, it's just not adapted to work at the LL-HLS edge with fragmentHint parts.

[adampfw]

@robwalch Any update about this one? #5111 has been closed due this issue and I can not see any updates since 2021.

[robwalch]

This comment sums it up #5111 (comment)

While the next release will include important fixes over v1.2.9, it will not prevent the player from switching at moments when future appends will be made behind the play head (currentTime).

That happens when the stream controller uses next auto level on idle tick without considering that the next independent part is not available ahead of currentTime. What we found was that this is much more likely to happen if your key-frame interval is not a multiple of part hold back. Smaller key frame intervals (or more independent parts) provide more opportunities to switch without having to write over the playhead.

The same thing can happen without low-latency and with VOD when doing a fast switch (setting 'nextLoadLevel').

[robwalch]

v1.4.0 has some improvements, which result in fewer of these events, but it can still happen when the player either does not or cannot switch down in time to avoid this kind of unreported stall.

Currently the player only recognizes stalls where currentTime stops advancing. Since there can inevitably be video appends that overlap with the playhead and result in this kind of 'frozen frame' stall, the player should attempt to reckon these late appends with the error/latency/abr controllers penalty processes. The available actions would be to:

1. switch down and penalize levels where this occurs by triggering an error (could be an existing stall error with a new property flagging it as a late append, or a new type of error)
2. increase target latency (already done in response to stalls, late appends just aren't treated as such if currentTime advances on polling - waiting event may or may not be fired). Note that increasing the target latency without stopping the playhead doesn't change the current latency.

[robwalch]

The second part is to look at switching timing for playlists with many parts per segment and fewer independent parts. Enforcing switching at the next independent part is not happening currently, forcing the player to pick independent parts that start before the playhead in some cases when a switch happens. The player should also warn if streams contain non-independent part sequences longer than PART-HOLDBACK as that would make smooth switching impossible over those part runs.

[robwalch]

Switching starts in stream-controller's idle tick here based on the abr-controller's pick regardless of what segment of part might come next:

hls.js/src/controller/stream-controller.ts

Lines 256 to 273 in 7782205

	// set next load level : this will trigger a playlist load if needed
	if (hls.loadLevel !== level && hls.manualLevel === -1) {
	this.log(`Adapting to level ${level} from level ${this.level}`);
	}
	this.level = hls.nextLoadLevel = level;
	const levelDetails = levelInfo.details;
	// if level info not retrieved yet, switch state and wait for level retrieval
	// if live playlist, ensure that new playlist has been refreshed to avoid loading/try to load
	// a useless and outdated fragment (that might even introduce load error if it is already out of the live playlist)
	if (
	!levelDetails ||
	this.state === State.WAITING_LEVEL ||
	(levelDetails.live && this.levelLastLoaded !== level)
	) {
	this.level = level;
	this.state = State.WAITING_LEVEL;
	return;

For Low-Latency, it might be a good idea to pick a part first, and only run the code above if the part is independent. The part would not be loaded if a switch is started. The assumption would be that the player hopes to find an aligned independent part in variant it is switching to. This would avoid switching until all non-independent parts are appended in the current selection.

There could still be some delay in fetching the new playlist if we wait until the last part of the old playlist is loaded before starting to load it. By the time we receive an update in the current level with an independent part at the end, ideally the player would have also completed the same update in the playlist we are trying to switch to, but there is no such mechanism to refresh multiple variants prior to switch yet.