Stall when playback reaches EOS of shortest track

[jpk-frameio]

What version of Hls.js are you using?

1.6.12

What browser (including version) are you using?

Firefox 142.0.1, Safari 20622.1.22.118.2, and others. (Logs in this ticket came from Firefox on Mac.)

What OS (including version) are you using?

MacOS 15.7 (24G222), and others.

Test stream

https://d1kt7uc00g7l93.cloudfront.net/hls-js-audio-eos-stall/video-180s-audio-5s/main.m3u8

Configuration

{
  "debug": true,
  "enableWorker": true,
  "lowLatencyMode": true,
  "backBufferLength": 90,
  "maxBufferLength": 120,
  "maxBufferSize": 512,
  "maxMaxBufferLength": 900,
  "maxFragLookUpTolerance": 0
}

Additional player setup steps

This test stream contains a video track 180s long, and an audio track 5s long, but this issue manifests with other track lengths and the generalized case is:

• An indefinite stall occurs when the shortest track is shorter than the longest track by a margin greater than the maxBufferLength, and playback reaches EOS on the shorter track.
• But the stall may be temporary when playback reaches the shorter track's EOS, if the longer track is permitted to buffer all the way to its EOS.

So for this test stream:

• With a maxBufferLength of 120, a stall occurs when playback reaches the end of the short audio track, and the player remains stalled indefinitely.
• With a maxBufferLength of 180, a stall may occur when playback reaches the end of the short audio track, but will resume once the long video track buffers to its EOS. The duration of the stall (or whether you get the stall at all) depends on how fast the client can buffer the long video track all the way to EOS. Throttling the client to ensure the video track can't buffer to its EOS quickly is useful to guarantee this condition.

Test stream:

• HLS URL
• Demo player URL

The source mp4 was generated with generate-video.sh:

#!/usr/bin/env bash

output_prefix="${1}"

video_duration="${2}"
audio_duration="${3}"

output_filename="${output_prefix}-video-${video_duration}s-audio-${audio_duration}s.mp4"

ffmpeg \
  -f lavfi \
  -i color=c=white:s=1920x1080:d=${video_duration} \
  -f lavfi \
  -i sine=frequency=440:sample_rate=44100:duration=${audio_duration} \
  -vf "drawtext=text='%{n}':x=100:y=100:fontsize=124:fontcolor=black" \
  -c:v libx264 \
  -g 60 \
  -t ${video_duration} \
  -pix_fmt yuv420p \
  -c:a aac \
  "${output_filename}"

$ ./generate-video.sh 180 5

And the HLS stream was packaged with mp4-to-hls.sh:

#!/usr/bin/env bash

input="${1}"
output_prefix="${2}"

mkdir -p "${output_prefix}"

ffmpeg \
    -i "${input}" \
    -map 0:v \
    -map 0:a \
    -f hls \
    -master_pl_name "main.m3u8" \
    -var_stream_map "a:0,name:audio,agroup:audio,default:yes v:0,name:video,agroup:audio" \
    -codec:v copy \
    -codec:a copy \
    -hls_list_size 0 \
    -segment_time 2 \
    -hls_time 2 \
    -start_number 1 \
    -hls_fmp4_init_filename "%v-000.m4s" \
    -hls_segment_filename "${output_prefix}/%v-%03d.m4s" \
    -hls_segment_type fmp4 \
    -hls_playlist_type vod \
    "${output_prefix}/%v.m3u8"

Checklist

• The issue observed is not already reported by searching on Github under https://github.com/video-dev/hls.js/issues
• The issue occurs in the stable client (latest release) on https://hlsjs.video-dev.org/demo and not just on my page
• The issue occurs in the latest client (main branch) on https://hlsjs-dev.video-dev.org/demo and not just on my page
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream

Steps to reproduce

1. Construct a stream in which the shortest media track is shorter than the longest media track by a margin greater than the configured maxBufferLength. (In this case the test stream has a video duration of 180s, an audio duration of 5s, and the maxBufferLength is set to 120s.)
2. Play stream until playback reaches the short track's EOS. (In this case, playback must reach the 5s mark.)

Expected behaviour

The short track's EOS was reached, which means we know there will be no more content on that track, and stalling to wait for more doesn't make sense. So the player should not stall at the end of the short track in this case, and it should continue playing the remainder of the long track.

What seems to be happening, without looking at the code:

• When hls.js makes the determination to stall playback or not, it seems to consider the shortest buffer among all the media tracks it's handling, which generally makes sense.
• However, hls.js does not appear to consider that a short buffer could be due to EOS being reached on that track, as we see in this test stream.
• If the maxBufferLength is long enough to buffer the remaining duration in the video track, playback resumes because hls.js recognizes that all tracks have reached EOS.

We know that hls.js can play the remaining video after a short audio track ends, as it demonstrates when it's allowed to buffer all the way to EOS on the video track, but the stall logic doesn't seem to take that into consideration.

So the ideal fix for this (again, without looking at the code) would be for hls.js to take EOS into consideration when choosing which buffer length to use to determine if a stall should happen or not. In the case of this test stream, if the audio reaches EOS, only the video buffer length should be considered in the stall logic.

Edit: Per @robwalch's comment, HLS.js isn't actually what's making the determination to stall or not in this case, rather it's up to the browser's playback behavior. So the short track needs to be filled with media in order to prevent a stall. The ideal solution for mismatched track durations like this would be for HLS.js to do something like insert silence once an audio EOS was reached, or insert the last video frame if the video EOS was reached. In this particular case, padding the audio track with silence after the last audio segment would be sufficient, but a generalized solution for any track being short would probably be useful.

What actually happened?

The player stalls when playback reaches the short track's EOS (at 5s in this test stream). The long track continues to buffer until it reaches EOS or maxBufferLength.

• If the long track's buffer reaches maxBufferLength, the player remains stalled indefinitely.
• If the long track buffers all the way to EOS, the player resumes from stall and plays the remainder of the longer track.

Indefinite Stall, maxBufferLength of 120s:

In this case, the maxBufferLength is short enough that the remainder of the video track won't be buffered after playback reaches the end of the audio track, and the stall lasts forever.

(Full logs for this case in the "Console output" field, prefixed with !!.)

Config, note the maxBufferLength:

{
  "debug": true,
  "enableWorker": true,
  "lowLatencyMode": true,
  "backBufferLength": 90,
  "maxBufferLength": 120,
  "maxBufferSize": 512,
  "maxMaxBufferLength": 900,
  "maxFragLookUpTolerance": 0
}

Audio stream buffers to EOS:

15:41:11.778 [log] > [stream-controller]: Loading main sn: 4 of level 0 (frag:[7.200-9.600]) cc: 0 [1-75], target: 7.28 hls.js:9903:12
15:41:11.778 [log] > [stream-controller]: IDLE->FRAG_LOADING hls.js:10853:16
15:41:12.019 [log] > [audio-stream-controller]: Loaded audio sn: 3 of track 0 hls.js:9590:18
15:41:12.026 [log] > [audio-stream-controller]: FRAG_LOADING->PARSING hls.js:10853:16
15:41:12.026 [log] > [buffer-controller]: queuing "audio" append sn: 3 of track 0 cc: 0 hls.js:19712:12
15:41:12.031 [log] > [transmuxer.ts]: Flushed audio sn: 3 of track 0 hls.js:17089:46
15:41:12.039 [log] > [audio-stream-controller]: PARSING->PARSED hls.js:10853:16
15:41:12.039 [log] > [audio-stream-controller]: Parsed audio sn: 3 of track 0 (frag:[4.017-5.023]) hls.js:10760:12
15:41:12.039 [log] > [audio-stream-controller]: Buffered audio sn: 3 of track 0 (frag:[4.017-5.023] > buffer:[0.056-5.079]) hls.js:9764:12
15:41:12.039 [log] > [audio-stream-controller]: PARSED->IDLE hls.js:10853:16
15:41:12.039 [log] > [buffer-controller]: audio buffer reached EOS hls.js:19807:23
15:41:12.039 [log] > [audio-stream-controller]: IDLE->ENDED hls.js:10853:16

Stall occurs when playback reaches end of audio buffer:

15:41:16.336 [warn] > [gap-controller]: Playback stalling at @5.094716 due to low buffer ({"len":0,"start":5.094716,"end":5.094716,"buffered":[{"start":0.08,"end":5.079229}],"bufferedIndex":-1}) hls.js:32812:14
15:41:16.336 [log] > [interstitials]: Primary player stall @5.094716 bufferedPos: 5.094716 hls.js:27077:18
15:41:16.336 Error event: 
Object { type: "mediaError", details: "bufferStalledError", fatal: false, error: Error, buffer: 0, bufferInfo: {…}, stalled: {…}, errorAction: {…} }
hls-demo.js:24817:14

Full mediaError object:

{
  "type": "mediaError",
  "details": "bufferStalledError",
  "fatal": false,
  "error": {},
  "buffer": 0,
  "bufferInfo": {
    "len": 0,
    "start": 5.094716,
    "end": 5.094716,
    "buffered": [
      {
        "start": 0.08,
        "end": 5.079229
      }
    ],
    "bufferedIndex": -1
  },
  "stalled": {
    "start": 966620
  },
  "errorAction": {
    "action": 0,
    "flags": 0
  }
}

Temporary stall, maxBufferLength set to 180s:

In this case, the maxBufferLength is long enough that the remainder of the video track can be buffered after playback reaches the end of the audio track, causing the video buffer to reach EOS. The player recognizes that all tracks have buffered to EOS, and the stall resumes allowing the remainder of the video track to play.

(Full logs for this case in the "Console output" field, prefixed with ##.)

Config, note the longer maxBufferLength:

{
  "debug": true,
  "enableWorker": true,
  "lowLatencyMode": true,
  "backBufferLength": 90,
  "maxBufferLength": 180,
  "maxBufferSize": 512,
  "maxMaxBufferLength": 900,
  "maxFragLookUpTolerance": 0
}

Stall occurs when playback reaches end of audio buffer:

15:53:03.152 [warn] > [gap-controller]: Playback stalling at @5.094716 due to low buffer ({"len":0,"start":5.094716,"end":5.094716,"buffered":[{"start":0.08,"end":5.079229}],"bufferedIndex":-1}) hls.js:32812:14
15:53:03.152 [log] > [interstitials]: Primary player stall @5.094716 bufferedPos: 5.094716 hls.js:27077:18
15:53:03.153 Error event: 
Object { type: "mediaError", details: "bufferStalledError", fatal: false, error: Error, buffer: 0, bufferInfo: {…}, stalled: {…}, errorAction: {…} }
hls-demo.js:24817:14

Full mediaError object:

{
  "type": "mediaError",
  "details": "bufferStalledError",
  "fatal": false,
  "error": {},
  "buffer": 0,
  "bufferInfo": {
    "len": 0,
    "start": 5.094716,
    "end": 5.094716,
    "buffered": [
      {
        "start": 0.08,
        "end": 5.079229
      }
    ],
    "bufferedIndex": -1
  },
  "stalled": {
    "start": 1673444
  },
  "errorAction": {
    "action": 0,
    "flags": 0
  }
}

Stall resumes as soon as player buffers all the way to the video EOS:

15:53:34.199 [log] > [stream-controller]: Loading main sn: 75 of level 0 (frag:[177.600-180.000]) cc: 0 [1-75], target: 177.68 hls.js:9903:12
15:53:34.199 [log] > [stream-controller]: IDLE->FRAG_LOADING hls.js:10853:16
15:53:34.573 [log] > [stream-controller]: Loaded main sn: 75 of level 0 hls.js:9590:18
15:53:34.575 [log] > [stream-controller]: FRAG_LOADING->PARSING hls.js:10853:16
15:53:34.575 [log] > [buffer-controller]: queuing "video" append sn: 75 of level 0 cc: 0 hls.js:19712:12
15:53:34.584 [log] > [transmuxer.ts]: Flushed main sn: 75 of level 0 hls.js:17089:46
15:53:34.596 [log] > [stream-controller]: PARSING->PARSED hls.js:10853:16
15:53:34.596 [log] > [stream-controller]: Parsed main sn: 75 of level 0 (frag:[177.600-180.000]) hls.js:10760:12
15:53:34.596 [log] > [stream-controller]: Buffered main sn: 75 of level 0 (frag:[177.600-180.000] > buffer:[0.080-180.080]) hls.js:9764:12
15:53:34.596 [log] > [stream-controller]: PARSED->IDLE hls.js:10853:16
15:53:34.596 [log] > [buffer-controller]: video buffer reached EOS hls.js:19807:23
15:53:34.596 [log] > [buffer-controller]: Queueing EOS hls.js:19820:16
15:53:34.596 [log] > [stream-controller]: IDLE->ENDED hls.js:10853:16
15:53:34.600 [log] > [buffer-controller]: Calling mediaSource.endOfStream() hls.js:19830:21
15:53:34.600 [log] > [buffer-controller]: Media source ended hls.js:19021:16
15:53:34.724 [log] > [gap-controller]: playback not stuck anymore @5.145381, after 31620ms hls.js:32689:16

Playback continues to end of video track:

15:56:29.653 [log] > [stream-controller]: setting startPosition to 0 because media ended hls.js:9382:15
15:56:29.654 [log] > [audio-stream-controller]: setting startPosition to 0 because media ended hls.js:9382:15
15:56:29.654 [log] > [subtitle-stream-controller]: setting startPosition to 0 because media ended hls.js:9382:15

Console output

Debug logs too long for this field.
Posted in this gist: https://gist.github.com/jpk-frameio/1cec493620b92f27e6cd4fb5fcc70c9f

Chrome media internals output

[robwalch]

It is not HLS.js's choice to to stall in the case described. This is browser playback behavior, independent of HLS.js.

Either do not call play() until buffering is complete, or fill the missing track with media.

[jpk-frameio]

Hi Rob, thanks for the quick response!

It is not HLS.js's choice to to stall in the case described. This is browser playback behavior, independent of HLS.js.

At least in Safari the native HLS playback (by just pasting the main.m3u8 URL from this stream into the address bar) plays the stream without a stall. The stall happens in Safari on the HLS.js demo player, though. I also observe that progressive mp4 playback of content with different track lengths like this works without stalls.

Either do not call play() until buffering is complete, or fill the missing track with media.

For context: We encountered this is on user-supplied files, many of which are artifacts of review and approval workflows. So the source content is often a WIP. We do some normalization on them during transcoding for delivery and playback, but track lengths are one of the things that are intentionally preserved. Padding the source media with silence to make the track durations the same is possible, but it would break other requirements.

While searching other issues prior to filing this one, my understanding is that HLS.js can insert silence in some cases? Would it be possible to do something similar on streams like these?

If the audio track must be padded with silence for playback to work, then doing so on the client and keeping the source media track durations as-is would probably be the best outcome.

[jpk-frameio]

Ah, I see there is hope!

@robwalch, if the idea of inserting silence in a case like this is attractive, I can edit the title/description here to reflect that since my premise of it being a stall logic thing was flawed.

[xinhong99]

@robwalch Is there an ETA for 1.7.0 release? Also can you provide an insight on JPK's question:

While searching other issues prior to filing this one, my understanding is that HLS.js can insert silence in some cases? Would it be possible to do something similar on streams like these?

Any instructions how we may be able to try that locally?