Server struggles with remote streaming of compressed video/audio

[chocolatkey]

This isn't quite a bug, but could be in some cases considered as such - the server included with the cli struggles to stream video and audio files that are compressed inside of a ZIP-based container, such as EPUB. This was encountered for example with one of the IDPF sample publications: https://github.com/IDPF/epub3-samples/releases/download/20230704/cc-shared-culture.epub
This publication contains compressed videos:

When an asset is compressed using DEFLATE, it becomes impossible to seek to a specific position in the file without first decompressing all content before that offset (in practice, basically the entire file due to ZIP APIs in Golang). Modern browsers' native video players attempt to use byte ranges to buffer a portion of the video (or audio), so it can start playing and save data for the client. However because the video file is compressed, every time a small range is read, the entire video file must be decompressed over and over again. And because the video file is large, it is not cached by our remote streaming caching logic, which currently defaults to 1MB as the max file size for cache eligibility. This leads to a very inefficient and resource-heavy process on the server-side, and a slow loading for the user:

Furthermore, the streaming of this large video seems to be triggered an additional bug in the server, where it is returning the wrong size of content, as logged by browsers: net::ERR_CONTENT_LENGTH_MISMATCH 206 (Partial Content).
And the publication server we have running on Google Cloud Run returns a 502 gateway timeout: https://publication-server.readium.org/aHR0cHM6Ly9naXRodWIuY29tL0lEUEYvZXB1YjMtc2FtcGxlcy9yZWxlYXNlcy9kb3dubG9hZC8yMDIzMDcwNC9jYy1zaGFyZWQtY3VsdHVyZS5lcHVi/EPUB/video/shared-culture.mp4

This is also definitely something we should include in our optimization of EPUBs, as compressing of any media content (image, video, audio) make streaming worse

[HadrienGardeur]

Thanks for raising that issue @chocolatkey, it's also something that we'll want to address in the future optimize command (by using store rather than deflate on all media resources).

The way I see it, there are two different (and maybe complementary ways) to look at this:

• should we disable support for byte ranges on such media?
• and can we cache something to optimize for this use case?

First of all, although browsers default to byte ranges for <audio> and <video> elements, they can't expect all servers to support them. Do we know how they behave in this case? Do they buffer the whole file?

When an asset is compressed using DEFLATE, it becomes impossible to seek to a specific position in the file without first decompressing all content before that offset (in practice, basically the entire file due to ZIP APIs in Golang).

Once we've done that, couldn't we cache this information in the manifest for further use though?