HDR support

[da2x]

Hi, any chance you could add support for 10 and 12-bit color depth, please? Either switching automatically based on the source image or using a command argument.

I’ve experimented with it in link-u/cavif, and the files are only 1–2 % bigger and still way smaller than the 8-bit JPEG/WebP equivalent.

[kornelski]

Yeah, that's doable.

[gitoss]

Yeah, that's doable.

+1 for this request, 10bit shows better performance even for 8bit sources

Libavif does 10bit, but releases or binaries around don't include rav1e (yet) :-\

[Ralith]

HDR metadata support would go nicely with this!

[kornelski]

I don't understand how HDR metadata support would help generating correct HDR files.

This crate supports JPEG and PNG as input, and AFAIK neither of these are good for HDR. I can put HDR metadata, but how do you get HDR pixels in?

[Ralith]

My interest is for using the ravif library to encode HDR data I generate myself; I don't know if it's relevant to transcoding from LDR using the command line tool, though it might be convenient for testing. Perhaps there's some esoteric form of JPEG that encodes the same metadata that would be nice to preserve?

[Ralith]

Is that something you consider in-scope, @kornelski?

[kornelski]

Use-case for programmatically-generated data makes sense. ravif could definitely support it, even if the CLI tool can't.

I haven't had time to play enough with HDR images to know how to best handle them. In my experience—for SDR—the separation between pixel data and color profiles that alter the meaning of the data is inelegant and has lead to a lot of broken software. Is is possible to make a better API for HDR data?

[Ralith]

HDR metadata is standardized and absolutely necessary, because it informs the ultimate display device of how to most effectively tonemap, avoiding unnecessary quality loss in common cases. I'm all in favor of designing an API to make it as difficult as possible to have missing/incorrect metadata, but it's not obvious to me what that would look like. Maybe require HDR metadata always be supplied if a HDR color space is used, for starters?

[kornelski]

10-bit encoding of sRGB has been released.

Encoding of HDR is still an open issue. What formats would you supply HDR images in? OpenEXR?

[Ralith]

That's again a non-issue for library users, which can just pass in raw uncompressed HDR10 bitmaps. Raw half-float bitmaps might also be useful.

[Walther]

Example usecase: computer graphics. I'm working on a rendering engine project, and all the internal processing is done in palette::Xyz<E,f32>. After the rendering is done, I want to export the result into an image.

For standard dynamic range output, I can use conversion methods to palette::Srgb<u8> and use image::ImageBuffer and its .write_to() method with ImageFormat::Png to create a .png, for instance.

For high dynamic range output, currently I can use palette::LinSrgb<Float> and ImageFormat::OpenExr in order to create a .exr file which will correctly show as HDR content in macOS Preview app, for instance. However, the support for .exr is limited, and won't be shown in e.g. web browsers. The same is true for the Radiance HDR format .hdr - able to create and view correctly, but limited in use.

This is where .avif would come in: easy to share HDR images, with widespread support¹.

The image crate has some barebones .avif support via the ravif crate in this repository, with ImageFormat::Avif. However, it only supports standard dynamic range:

The encoder currently requires all data to be RGBA8, it will be converted internally if necessary.²

I also attempted to use the ravif crate directly, with the encode_raw_planes_10_bit method.

Code sample

    let encoder = ravif::Encoder::new()
        .with_speed(3)
        .with_quality(100.0)
        .with_depth(Some(10));

    let planes: Vec<[u16; 3]> = pixelbuffer
        .iter()
        .map(|&color| {
            let color: palette::Srgb<Float> = color.adapt_into();
            let color: palette::Srgb<u16> = color.into_format();
            // divide by 64 to get 65535 -> 1023
            // avif encoder is using GBR color order for some reason?
            [color.green / 64, color.blue / 64, color.red / 64]
        })
        .collect();

    let alpha = None::<[_; 0]>;
    // NOTE: not re-exported in `ravif`, need to add dependency manually for this enum
    let color_pixel_range = rav1e::color::PixelRange::Full;
    let matrix_coefficients = ravif::MatrixCoefficients::Identity;

    let res = encoder
        .encode_raw_planes_10_bit(
            *width as usize,
            *height as usize,
            planes,
            alpha,
            color_pixel_range,
            matrix_coefficients,
        )
        .or(Err("Unable to encode image"))?;

    Ok(std::fs::write(target, res.avif_file).or(Err("Unable to write file"))?)

While I am able to create an .avif file this way, it appears to only show as SDR. The resulting image is visually very similar to the .png I create - limited dynamic range, easily blown-out highlights, and no visible change in brightness response on my monitor (which happens with the .exr and .hdr. EDIT: and with a downloaded avif sample).

EDIT: and as always, thank you for your work on this useful project! 🧡

Footnotes

1. https://caniuse.com/avif ↩

2. https://docs.rs/image/latest/image/codecs/avif/struct.AvifEncoder.html#method.write_image ↩

[MarijnS95]

@Walther thanks for mentioning! We ran into the same trouble while reworking these crates to support "HDR formats" (#94, #96).

For us however such HDR images are shown completely blown out and grey in browsers. We're currently chasing a few leads in avif-serialize, as re-coding the header of .avif files via ffmpeg (while copying the encoded data verbatim with -c:v copy), or showing them in browsers using a <video> rather than <img>, shows the HDR .avif image in all its glory :)

I'll let you know / link this issue whenever we have a solution!

[zachlewis]

If I can offer some advice...

tl;dr -- handling EXRs properly is a lot of work. I think HDR source images should be supplied as 16-bit PNGs.

The key to using AVIF as a container for HDR is to only feed it PQ-encoded or HLG-encoded data -- i.e., "display-referred" values whose intent-preserving creative and technical transforms have already been "baked" into normalized values suitable for 10- and 12-bit integer encodings.

The "scene-referred" (radiometrically linear relative exposure) values that EXRs were designed to hold aren't really suitable for viewing "out of the box", without applying additional creative and technical transforms responsible for realizing the intended appearance on a given display device. In order to display EXR data, image viewer applications must either naively implement their own tone-mapping and gamut-mapping algorithms; or they delegate color management entirely to an external framework.

HDR PNGs, on the other hand, (ostensibly) contain a cICP metadata chunk conveying the transfer function (either PQ or HLG), the primaries (almost always P3-D65 or BT.2020), and the video full range flag (usually set to true). PNGs only encode RGB data, so the CICP third byte ("color matrix") is always set to 0. The fastest and easiest thing to do is to also write RGB data out; and then, down the line, if you guys wanted, you could work on implementing support for the various HDR luma-chroma sub-sampled encodings.

I hope that makes sense...!