ML Scenario Speedup: Make UnpackIntoRgbPlanes and PackFromRgbPlanes public APIs

[georg-jung]

rel: #1955
rel: #1462

I think it would be great to make PackFromRgbPlanes and UnpackIntoRgbPlanes publicly available.

src/ImageSharp/PixelFormats/PixelOperations%7BTPixel%7D.cs

-   internal virtual void PackFromRgbPlanes(
+   public virtual void PackFromRgbPlanes(
        ReadOnlySpan<byte> redChannel,
        ReadOnlySpan<byte> greenChannel,
        ReadOnlySpan<byte> blueChannel,
        Span<TPixel> destination)
    {

    /// <summary>
    /// Bulk operation that unpacks pixels from <paramref name="source"/>
    /// into 3 seperate RGB channels. The destination must have a padding of 3.
    /// </summary>
    /// <param name="redChannel">A <see cref="ReadOnlySpan{T}"/> to the red values.</param>
    /// <param name="greenChannel">A <see cref="ReadOnlySpan{T}"/> to the green values.</param>
    /// <param name="blueChannel">A <see cref="ReadOnlySpan{T}"/> to the blue values.</param>
    /// <param name="source">A <see cref="Span{T}"/> to the destination pixels.</param>
-   internal virtual void UnpackIntoRgbPlanes(
+   public virtual void UnpackIntoRgbPlanes(
        Span<float> redChannel,
        Span<float> greenChannel,
        Span<float> blueChannel,
        ReadOnlySpan<TPixel> source)
    {

This way it would be easy to create efficient, vectorized code that transforms image data loaded using ImageSharp to arbitrary tensor formats for machine learning scenarios. The code to support this use case seems to already exist as far is I can tell, it simply is no public API but an internal one.

If I'm overlooking any well-suited public APIs please point me in the right direction :)

[antonfirsov]

I'm not against exposing this functionality but we need to get the API right. The main reason why this is internal is the asymmetry between the pack and unpack methods: pack consumes byte buffers while unpack will produce float buffers. In this form it feels like an implementation detail of our Jpeg codec rather than a well-designed generic solution that can be made public.

Doesn't this asymmetry limit you or force you to do pre/post conversions? Can you present some specific use-cases (best with sample code/pseudocode)?

[georg-jung]

Thanks for the timely reply! Actually I'm just interested in UnpackIntoRgbPlanes with Span<float> as I use ImageSharp for preprocessing images before feeding them into onnx-runtime for face detection. I just included PackFromRgbPlanes due to the symmetry you mentioned - sorry for the confusion. My use case is quite much like in #1955.

I started with the code linked there and added some mostly cosmetic tweaks.

/// <summary>
/// Efficiently converts images to an input tensor for batch processing.
/// Preprocesses for {r, g, b}: <c>r = (r * (1 / 255 * stddev.r)) - (mean.r / stddev.r)</c>.
/// Thus, if you need simple r = r, you could pass stddev = 1/255 and mean = 0.
/// </summary>
/// <param name="images">The images to convert.</param>
/// <param name="mean">The rgb mean values used for normalization.</param>
/// <param name="stddev">The rgb stddev values used for normalization.</param>
/// <param name="inputDimension">The size of the tensor that the OnnxRuntime model is expecting, e.g. [1, 3, 224, 224].</param>
/// <returns>A tensor that contains the converted batch of images.</returns>
public static DenseTensor<float> ImageToTensor(IReadOnlyCollection<Image<Rgb24>> images, float[] mean, float[] stddev, int[] inputDimension)
{
    // Calculate these outside the loop
    var normR = mean[0] / stddev[0];
    var normG = mean[1] / stddev[1];
    var normB = mean[2] / stddev[2];

    var stdNormR = 1 / (255f * stddev[0]);
    var stdNormG = 1 / (255f * stddev[1]);
    var stdNormB = 1 / (255f * stddev[2]);

    inputDimension[0] = images.Count;

    var input = new DenseTensor<float>(inputDimension);
    var imgCnt = 0;
    foreach (var image in images)
    {
        var imgStrides = input.Strides[0] * imgCnt;
        image.ProcessPixelRows(pixelAccessor =>
        {
            var strides = input.Strides;
            var inputSpan = input.Buffer.Span;
            for (var y = 0; y < image.Height; y++)
            {
                var index = imgStrides + (y * strides[2]);
                var rowSpan = pixelAccessor.GetRowSpan(y);

                // Faster indexing into the span
                var spanR = inputSpan.Slice(index, rowSpan.Length);
                index += strides[1];
                var spanG = inputSpan.Slice(index, rowSpan.Length);
                index += strides[1];
                var spanB = inputSpan.Slice(index, rowSpan.Length);

                // Now we can just directly loop through and copy the values directly from span to span.
                for (int x = 0; x < rowSpan.Length; x++)
                {
                    spanR[x] = (rowSpan[x].R * stdNormR) - normR;
                    spanG[x] = (rowSpan[x].G * stdNormG) - normG;
                    spanB[x] = (rowSpan[x].B * stdNormB) - normB;
                }
            }
        });
        imgCnt++;
    }

    return input;
}

The performance I get is 1-2 orders of magnitude better than

private static DenseTensor<float> Naive(Image<Rgb24> img)
{
    var ret = new DenseTensor<float>(new[] { 1, 3, img.Height, img.Width });

    var mean = new[] { 0.5f, 0.5f, 0.5f };
    img.ProcessPixelRows(accessor =>
    {
        for (var y = 0; y < accessor.Height; y++)
        {
            Span<Rgb24> pixelSpan = accessor.GetRowSpan(y);
            for (var x = 0; x < accessor.Width; x++)
            {
                ret[0, 0, y, x] = (pixelSpan[x].R / 255f) - mean[0];
                ret[0, 1, y, x] = (pixelSpan[x].G / 255f) - mean[1];
                ret[0, 2, y, x] = (pixelSpan[x].B / 255f) - mean[2];
            }
        }
    });

    return ret;
}

With this improvement the tensor loading is not too much of a blocker for me anymore, but during my research on that topic I noticed the preexisting API that seems to be even better by leveraging the intrinsics. I understand that the issues with the naive version arise on the DenseTensor indexer-side, not on ImageSharp's. Nevertheless the naive version is so straightforward to write that I could imagine many people writing code like this that leads to bad performance. It is also done this way in OnnxRuntime's official docs. Thus, I thought that an API that makes it easier to get right might be beneficial.

One further really useful step could be an option to provide mean and stdDev values for normalization, done by ImageSharp, vectorized and written by people who know what they are doing like you :) before writing to a Span<float>. I'd understand if you consider this out of scope, but it could be quite useful for ML scenarios. If it just writes to a Span, it also fixes #1955 without taking a dependency on any specific tensor type or machine learning framework or math library or whatnot.