mjpeg vs. webRTC color range inconsistency

[VoeGalore]

This is noticeable especially when looking at CLI with black background.
In mjpeg stream, black background is not completely black, color picker indicates RGB(16,16,16) a very faint grey.
In webrtc(H264) stream, black background is perfect black, RGB(0,0,0)

Expected behavior
mjpeg and webrtc should have same color range, same white/black levels.

Screenshots

Desktop:

• OS: Windows 10
• Browser: Chrome, Firefox
• Version: 100.0.4896.127(Chrome), 99.0.1(FF)

PiKVM info:

• Raspberry Pi board version: Raspberry Pi 4 model B
• PiKVM platform: v3.3 HAT
• Video capture type: v3 HAT
• KVMD version: 3.90-1
• uStreamer version: 5.4-1
• Linux kernel: 5.15.34-1-rpi-ARCH

Additional context
white level is also slightly different, in webrtc stream full white and full black is #ffffff and #000000, while in mjpeg stream they are slightly washed.

This is also present in USB dongle/loop capture, but noticeable only in black levels (full white seems to be correct)

[arch1mede]

My 2 cents on this, that's the nature of compression, this is a troubleshooting appliance first and foremost, I'm not sure I can see the value in maintaining the right color for troubleshooting. If you plan on using this outside its intended use case (bringing up a broken system) then I think you are using the incorrect project for your use case.

[mdevaev]

It's actually related with limited colorspace of the source. H.264 decoder on the client side can handle it natively using a special flag in bitstream, but JPEG not. I couldn't fix it in a simple way, so let's postpone it to the future.

[mdevaev]

Hints:

• https://en.wikipedia.org/wiki/YCbCr
• https://www.audioholics.com/home-theater-calibration/hdmi-black-levels-xvycc-rgb
• raspberrypi/rpicam-apps@4bb16fe
• https://forums.raspberrypi.com/viewtopic.php?t=204694
• https://bugs.chromium.org/p/chromium/issues/detail?id=1116922

http://parallel.vub.ac.be/~johan/MpegStills/jpeg2yuv.c

/**
  Rescales the YUV values from the range 0..255 to the range 16..235 
  @param yp: buffer for Y plane of decoded JPEG 
  @param up: buffer for U plane of decoded JPEG 
  @param vp: buffer for V plane of decoded JPEG 
*/
static void rescale_color_vals(int width, int height, uint8_t *yp, uint8_t *up, uint8_t *vp) 
{
  int x,y;
  for (y = 0; y < height; y++)
    for (x = 0; x < width; x++)
      yp[x+y*width] = (float)(yp[x+y*width])/255.0 * (235.0 - 16.0) + 16.0;

  for (y = 0; y < height/2; y++)
    for (x = 0; x < width/2; x++)
      {
	up[x+y*width/2] = (float)(up[x+y*width/2])/255.0 * (240.0 - 16.0) + 16.0;
	vp[x+y*width/2] = (float)(vp[x+y*width/2])/255.0 * (240.0 - 16.0) + 16.0;
      }
}

[mdevaev]

It looks like we have some kind of hell of specs

• The UVC spec seems to suggest that limited-range is correct for MJPEG.
• JPEG traditionally is encoded as full-range.
• Zoom, Safari, Firefox, Chrome all follow the UVC spec and decode MJPEG as limited-range.
• The ATEM Mini, my ThinkPad webcam, and the Logitech C922 all encode their MJPEG output as full-range.
• QuickTime player on Mac, and qv4l2 on Linux are the only programs I know that assume MJPEG is full-range from USB.
• Webcams don't have fantastic quality, so this range mismatch has been hidden for a long time. The ATEM Mini is one of the few high-quality devices that use MJPEG, so people are noticing.

So on the one hand, we have a lot of hardware that seems to ignore the spec, we have some software that also ignores the spec, but new important cross-platform software that follows the spec.

Next. I found a similar problem related to Pi Camera and Python library. The color range should indeed be specified for the input port: waveform80/picamera#512

uStreamer doesn't do this, but it doesn't matter since it doesn't supported by M2M encoder in the kernel: https://github.com/raspberrypi/linux/blob/rpi-5.15.y/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c#L900 (see setup_mmal_port_format()). However, this is done by libcamera-apps, which can be considered a reference implementation (also this). That is, I will have to patch the kernel to fix this.

In the case of CPU coding, we can correct the range very easily.

In the case of receiving an MJPEG stream from the camera or capture device, I think we should not fix anything, since this is not the uStreamer's concern.

In total, two out of three cases can be fixed. It remains only to decide how the color space will be set, but I will think about it.