Sending ARGB Frames

[graphemecluster]

I am sorry to open an issue here but I am trying to send ARGB data (with shape (h,w,4)) to OBS but it seems that neither the shape is correct nor the OBS Virtual Camera support ARGB data. I've asked a question on StackOverflow but I think no one would be able to answer my question, so I decided to ask here. Any help would be much appreciated.

[letmaik]

Out of curiosity, where are you getting your ARGB frames from?
In any case, you can easily convert ARGB to RGB with frame[:,:,1:], or frame[:,:,:-1] for RGBA.

[graphemecluster]

I am trying to send image data with alpha data such that chroma key is not needed. I know how to blend an ARGB image with a background and output RGB. But I would like to send, something like a semi-transparent video, to OBS directly. In this case chroma key won’t work for intermediate opacity.
The frames are generated programatically.

[letmaik]

(Virtual) webcams generally don't support transparency, as far as I know. The native format of the OBS virtual camera on Windows is NV12 and UVYV on macOS, both of which don't have an alpha channel.

[graphemecluster]

Oh I see. I just think it’s possible. OBS has an option setting the format to ARGB for other applications’ virtual camera, though, and it does show up a camera capture with alpha channel when I use it.

[letmaik]

I'm not too knowledgeable on OBS itself, but the virtual camera device itself really doesn't support alpha.

[graphemecluster]

OK, I’ll find others for help or try another way then. Thank you for your replies!

[graphemecluster]

Good news! After many trials and errors I wrote my ARGB camera with the help of this library!

Here is my repository. You will need to manually install the camera by just double clicking pyvirtualcam/native_windows_argbcam/install/Install.bat. (The whole folder is copied from the above library.)

I hope if it helps!

[letmaik]

Looks great! I'll have a more in-depth look later on, maybe it could become an official backend in pyvirtualcam :)

[letmaik]

I had a look and tried it out as well. Good job! There are some minor issues that would have to be fixed. What concerns me a bit is that https://github.com/schellingb/UnityCapture doesn't seem to be maintained anymore. Given that it seems to work well, has a unique feature (alpha) and has no latency I think it's ok to include it with a disclaimer, something like "this is experimental".

Are you willing to work on this more and open a PR?

The following things would need some work:

• There is a pixel format confusion. UnityCapture expects RGBA, whereas you are converting to BGRA. Because of that the red-blue channels are switched. I also just noticed that all the conversion functions in image_formats.h containing argb in the name should really be named bgra because that's what they are in memory. libyuv::FOURCC_ARGB is really BGRA. So, in practical terms I think the new PixelFormat in the Python enum should be named RGBA. I think at a minimum RGB and RGBA should be supported in the backend, and libyuv has a function for rgb->rgba.
• I see that you flip the image vertically with np.flip before passing it to the backend. libyuv actually supports flipping by using a negative height argument. All the width and height arguments in image_formats.h are uint32_t, so they would have to be changed to int32_t to allow that. For flipping RGBA without conversion you could use ARGBCopy from libyuv with negative height and expose that as a new convenience function in image_formats.h as rgba_to_rgba.
• Currently the backend gets picked up even if the camera device is not installed. The same check as in the OBS backend should be done here, using the 64 bit CLSID: https://github.com/schellingb/UnityCapture/blob/fe461e8f6e1cd1e6a0dfa9891147c8e393a20a2c/Source/UnityCaptureFilter.cpp#L40
• The backend should be renamed to the name of the project, "unitycapture".
• The pre-built .dll files and install scripts should not become part of pyvirtualcam but rather the readme should point to the repo and contain short instructions on how to install it.
• device() needs to return the camera name as it is registered in Windows: Unity Video Capture. Similarly, like in the OBS backend, if a device name is given and doesn't match this string, then an exception should be thrown.
• I would move _sender->SendIsReady() to the constructor and throw an exception if it returns false. This is important as it may mean that another process is already sending to the camera, and to be consistent with the other backends this should throw an error instead of silently dropping frames.
• The backend should be registered after the OBS backend so that it is given priority if both are available.

[graphemecluster]

Thanks for the detailed advice!
My fork is originally for personal use, that’s why I register the backend before the OBS one.
I think that editing the UnityCaptureFilter.cpp and rebuilding the .dll files may be a good choice since 1. there are some flaws that I want to make better and 2. the name unitycapture is misleading as it does not require Unity at all.
However I think you will do better in handling this since I am only an amateur.

Currently other formats like YUYV are not converted to RGB at all and I am actually not too familiar with FourCC and the libyuv library, but I will try to do what you’ve mentioned. And should we add other formats containing an alpha channel too?

I would be glad if you could help me something.

[letmaik]

In general, I don't want to move virtual camera device implementations into pyvirtualcam since they are useful outside this Python package. Also, it becomes a maintenance burden.
Contributing a new backend is fine as long as the corresponding device implementation (UnityCapture in this case) is reasonably popular and established. If you want to make any improvements to UnityCapture then I recommend submitting a PR to that repo. In the README of pyvirtualcam I wouldn't link to unknown forks but rather just to the main project.
I understand that the name may be confusing but I don't think it's worth forking it just to change the name. This can be explained in the README.

Regarding formats, yes, it would be great if the other formats are supported as well, but I can help with that. I don't think we need multiple formats with alpha channels, RGBA is enough I think. Did you have a specific one in mind?

Once you're comfortable with your changes, feel free to submit a PR, even if it's not fully complete.

[letmaik]

Merged now: #56