PipeWire support

[columbarius]

Summary of the current situation:

Small improvements for screencapture:

• Improve detection of invalid buffers: linux-pipewire: Reject invalid buffers #5614
• Handle buffer transformation on import (rotations and flips): PipeWire buffer transform #7762

Small bug fixes:

• Not crash on software rendering: linux-pipewire: Check format availablity against DRM only for dmabufs #7986

Big features:

• Uncouple the PipeWire code from screencast so we can reuse it more easily for other features like camera, maybe audio (contains lot of refactoring/cleanups) Draft: Pipewire unified wrappers #6669 (this blocks the following video related MRs)
• Virtual camera PipeWire Virtual Camera #5377
• Webcam/Device capture PipeWire-based Camera #5728
• Audio capture linux-pipewire: Add PipeWire audio captures #6207

Problems of the current situation:

The unified PipeWire wrapper collects a lot of cleanup and small fixes together with design decisions influencing the way forward for the other feature MRs. This slows down the feature MR since it is not certain on what state they should be based on. Additionally this catch all approach makes it annoying to review.

Ways forward:

• Move a lot of those cleanup commits into small MR's or a series, which can be merged easier and quicker, making the review of the wrapper simpler. Here the quicker is important to get the wrapper changes close to master again without having it depend now on a list of smaller commits.
• Don't base the new features (audio, device, vcam) on the wrapper idea, but duplicate the core PipeWire code and then unify it later. This unblocks those immediately.

Caveats

• All of those big video related MR's are currently in a not well defined/rough state. They should be rebased and I might have different versions of them depending on integration with the wrapper idea.
• The device capture has some jumps/frame drops. To debug them we probably need another client implementation to compare.

[kkartaltepe]

I'm happy to review a large collection of small fixup MRs (preferably as few PRs as makes sense, e.g. a bunch of one line fixup commits can be one PR but larger fixes might need their own), so we can merge those sooner. But I would assume Georges might prefer to review the feature/subsystem merge work.

[vkedwardli]

Hi, I'm not familiar with the PipeWire in Linux env, is it similar to Syphon (Mac) or Spout (Windows), which can route the video directly using GPU texture? (DMA-BUF?)

Seems that the Desktop Capture or Window Capture is using PipeWire, but is there a direct way to route the GPU texture into OBS using PipeWire?

[pallaswept]

I just left Windows on my home PC since I work with linux and prefer it and it's almost ready for the desktop. I'm familiar with how Windows works. In linux land the kernel shares the DMA buffers. All the details you need are in the link I shared with you and the link you just shared with me.

[vkedwardli]

I'm not trying to implement the relevant part on OBS, I know it is possible to implement sharing DMA-BUF in my program, I'm posting this question to ask if there is already a way to accept this shared buffer directly in the current implementation of OBS.

What confuses me is that, OBS seems to be support pipewire already, but only limited to Desktop/Window capture.
So I'm posting this question and hoping someone would clarify on:
"If it is possible to use the current OBS, or with the help of an extra plugin, so that I can display the shared buffer directly in OBS."

Thanks for sharing the qpwgraph and helvum, can I use these 2 programs to display a shared buffer in OBS?

[stephematician]

gpwgraph and helvum are mostly for audio filter graphs and won't help with displaying a shared buffer in OBS.

As far as I can tell, OBS does not have the capability to use pipewire's DMA-BUF API to capture and display video.

I believe OBS can create textures from DMA-BUFs on Linux ... but this might not be what you're after. If it is, see:

• https://blaztinn.gitlab.io/post/dmabuf-texture-sharing/
• https://docs.obsproject.com/reference-libobs-graphics-graphics#c.gs_texture_create_from_dmabuf

I think the process then would be to 1. write a program that sets up the capture and gets the source DMA-BUF, then 2. write a plugin/module for OBS that provides a source that can render the texture (presumably created from the DMA-BUF).

[columbarius]

Pipewire provides the sinks and sources, and presumably some audio routers could just do the routing for video too. But no one has built any such tools or defined any protocols to coordinate between video tools like that (ala Jack). And im not aware of anyone working on that.

I think there are thoughts that implementing camera capture can functionally serve as the protocol for general program to program sharing of video. Where programs just pretend they are "cameras". And there are people working on making cameras work on pipewire.

There are two kinds of pipewire video sinks that are interesting: context sinks (camera and screencast), which connect to PipeWire via the portal and request access to a specific node and a general sink, which will behave as a jack node, just for video and can be managed with qpwgraph. (Note: PipeWire can't mix, so only one source per input port and it won't do format conversions, this is currently worked on.)

[stephematician]

Thanks. I presume by "connect to PipeWire via the portal" you mean xdg-desktop-portal (i.e. https://docs.pipewire.org/page_portal.html)?

[stephematician]

I have some questions about the progress made on PipeWire support.

1. Do PRs Preparations for Portals + PipeWire camera, part 2 #9103 and Dimidium Totius (in preparation for PipeWire camera and audio, part 1) #8153 address the constraints for the virtual camera, webcam/device capture and audio capture that were mentioned above?
2. It isn't clear to me in this discussion if OBS supports pipewire audio capture already or not (I thought it did)?
3. It also isn't clear to me - is OBS using pipewire screencapture on both X11 and Wayland?
4. Are there branches/forks being used to develop pipewire video sources?

Thanks for any clarification provided.

[dimtpap]

1. Do PRs Preparations for Portals + PipeWire camera, part 2 #9103 and Dimidium Totius (in preparation for PipeWire camera and audio, part 1) #8153 address the constraints for the virtual camera, webcam/device capture and audio capture that were mentioned above?

Partially, yes. There will be at least one more part.

2. It isn't clear to me in this discussion if OBS supports pipewire audio capture already or not (I thought it did)?

PipeWire's compatibility layer, pipewire-pulse, makes OBS's PulseAudio stuff work on PipeWire.

[stephematician]

PipeWire's compatibility layer, pipewire-pulse, makes OBS's PulseAudio stuff work on PipeWire.

Oh, right, that makes sense. So it isn't using pipewire API directly, got it.

I noticed most branches on @dimtpap, @columbarius, and @GeorgesStavracas haven't been updated in a year or more; where is the most recent work on pipewire (in particular video capture) held?

[danir-de]

Thank you for your work! Is there an estimate, when MJPEG and H264 cameras will also be supported or whats holding that back?
The MR #9771 (Camera portal (feat. PipeWire)) currently only includes YUY2 (see discussion comment), but is still missing these essentials codecs.

Elgato cameras (like the Elgato Streamcam and Streamcam Pro, possibly also Elgato Capture cards) only support H264 and MJPEG.

[GeorgesStavracas]

A patch exists, but it stopped working after one of the various refactorings. It needs someone to sit down and figure it out. There's no time estimation for that to happen.

[dimtpap]

#11371