Platforms: not Working with LookingGlass plugin

[Mr-Anderson]

I have an obscure issue here that I think would just take a few lines changes to get fixed. I would be happy to test it and put in a PR myself. I'm up against the edge of my knowledge of Unity's command buffers though and I'm having trouble finding the commands I need.

I'm not sure if anyone here is familiar with LookingGlass volumetric displays. They are cool 3D displays that take up to 100 horizontal views of a scene and then display them using a lenticular display that can be viewed without glasses. You can get a feel for what they do from their Looking Glass Blocks website where you can view the images on a regular monitor. It is not as cool as the display in person though since the display has so many views the objects have a depth that you can't reproduce with anything else. It's even better than VR since each eye is getting multiple views leading to multiple focal planes within the scene. I highly recommend their portrait display if you are interested.

I thought it would be cool to mess around and get gaussian splats rendering to them. Looking Glass has a nice Unity plugin already that places a volumetric camera in the scene to render to the looking glass display.

This plugin seems like it is very close to working out of the box with the UnityGaussianSplatting project. Initially, it just renders a blank image to the looking glass display with traditional meshes rendering fine. From digging through your code and the Looking Glass code I discovered that if I change the UnityGaussianSplatting code to never call m_CommandBuffer.SetRenderTarget I can get the image intended for the GaussianSplatRenderer.Props.GaussianSplatRT to render to the display. This looks great spatialy, but the colors are washed out since it is skipping the compositing step.

You can see one of my renders from this trial here:
https://blocks.glass/mr_anderson/22601

I believe the issue is that the looking glass plugin is setting up its camera for multiview by changing the render target directly to the output image they are generating. The UnityGaussianSplatting plugin sets the render target back to BuiltinRenderTextureType.CameraTarget though. The relevant code is here in GaussianSplatRenderer.cs.

I have tried some hacks for setting the render targets back to all of the following render textures. None of these seem to be the correct target though. If I change the composite shader to just output a solid color I can only get that to render to the display if I remove all of the SetRenderTarget calls. I just get a black image if I set it to any of the following.

`RenderTexture.active
BuiltinRenderTextureType.CurrentActive
BuiltinRenderTextureType.CameraTarget
cam.activeTexture
cam.targetTexture

`

Because of this testing, I am fairly sure that I just need to find a way to store the current render target for the camera and then set it back to it before the composite step. I was hoping there would be some type of GetRenderTarget command that would allow me to store the current render target before rendering GaussianSplatRT and then move back to that target before compositing. I'm having trouble finding anything like that though. Any ideas would be appreciated.

I have a thread going for this on the Looking Glass discord as well:
https://discord.com/channels/573562186597662720/1164052492368224376

[Mr-Anderson]

I think this bug can be better generalized as the UnityGaussianSplatting plugin is not working correctly for cameras that have custom render targets set. I was wondering if this issue could be related to #17 . I have not messed with VR and UnityGaussianSplatting yet but had a hunch that this fix could be a solution for VR multi-view rendering as well. The VR cameras may be setting custom render targets that are then overwritten by the move back to BuiltinRenderTextureType.CameraTarget.

[aras-p]

I'm not sure if it's the same as VR issue. Maybe yes, maybe no. Hard to test myself since I have zero VR things around.

But yeah, the LookingGlass issue seems to be because presumably both LookingGlass and 3DGS want to render into some other render target "for a bit". The whole reason why I am rendering into a temporary render target, is to solve the "washed out colors" (since 3DGS are trained in gamma color space, whereas any normal rendering would use linear color space). So I need a temporary render texture, and composite that in the end by doing manual gamma-to-linear conversion.

And I can't quite find any good way of expressing "change to my render texture, and then change to a previous render texture" in a CommandBuffer API. So I change to the "what would camera be rendering into" render texture in the end, which apparently is not always the case.

[Mr-Anderson]

Yeah, I agree the link to the VR issue may be a long shot. Thanks for the explanation of the temporary texture in the render. This is my first dive into unity's command buffers and digging through your code has been a great learning experience.

I have been digging around the unity forums and stack exchange for a good way to store the existing render target. I did find a SetGlobalTexture which people have used to pass textures between command buffers. I tried it out but didn't have any luck. I think it may be making a global copy of the texture which wouldn't work in this case. That or I am just using it incorrectly.

https://docs.unity3d.com/ScriptReference/Shader.SetGlobalTexture.html
https://forum.unity.com/threads/a-way-to-reference-texture-across-commandbuffers.1117321/

I could also investigate ways to pass the texture from the Looking Glass code as a workaround. I was hoping to keep things generic though and it seemed strange that command buffers didn't have an easy way to store a texture reference.