Ideas for Gaussian Splatting Shader Plugin for Gazebo

[anshium]

This is related to the project titled, "Gaussian Splatting Shader Plugin for Gazebo". I have spent quite some time thinking about how to go about it, some nuances and some optimizations that can be done to make it render faster.

Here, I propose a method to implement this. I am very interested in working on this problem because I really like the idea of super-realistic simulations with robots. Most simulators I have worked with (PyBullet, Isaac Gym, Isaac Sim, MuJoCo, ManiSkill, and, of course, Gazebo!) have visually appealing scenes. However, achieving super-realistic simulations and directly mapping them to a real-world outdoor scene using Gaussian splats would be a really cool feature.

I would love to receive suggestions on this.

The Flow

I propose the following process flow to implement the Gaussian Splatting Shader Plugin. (plantuml link)

Explanation of the Process Flow

1. We can have the user load the SfM points (Structure from Motion, already generated using COLMAP, etc.) and take this point cloud as an input for initialization. Or, alternatively import pre-saved splats from a file.
2. If SfM is loaded, we would extract the 3D points and Camera poses from that data and initialize the Gaussian Splats.
3. After this, we will start the main process that would load the splat data, initialize OpenGL buffers and potentially put these on an available GPU. (I think if GPU is not available, we can either try on CPU or tell the user that the feature would only be possible to be used with a GPU setup - I am not too sure about which one would be a better option)
4. Then, for the duration of the simulation, as the camera moves, we would use the conventional 3D Gaussian Splatting technique which involves optimizing iteratively by
   • Projecting to Camera View and
   • Adaptive Density Control
5. Then finally using the Differentiable Tile Rasterizer to project it to the image plane, given the camera position + orientation.
6. Cleaning GPU buffers would be an important step to do after the simulation is quit.

Optimization Techniques

1. Level of Detail (LOD) - reduces far-away splats while keeping high detail for close splats.
2. Frustum Culling (removing splats out of view with the current camera and FOV)

Code Structure and Organisation

This is one of the initial, crude way of implementing the above. Changes might be needed as the project progresses. (plantuml link)

More points

1. Since this technique makes the rendered scenes directly editable (because we can just modify the splats), we can also perhaps include a feature to edit a scene mid-simulation (e.g., removing a bush that comes in the way of a ground robot). This would be a cool addition.
2. This would have to be coupled with a ROS2 package.

[arjo129]

Thanks for doing this detailed breakdown. My suggestion would be to implement the rendering in gazebo-sim itself as a gz-sim system. This would not be unlike the shader param system https://github.com/gazebosim/gz-sim/tree/gz-sim9/src/systems/shader_param . While L.O.D is important, we need to first have a working pipeline that can display splats in gazebo itself. Its important that your project proposal is broken down into small manageable chunks.

[fortmeier]

Is this still active? by chance I currently write a cross-plattform gaussian splatter in Vulkan (https://github.com/fortmeier/klartraum) and would like to add it as a gazebo rendering backend.

[arjo129]

This is not being worked on right now. But we would be happy to see integration between your shader and our ecm.

If you'd like we can have a chat.

[fortmeier]

This is not being worked on right now. But we would be happy to see integration between your shader and our ecm.

If you'd like we can have a chat.

Sounds interesting, drop me a message at LinkedIn (https://de.linkedin.com/in/dirkfortmeier) or if you prefer, I contact you via email.