Render 3D Gaussian Splats using Metal on Apple platforms (iOS/iPhone/iPad, macOS, and visionOS/Vision Pro)
This is a Swift/Metal library for rendering scenes captured via the techniques described in 3D Gaussian Splatting for Real-Time Radiance Field Rendering. It will let you load up a PLY and visualize it on iOS, macOS, and visionOS (using amplification for rendering in stereo on Vision Pro). Modules include
- MetalSplatter, the core library to render a frame
- PLYIO, for reading and writing binary or ASCII PLY files; this is standalone, feel free to use it if you just have a hankering to load up some PLY files for some reason.
- SplatIO, a thin layer on top of PLYIO to interpret these PLY files (as well as .splat files) as sets of splats
- SampleApp, a mini app to demonstrate how to use the above (based on Apple template code) -- don't expect much, it's intentionally minimal, just an illustration
- SampleBoxRenderer, a drop-in replacement for MetalSplatter for debugging integration, which just renders the cube from Apple Metal template
- Support spherical harmonics
- API documentation
- Explore possibilities for further performance optimizations. Mesh shaders? OIT? Who knows?
You're right, the documentation is entirely missing; it's a major TODO list item. In the meantime, feel free to try out the sample app -- just, like I said, don't expect much.
- Get yourself a gaussian splat PLY (or .splat) file. a. Capture one with Luma AI's iPhone app, and then export in "splat" format b. Or try the scene data from the original paper c. Go hardcore and learn to use Nerfstudio to train your own locally
- Clone the repo and open SampleApp/MetalSplatter_SampleApp.xcodeproj
- If you want to run on iOS/visionOS, select your target and set your development team and bundle ID in Signing & Capabilities. On macOS, just have at it.
- Make sure your scheme is set to Release mode. Loading large files in Debug is more than an order of magnitude slower.
- Run
- Note: framerate will be better if you run without the debugger attached (hit Stop in Xcode, and go run from the app from the Home screen)
There's a simple, official MetalSplatter model viewer app based on this library, available on visionOS for Vision Pro (support for iOS/macOS is coming later). It's also called MetalSplatter, go figure.
There are no external dependencies, but the math to render gaussian splats is straightforward (the basic representation has been around for decades), so there are a lot of great references around and I drew on a lot of 'em to try and understand how it works. There's really very little new here, the recent innovations are about training, not rendering. Nonetheless, I pretty much made every mistake possible while implementing it, and the existance of these three implementations was invaluable to help see what I was doing wrong:
- Kevin Kwok's WebGL implementation
- Aras Pranckevičius's Unity implementation and series of very insightful blog posts: 1, 2, 3
- The original paper's reference implementation