[TODO] Support gaussian splatting format(points data + extra)

[syoyo]

• usdPly?
• Points primitive with primvars
  • colors
  • normals
  • sh coeffs
  • scales/radius?
  • etc.

Goal

With TinyUSDZLoader WASM, we'd like to import/load gaussian splatting data in USD(Z) format and render it with three.js on the browsers.

References

mkkellogg/GaussianSplats3D#47
https://github.com/huggingface/gsplat.js
https://github.com/playcanvas/supersplat

[syoyo]

usdPly spec

Reference

https://github.com/adobe/USD-Fileformat-plugins/tree/main/ply

[syoyo]

Specification proposal

PrimType: GeomMesh(uses positions attribute only) or GeomPoints.
Use primvar:gsplat namespace for Gaussian Splatting attributes.
Half-type(fp16) is allowed(except for positions) to reduce the size of Point cloud data.

• float3 positions (x, y, z)
• half3/float3 primvar:gsplat:scales
• quath/quatf primvar:gsplat:rotations ( quaternion )
• half/float primvar:gsplat:alphas (value is in range [0.0, 1.0])
• half3/float3 primvar:gsplat:sh_l0
  • This value is the intensity of SH coefficient of band 0(DC component). Not actual RGB value. So the value could be negative.
  • 0.5 + SH_C0 * x to get base color value
• half3/float3 primvar:gsplat:sh_l1 SH band 1 coefficient( 3 * RGB )
• half3/float3 primvar:gsplat:sh_l2 SH band 2 coefficients( 5 * RGB )
• half3/float3 primvar:gsplat:sh_l3 SH band 3 coefficients( 7 * RGB )
• uniform int primvar:gsplat:shDegree(0, 1, 2, 3)

On the contrary to Splat PLY/SPZ format, scales and alphas are actual values(not before activation function applied: exp(x) for scales, sigmoid(a) for alphas in Splat PLY/SPZ format)

Discussion

Is it better to use elementSize attribute metadata to specify the number of SH coefficients like usdSkel?

# elementSize will be 3(band 0), 12(up to band 1), 27(up to band 2), 45(up to band 3)
float primvar:gsplat:sh = (.... ) { elementSize = 45 )

Compressed points

T.B.D.

We can reuse the idea from SPZ format: https://scaniverse.com/news/spz-gaussian-splat-open-source-file-format