KHR_meshopt_compression proposal

[zeux]

This PR proposes a new extension, KHR_meshopt_compression, which is a successor to the existing extension EXT_meshopt_compression.

Motivation

KHR_meshopt_compression provides general functionality to compress bufferViews. This compression is tailored to the common types of data seen in glTF buffers, but it's specified independently and is transparent to the rest of the loading process - implementations only need to decompress compressed bufferViews and the accessors behave as usual after that. The compression is designed to make it possible for optimized implementations to reach decompression throughput of multiple gigabytes per second on modern desktop hardware (through native or WebAssembly code), to ensure that the compression is not a bottleneck even when the transmission throughput is high.

As a result, it's possible to compress mesh vertex and index data (including morph targets), point clouds, animation keyframes, instancing transforms, Gaussian Splat data and other types of binary data. Compression can be lossless, taking advantage of preprocessing to optimally order and optionally pre-quantize the data, or lossy with the use of filters, which allows arbitrary degree of tradeoff between transmission size and quality for multiple types of common glTF data.

The compression process is versatile and different data processing tools may make different tradeoffs with how to preprocess the data before compression, whether to use lossy compression and of what kind, etc. - by comparison, the decompression process is straightforward and fast. This is by design, and means that it's comparatively easier to implement this extension in renderers compared to processing pipelines, which removes as many barriers to adoption as possible.

Compared to EXT_meshopt_compression, this extension uses an improved method of compressing attribute data, which delivers improved compression ratios at the same decompression speeds for a variety of different use cases, and incorporates special support for lossy color encoding which allows to reduce the size further for cases where color streams are a significant fraction of the asset (some 3D scanned meshes, point clouds, Gaussian splats). All existing use cases of EXT_meshopt_compression are supported well, and no significant performance compromises are being made - as such, all existing users of EXT_meshopt_compression should be able to upgrade (pending ecosystem adoption) to KHR_meshopt_compression.

Why a new extension?

EXT_meshopt_compression has been completed ~4 years ago; it serves as a good and versatile compression scheme that transparently supports geometry, animation, instancing data and allows to maintain maximum rendering efficiency and in-memory size while using additional compression during transfer, with decompression throughput in gigabytes/second on commodity hardware.

Since then, the underlying compression implementation for attributes in meshoptimizer has been revised to version 1 (from version 0 that's used in the EXT extension) for better compression; this is currently used outside of glTF ecosystem by some native and web applications. Additionally, some use cases like point clouds and 3D scans have emerged after the extension was initially standardized, that benefit from better color compression (which has been considered for EXT but not included since at the time it was focused more on "traditional" 3D geometry).

Because of the change in the underlying compression bitstream, the bitstream specification needs to be revised as implementations of EXT_meshopt_compression may not be able to decode the new format - thus, a new extension name is necessary.

What changed?

KHR_meshopt_compression uses the same JSON structure as EXT_ extension, keeps the same three filters and keeps two existing schemes for index compression as is. It upgrades attribute compression to use a more versatile encoding (v1), which supports enhanced bit specification for deltas and customizable per-channel delta modes that improve compression further for 16-bit as well as some types of 32-bit data. For compatibility, v0 encoding is still supported. It also adds one new filter, COLOR, which applies additional lossy transform to quantized RGBA color data to decorrelate input channels (similarly to OCTAHEDRAL encoding for vectors, this improves compression and provides more optionality wrt using variable number of bits, without any changes needed to the renderer as the filter unpacks data back into quantized normalized RGBA).

On typical geometry data, enhanced attribute compression provides approximately 10% geomean reduction in vertex data size; the gains obviously highly depend on the content. On point clouds, as well as some 3D scanned models that use vertex colors, the new color filter together with the new attribute compression results in ~15% geomean reduction, with even stronger (20-25%) gains when non-aligned bit counts are used (e.g. 6-bit or 10-bit colors).

Why KHR?

These improvements require a new extension, as they update the data format / bitstream as well as the JSON enum for color filter. While it's possible to specify this as a new EXT extension, like EXT_meshopt_compression2, it seemed better to promote to KHR:

• This matches existing compression formats, like Draco and Basis Universal, as well as some formats pending discussion like SPZ
• This makes the compression scheme, which has proven to be versatile and useful since it got standardized, more first class, with associated ecosystem benefits in the coming years (more support)

Since a lot of different parts of glTF ecosystem can be supported by meshopt compression, including core functionality and extensions like Gaussian splats or mesh instancing, specifying a KHR version provides a more comprehensive/coherent story wrt compression for glTF ecosystem.

The minimal "upgrade" path from EXT to KHR would involve just changing the extension name, as the original bitstream should be fully compatible with the new bitstream. The ideal upgrade path would involve re-encoding the original data (helpful if COLOR filter is useful), or at least losslessly re-encoding the attribute data from v0 to v1 - this doesn't require parsing any accessor data, and merely requires decompressing buffer views and re-compressing them with new encoding.

Implementations

Since this is a proposal that just got created, this extension obviously does not have implementations yet :) Having said that, because the JSON structure is exactly the same, except the addition of a COLOR filter, and most implementations of EXT_meshopt_compression use meshoptimizer library which supports the new additions (color filter was released in 0.25), I'd expect that existing implementations for EXT_ can be made compatible with KHR_ with minimal effort:

• For loaders that currently implement support for EXT_ (three.js, Babylon.JS), updating meshoptimizer module and tweaking the JSON parsing code to recognize KHR_ extension should be only a few lines of changes and should be sufficient (for reference, full three.js implementation of EXT variant);
• For data processors that currently implement support for EXT_ (gltfpack, glTF-Transform), updating meshoptimizer module and exposing a user option that would serialize KHR_ extension and encode attribute data using new attribute encoding, plus optionally support color filter, should be easy
• For any loader that wants to implement this without relying on meshoptimizer library for some reason, I've updated the reference decoder by following the updates made to this specification, so it should be comprehensive and while it's more changes than you'd need if you were just using the library, it's a manageable amount of extra complexity.

[lexaknyazev]

it seemed better to promote to KHR

KHR vs EXT prefix choice generally means different treatment of the extension's IP besides perceived "ecosystem support". In particular the following details should be clarified upfront (not a legal advice, though):

• Khronos would be the extension text copyright holder
• The extension should not include 3rd-party trademarks or their use should be explicitly allowed (is "meshopt" registered in some way?)
• Any technology needed by the extension may become included in the Khronos IP framework (with some caveats)

[zeux]

Understood. I'm not a lawyer, but none of these seem like blockers to me (pending closer review). meshopt is not a registered trademark.

[zeux]

Some updates!

1. There should not be any copyright/trademark/IP issues as far as I'm concerned.
2. meshoptimizer 0.25 released last week includes support for color filter and v1 vertex codec in JS module, as well as an update to the reference decoder that supports both as well.
3. While no tools exist that can produce files with this extension, and I intentionally deferred official support in gltfpack for it, I have an implementation of encoding that requires just a few small tweaks to the source code (to use the correct extension name), so it should be very easy to produce test assets if necessary
4. I took a stab at seeing what it takes to upgrade an implementation of EXT_meshopt_compression to KHR; since the heavy-lifting is done by meshoptimizer library, the actual change should only require a few lines of JSON parsing changes. See mrdoob/three.js@dev...zeux:three.js:khr-meshopt - the first commit there is just updating the meshopt_decoder to latest version.

It would be great to understand what the next steps could be here, as from my perspective all blockers have been resolved. Quite confident the actual implementations are going to be quick to finalize if there is agreement from Khronos in principle that the extension should be included. I'm obviously also happy to adjust the proposed text if corrections or clarifications are necessary.

[lexaknyazev]

The three.js update suggests that the decoder does not need to know whether the glTF asset uses the original EXT or the new KHR extension name. This implies that existing EXT (v0) files could be upgraded to KHR (v1) without re-encoding anything at all.

Since v0 support is not going anywhere (because the EXT extension has been ratified and tools are expected to accept such files indefinitely), I'd suggest allowing v0 in KHR as well (with a note about v1 benefits), given that the intention is to keep using the same extension name.

[zeux]

Since v0 support is not going anywhere (because the EXT extension has been ratified and tools are expected to accept such files indefinitely), I'd suggest allowing v0 in KHR as well (with a note about v1 benefits), given that the intention is to keep using the same extension name.

That sounds good to me. I originally specified KHR as just accepting v1 in order to have the smallest/simplest possible specification. But indeed, this restriction is not strictly necessary - v0 support is never going away, and the format version is encoded in the first byte of the stream so the meshoptimizer library can decode both. The reference decoder decodes both as well. I can change this, it would just require a more complicated bitstream description where specific additions are called out as only being there if version is 1.

[zeux]

Separately: Should the filter decoding be bit exact? As far as I can tell, there's parts of glTF specification that do not require bit exact decoding - for example, normalized values mentioned above use a / 2^N-1 formulas or equivalent, but hardware decoding is unlikely to use a full division, and is more likely to use a multiply-by-reciprocal - these will produce different values. As another example, anywhere where glTF specifies matrix math, in practice a lot of implementations will end up using fma to contract parts of the equations, again resulting in different behaviors. I haven't checked Basis specification but I will be very surprised if the transcoding to BCn blocks is specified to be bit exact as well, because some parts of it require a full block reencoding if memory serves?

update seems resolved per comments above, we want the specification to be precise even if real implementations are not always precise wrt floating point math.

Worth noting is that the filters defined here always imply some amount of lossiness on encoding, so minor differences should not be consequential in practice. I also don't think in practice right now there's specific implementation issues with portability of filter decoding, it's just that this is an area where I'm used to GPU API specifications allowing some amount of tolerance to permit multiple possible implementations, and not require IEEE-754 exactness in all places.

[zeux]

Updated the spec as suggested, including adding v0 back to the extension text. The bitstream reference now specifies cases where aspects of the encoding depend on the version (different tail encoding, channel modes for multi-byte deltas, control bits for more flexible bit packing including 1-bit deltas). With this change, any asset with EXT_meshopt_compression can be upgraded to KHR_meshopt_compression by simply changing the extension name - so KHR extension becomes a full superset of the EXT extension.

[zeux]

I've drafted optional support for this in gltfpack as well (making it trivial to produce test assets that use the full set of features, not just a renamed EXT -> KHR extension), linked above.

I don't really have any outstanding spec changes that are known to be necessary. Please let me know what the next steps here are; I believe what happened with the EXT version years ago is it was submitted as a draft, which then made it easier to contribute support to a few projects within the ecosystem. I can contribute support for this to cgltf, three.js and Babylon.JS - per notes above it's quite straightforward.

[zeux]

Here's a couple test assets converted with the PR above; they aren't necessarily super representative, but this makes it easy to test renderer support.

Asset	Size	Attribute size	Source
brainstem_ext.glb	370464	258176	glTF-Sample-Assets/BrainStem
brainstem_khr.glb	345856 (-6.6%)	233575 (-9.6%)	--
flower_ext.glb	2840840	2839091	KhronosGroup/glTF-Sample-Assets#31
flower_khr.glb	2579496 (-9.2%)	2577738 (-9.2%)	--

meshopt-khr-assets.zip

[zeux]

We now have a few outstanding draft implementation PRs (gltfpack zeux/meshoptimizer#966, glTF-Transform donmccurdy/glTF-Transform#1745, three.js mrdoob/three.js#32163). For some of them to get merged, it would be great if the status of this extension was more clear. As far as I know the extension text is in a good shape but of course I'd be happy to address any further feedback.

Can someone outline the next steps here? For example, can this PR get merged with the extension marked as a draft? Or can the list of extensions be updated to officially mark this as "Review Draft", linking to the PR? Or some other variant of this would be great. Essentially I'm looking for some signal along the lines of "this extension looks reasonable and we'll review this for the inclusion", so that dependent projects can feel safer in merging the implementation PRs with the assumption (but not a promise, of course) that the extension will be added in the future.