Optimize VP2 rendering of USD objects with geometric cut-outs

[jufrantz]

Building upon #3947 , this PR leverages transparency cut-outs, achieved through UsdPreviewSurface.opacityThreshold>0 or glTF alpha_mode=MASK, to optimize instancing in vp2RenderDelegate.

In HdStorm, draw items using these materials are detected as "masked". They are rendered efficiently without depth sorting or OIT, and they still contribute to shadows, unlike translucent items.

VP2 sdk does not expose such feature. An MRenderItem can only be declared as "transparent" which implies alpha blending. These transparent items also receive special treatment in shadow and post-effects passes (e.g. SSAO). To achieve correct blended transparency, VP2 will automatically de-instantiate items with transparent shader. In cases of massive instancing, this severely impacts draw performance. For example, depth sorting thousands of individual objects is quite expensive, and consolidating all instanced geometries will produce very fat geometry data.

This PR specifically optimizes the case of instanced geometry with masked materials thanks to MPxSubsceneOverride::setAllowTransparentInstances. This flag allows us to preserve instancing even for transparent items. These items are consequently correctly treated in shadow and post-effects passes, beauty pass remains correct without sorting (without the huge cost of de-instantiation) since the material does not produce semi-transparent fragments.

These geometric cut-outs are typically used for vegetation foliage, which is important to instantiate massively in complex environments. Additionally, USD 23.02+ uses UsdPreviewSurface.opacityThreshold for cards drawMode , it is aimed at lightweight and fast rendering; instanced cards will also be optimized. These two cases are common in our productions, and we need them to be efficient within maya.

Changes

• Added "masked" transparency feature detection, inspired by HdStorm ( materialNetwork.cpp, materialXFilter.cpp )

• Added HdVP2Material._drawItemFlags: These flags indicate if the material is transparent and masked. They are computed during material sync, when parameters changed, and are retained for later access during rprim sync.

• Ensured affected renderables are re-synced, and MRenderItems are updated, whenever a material changes its drawItemFlags.

• Finally, set up MRenderItems during this rprim sync so GPU instancing is not broken, when possible, thanks to MPxSubsceneOverride::setAllowTransparentInstances.

Results

Tests were conducted in maya 2025.2 with this scene, featuring 100000 instances of a sphere that has textured opacity and a masked UsdPreviewSurface. The camera is framing the whole instancer. I profiled using this script:

# Avoid consolidating potentially un-instanced items
cmds.setAttr("hardwareRenderingGlobals.consolidateWorld", 0)
# Use Object Sorting transparency
cmds.setAttr("hardwareRenderingGlobals.transparencyAlgorithm", 1)

# Temporary turn on texturing and profile one VP2 refresh
cmds.modelEditor("modelPanel4", e=True, displayTextures=True)
cmds.refresh()
cmds.ogs(traceRenderPipeline=True)
cmds.profiler(reset=True)
cmds.profiler(sampling=True)
cmds.refresh()
cmds.profiler(sampling=False)
cmds.ogs(traceRenderPipeline=False)
cmds.modelEditor("modelPanel4", e=True, displayTextures=False)

Before optimization:

TraceRenderPipeline reports:

# Count for list: [View Cube] = 26
# Count for list: [HUD] = 2
# Count for list: [Pre-Opaque UI] = 2
# Count for list: [Transparent] = 100000

Profiler reports:

• VP2UpdateScene: 114ms
• VP2Draw3dBeautyPass: 1667ms

After optimization:

TraceRenderPipeline reports a single render item, instanced:

# -- VP2 Trace[modelPanel4][3d Beauty Pass]
# Count for list: [View Cube] = 26
# Count for list: [HUD] = 2
# Count for list: [Pre-Opaque UI] = 2
# Count for list: [Transparent] = 1

Profiler reports:

• VP2UpdateScene: 885us
• VP2Draw3dBeautyPass: 952us

[JGamache-autodesk]

Hi @jufrantz, looks very interesting, but I definitely need to ask... Have you tried making _isTransparent return false for the cut-out cases? That would remove any need for transparency mitigation at the render item level since these items would correctly declare their shading to be opaque.

[JGamache-autodesk]

Tried it locally. I only kept your new _IsMaskedTransparency function, and modified _isTransparent to do:

//! Return true if the surface shader needs to be rendered in a transparency pass.
 bool _IsTransparent(const HdMaterialNetwork& network)
 {
+    if (_IsMaskedTransparency(network)) {
+        return false;
+    }
+
     using OpaqueTestPair = std::pair<TfToken, float>;

And, for your scene, performance went from bad enough to cause a TDR to 30fps with correct visual results.

[jufrantz]

Hello @JGamache-autodesk,

That's definitely a good question especially since you are right, I just tested this and it works correctly on my scene, rendering is correct including shadows and SSAO (wow!).

To be myself transparent, this was my first intention and I tried it. I didnt test on this kind of scenes but on a simpler scene with just a non-instanced mesh and I detected issues with SSAO that was occluding the discarded surface. We are encountering these performance issues on scenes with instancing and I didn't want to introduce a regression with SSAO, I ended up with this solution and its complications.

I re-tested with this scene and still have the issue on non-instanced geometry, while it works on the same geometry and shader when point-instanced. Something is happening behind the scene but I have no clue.

[jufrantz]

Another test I did during #3947. It isn’t related to vp2RenderDelegate / MPxSubsceneOverride but might still be interesting, it led me to the idea that beauty fragment discards and marking the shader as opaque would not be enough for post effects.

On this scene with maya native meshs and shading, if usdPreviewSurface shading node sets transparencyOn output based on opacityThreshold input, like this:

@@ -479,6 +479,7 @@ MStatus PxrMayaUsdPreviewSurface::compute(const MPlug& plug, MDataBlock& dataBlo
         }
 
         float transparencyOn = false;
+        if (opacityThreshold <= 0.0) {
             if (opacityConnected) {
                 transparencyOn = true;
             } else {
@@ -489,6 +490,7 @@ MStatus PxrMayaUsdPreviewSurface::compute(const MPlug& plug, MDataBlock& dataBlo
                     transparencyOn = true;
                 }
             }
+        }

I traced that VP2 correctly classifies shaded items as opaque and I get this result with SAAO:

Versus the PR unchanged (all items marked transparent):

Hope it helps,
Julien

[JGamache-autodesk]

Thanks for all the tests, they helped me do a quick investigation in Maya rendering code.

Also interesting in your last scene is instancing the masked panels and seeing better SSAO results with GPU instancing enabled.

Can you update the PR with your latest code that properly declares masked as opaque in material.cpp and usdPreviewSurface.cpp? We will accept the SSAO issue this reveals and log a bug to eventually have this fixed in Maya.

[jufrantz]

Hi @JGamache-autodesk,

I have pushed the requested changes. All materials with masked transparency are now simply marked as opaque in this PR, and I’ve also updated usdPreviewSurface shading node in #3947.

Could you please provide us with the Maya issue ID once it’s logged so that we can follow up with the support team?

Thank you for your feedback.
Julien

[JGamache-autodesk]

The SSAO issue with cut-out shaders is logged as MAYA-135575.

[jufrantz]

Hello,
I fixed the formatting (by dropping an incorrect formatting commit). Sorry for this error; it appears something is wrong with my clang-formats.

[seando-adsk]

There was a single test failure on Windows 2022 with testUsdExportPackage - I've seen this test randomly fail. I'll just re-run the preflight to be sure.

[seando-adsk]

@jufrantz Just as an FYI, we are using clang format version 10 (which is quite old). It formats differently than newer versions which is why we haven't updated yet.

[jufrantz]

Hello @seando-adsk,
I finally found it in the workflow scripts and now have a container ready for my upcoming formats. Everything will be set for the next PR. Thanks for the info and for wrapping up the two PRs.