Unexpected render item behavior for 'casts shadows'

[csyshing]

Describe the bug

Hi,

We received a feature request from production to see if we can toggle "casts shadows" for USD geometries in the same way as native Maya geometries, since proxy render delegate internally creates MRenderItem, we were experimenting such possibility by setting MRenderItem::castsShadows() , it seems straightforward if we make the changes in two places in lib/mayaUsd/render/vp2RenderDelegate/mayaPrimCommon.cpp (see attached diff files for the full changes):

1. MayaUsdRPrim::_InitRenderItemCommon(): set MRenderItem::castsShadows() when initialize the render item;
2. MayaUsdRPrim::_SyncSharedData(): keep tracking of primvar dirtiness to toggle the "casts shadows" value for corresponding render item.

The "casts shadows" attribute is exposed as an additional primvar named "primvars:visibility:castsShadows" (see attached USD file CORRECT_casts_shadows_off_two_groups.usda for the full content), we created a directional light, turned on "all lights" and "shadow" on the viewport, everything seems working as expected:

However, when we started to test production scene, something looks fishy, for example the above example has two groups of planes, the left & right groups, if we have them separated in different USD files, toggling the primvar does not change anything, it remains as initial state ("castsShadows" == false):

The left group (see attached USD file INCORRECT_casts_shadows_off_left_group_only.usda) - looks incorrect:

The right group (see attached USD file INCORRECT_casts_shadows_off_right_group_only.usda) - looks incorrect:

By having further tests, it looks like if foreground plane is smaller than background plane (see attached USD file CORRECT_casts_shadows_off_fg_smaller_than_bg.usda) , it works as expected:

But if foreground plane is larger than background plane (see attached USD file INCORRECT_casts_shadows_off_fg_larger_than_bg.usda) , it DOES NOT work as expected:

When debugging, we accidently found something even confusing, if we leave MayaUsdRPrim::_InitRenderItemCommon() untouched (which expected to see inconsistent result at init) , keeps the changes in MayaUsdRPrim::_SyncSharedData() (which expected to see inconsistent result at the first time when toggling the primvar), and the further toggling works as expected. For example the previous INCORRECT_casts_shadows_off_fg_larger_than_bg USD file now behaves as expected:

We can't reproduce the issue if they were in native Maya geometries, so it looks like related to MayaUSD only.
What did we miss? I feel like there is a bug somewhere but not sure where it is.
I attached our changes but I could submit a draft PR if that's easier to review - let me know!

Steps to reproduce
Steps to reproduce the behavior:

1. Apply the attached file casts_shadows_changes.diff.txt to get the cast shadows reacts on toggling
2. Unzip attached example USD files
3. Load one of the example USD files in Maya
4. Turn on "Usa all lights" and "Shadows" on the viewport
5. Observe the expected / unexpected shadows behaviors

Expected behavior
MayaUSD could properly supports "casts shadows".

Attachments

• GIFs - see above inline images
• Example USD files:
  casts_shadows_off_example_files.zip
• Code changes in diff format:
  casts_shadows_changes.diff.txt

Specs (if applicable):

• OS & version : CentOS 7.8
• Compiler & version : gcc 9.3
• Maya version: tested against Maya-2023.2.x - 2023.3.3
• Maya USD commit SHA: release/v0.30.0 branch b3d4a06 + custom changes
• Pixar USD commit SHA: official v24.08 / v22.05

[wallworm]

Thanks for reporting. We will discuss this one internally and get back to you.

[pierrebai-adsk]

Note: I'm not the expert of VP2 delegate, but one possible issue I see with your change: in the _Init, you reuse the EnqueueCommit, but other values set in _Init are set directly without Enqueue. Maybe try initializing the render item directly instead of post-poning it with Enqueue?

[pierrebai-adsk]

I tried my suggested change and it worked for me. In _InitRenderItemCommon, I wrote:

bool isCastsShadows = true;
if (getCastsShadows(_delegate, SdfPath(_rprimId.asChar()), isCastsShadows)) {
        renderItem->castsShadows(isCastsShadows);
        renderItem->receivesShadows(isCastsShadows);
}