Mesh Weight Using Wrong Joint Id

[AMZN-Gene]

Apple's toy drummer asset does not load the correct mesh joint weights

Repro

1. Download Apple's toy drummer
   toy_drummer_idle.zip
2. Open it in Blender and select the drum. Note the mesh name: polySurface223
   
3. Notice "drum_joint" joint controls the drum (polySurface223)
   
4. Load toy_drummer_idle.usdz using TinyUSDZ using the provided code which,
   a. Finds and prints the joint id for the joint named "drum_joint": 8
   b. Finds and prints the "polySurface223" mesh skinned mesh weights

#include "tinyusdz.hh"
#include "str-util.hh"
#include "io-util.hh"
#include "tydra/scene-access.hh"
#include "tydra/render-data.hh"


int main(int argc, char** argv)
{
    const std::string filePath = "C:/Users/genewalt/Downloads/toy_drummer_idle.usdz";
    tinyusdz::USDLoadOptions options;
    tinyusdz::Stage stage;

    tinyusdz::tydra::RenderScene render_scene;
    tinyusdz::tydra::RenderSceneConverter converter;
    tinyusdz::tydra::RenderSceneConverterEnv env(stage);
    std::string usd_basedir = tinyusdz::io::GetBaseDir(filePath);
    env.set_search_paths({ usd_basedir });
    tinyusdz::USDZAsset usdz_asset;
    std::string warn, err;
    LoadUSDZFromFile(filePath, &stage, &warn, &err, options);
    tinyusdz::ReadUSDZAssetInfoFromFile(filePath, &usdz_asset, &warn, &err);

    tinyusdz::AssetResolutionResolver arr;
    tinyusdz::SetupUSDZAssetResolution(arr, &usdz_asset);
    converter.ConvertToRenderScene(env, &render_scene);


    // Iterate all the joints in the hierarchy to find the drummer bone
    int drumJointId = -1;
    std::vector<const tinyusdz::tydra::SkelNode*> skeletonNodes;
    skeletonNodes.push_back(&render_scene.skeletons[0].root_node);
    for (int i = 0; i < skeletonNodes.size(); ++i) {
        const tinyusdz::tydra::SkelNode* skeletonNode = skeletonNodes[i];
        if (skeletonNode->joint_name == "root/drum_joint") {
            printf("Drum joint name: %s, id %i\n", skeletonNode->joint_name.c_str(), skeletonNode->joint_id);
            drumJointId = skeletonNode->joint_id;
            break;
        }

        for (const auto& child : skeletonNodes[i]->children) {
            skeletonNodes.push_back(&child);
        }
    }

    for (const auto& mesh : render_scene.meshes) { 
        if (mesh.prim_name == "polySurface223") {
            const size_t indexCount = mesh.joint_and_weights.jointIndices.size();
            for (size_t i = 0; i < indexCount; ++i) {
                if (mesh.joint_and_weights.jointWeights[i] == 1.0f) {
                    if (drumJointId != mesh.joint_and_weights.jointIndices[i]) {
                        printf("Drum mesh is using the wrong joint id: %i\n", mesh.joint_and_weights.jointIndices[i]);
                    }
                    else {
                        printf("Drum mesh is using the correct joint id: %i\n", mesh.joint_and_weights.jointIndices[i]);
                    }
                }
            }

        }
    }

    return 0;
}

Expected Results
PolySurface223 mesh should be affected by drum_joint index 8

Drum mesh is using the wrong joint id: 3
Drum mesh is using the correct joint id: 3

Actual Results
PolySurface223 mesh is affected by the wrong joint index: 3

Drum joint name: root/drum_joint, id 8
Drum mesh is using the wrong joint id: 3

[syoyo]

FYI you are also better to use 'tydra_to_renderscene --dumpusd' and open Tydra processed USD in Blender for efficient debugging.

#148 (comment)

[AMZN-Gene]

Thanks for the pro-tip.

tydra_to_renderscene.exe C:/Users/genewalt/Downloads/toy_drummer_idle.usdz --dumpusd

The exported toy_drummer_idle.export.usda in Blender shows the drum vertex is indeed using the wrong joint "arm_lf_end_joint"

[syoyo]

Found multiple(duplicated) Skeleton Prims are exported with RenderScene conversion.

tydra_to_renderscene.exe toy_drummer_idle.usdz --dumpusd

This issue should also be fixed.

[AMZN-Gene]

Found multiple(duplicated) Skeleton Prims are exported with RenderScene conversion.

tydra_to_renderscene.exe toy_drummer_idle.usdz --dumpusd

This issue should also be fixed.

Ah yeah, RenderScene::skeletons.size() == 1 (correctly), but the exported usda ends up containing 1 skeleton per mesh.

[syoyo]

exported USDA contains multiple Skeleton Prims. You are encouraged investing where duplication happens(guess its in Tydra's USDA exporter)

And the fix and PR is also much appreciated.

[syoyo]

After some investigation, I found toy_drummer model uses Explicit Joint Orders(skel:joints in Mesh)

https://openusd.org/release/api/_usd_skel__schema_overview.html#UsdSkel_SchemaOverview_ExplicitJointOrders

so reordering joints if skel:joints exists would solve this issue
(currently Explicit Joint Orders is not supported in TinyUSDZ and Tydra)

[syoyo]

Explicit Joint Orders is now supported via #221

And I think now joint ids are now properly handled, so at least

Mesh Weight Using Wrong Joint Id

This issue should be now solved.
(Drum mesh's joint id should now be 8)

But, Blender(4.3.2) still can't import Skeletal rigs and animation of Tydra converted toy_drummer_idle model.

Situation is also the same when exporting USD from Blender(using Blender's built-in USD exporter) for toy_drummer_idle model, then re-import it. So there will be other factors we need to take into account for toy_drummer_idle model.

[syoyo]

Found that scales attribute must be set(authored) when importing USD in Blender.
In some situation(like toy_drummer), Tydra's USD exporter does not export scales SkelAnimation attribute.

[syoyo]

Tydra's USD exporter does not export scales SkelAnimation attribute.

This issue was fixed via 375a00e

Now USD file from tydra_to_renderscene toy_drummer_idle.usdz --dumpusd can be read correctly in Blender.

toy_drummer_fix.mp4