Batched export and merge to USD

[jufrantz]

With this PR PrimUpdaterManager::mergeToUsd can now merge multiple pulled DAG paths to separate prims in a single invocation.

This is mostly interesting in the context of animated rigs, referenced with MayaReference prims.

The feature matches our previous Alembic "cache-and-swap" workflow that used AbcExport with multiple jobArgs flags, to export multiple rigs animation to separate files in one go.

---

Why it matters

Constraint‑safe

mergeToUsd removes the Maya data at end of the operation, swapping it to the USD data. For a MayaRefeference prim, the Maya reference gets unloaded.

In animation work, rigs are often connected to each others. Characters might be constrained to vehicles, props to characters, characters to characters, etc

Artists (and tools) must therefore cache the rig animations in an appropriate order (driven first then driver) so every constrained rig evaluates with their complete upstream graph, this can become a complex task. If the rig graph contains a cycle, which is possible at the rig-level, caching becomes impossible without another manual pre-baking.

With a "merge batch", all referenced rigs remain loaded until all USD data is authored, so the rigs remain connected to live Maya data during the whole evaluation. Artists can export the whole scene, or a group of related rigs, in one go. Order and cycles stop mattering.

Less waiting

Batching merge operations optimizes “Cache to USD” for animated rigs. In production, rig animation evaluation is the main cost of this caching. Currently, artists must export each referenced rig one‑by‑one; each export walks the same frame range again, making some rigs recompute identical results.

The new batched export walks the timeline once and shares evaluations across all rigs, eliminating this redundancy.

I tested performances with this scene four_animated_rhinos.ma and using this script test_merge_four_animated_rhinos.py. The scene is featuring 4 animated references to rhino_rig_001.ma, the rig file was found in maya-usd unit-tests.

	4 sequential mergeToUsd	Batched mergeToUsd	Speed‑up
DG	56 s	35 s	1.8×
EM parallel	42 s	25 s	1.7×

Details

UsdMaya_WriteJobBatch

New class, a collection of UsdMaya_WriteJob, that will execute all jobs, each writing to a separate stage, and write the timeSamples in a single unioned Maya timeline pass. jufrantz@aacaf4d

UsdMaya_WriteJob

Destination file name and append flag are now supplied in the constructor, instead of the Write method. This is to ease implementation of UsdMaya_WriteJobBatch. If this is a concern, I could rework this change. jufrantz@5e32d35

Internal refactors

• Shared write logic for both UsdMaya_WriteJob and UsdMaya_WriteJobBatch is centralized in a new private class UsdMaya_WriteJobImpl. jufrantz@aacaf4d
• UsdMaya_WriteJob::_BeginWriting was adapting the maya scene forrenderLayerMode, upAxis, and unit options.
• These tweaks need to remain for whole scene evaluation with multiple active (began) UsdMaya_WriteJob, so I refactored those aspects:
  • Maya scene tweaking is now done in UsdMaya_WriteJobBatch::WriteJobs and encloses multiple jobs evaluation.
  • USD authoring related to these options is still done by each UsdMaya_WriteJob.
  • cf jufrantz@5a06213, jufrantz@afb00e2

PrimUpdaterManager

• Added a new PushToUsdArgs struct that bundles all inputs common to push‑style operations (mergeToUsd, duplicateToUsd). jufrantz@aaa06a7
• PrimUpdaterManager::mergeToUsd now takes a std::vector<PushToUsdArgs>, so multiple DAGs can be exported in one call. jufrantz@3f68054
• Added a mergeToUsd python overload taking a sequence of tuples giving the dagPath and associated options dict. jufrantz@80d417d.

mayaUsd.lib.PrimUpdaterManager.mergeToUsd([
    ("MayaReference1", {"rn_layer":"cache1.usd", "rn_primName":"Cache1"}), 
    ("MayaReference2", {"rn_layer":"cache2.usd", "rn_primName":"Cache2"})])

• The original single‑object python signature remains for backward compatibility.
• The mayaUsdMergeToUsd command is also extended. It can export multiple DAGs with their exportOptions. jufrantz@d857156.

cmds.mayaUsdMergeToUsd(
    "Xform1", "Xform2",
    exportOptions=["animation=1;startTime=1;endTime=5", "animation=1;startTime=1;endTime=10"])

Tests

Added unit tests for batched merge, custom rig updaters, MayaReference caching, undo/redo integrity.
jufrantz@0bbf955, jufrantz@a537f29

I did not

• Add batched export support to the mayaUsdExport command. I imagine this could be added with a flag similar to AbcExport -jobArg / mayaUsdMergeToUsd -exportOptions, but I am unsure of the best syntax.
• Implement a batched duplicateToUsd; though the same technique would work.
• Expose the feature in UFE context menus (e.g., a multi‑selection “Cache To USD”). For now it’s meant mainly for studio pipeline tools.

I'd be happy to add any of these in a later PR if you think they are useful.

PS

This PR introduces a large set of changes, but I prefered to keep everything in one pull request to give you the full picture. However there are clearly 2 subsets of changes (UsdMaya_WriteJob + PrimUpdaterManager), I could split this PR in two if you prefer.

I've organized the changes into logical commits to make the review easier. Hope that helps.

Feedbacks are very welcome,
Julien

[pierrebai-adsk]

There was already UsdMayaUtil::ConvertMDistanceUnitToUsdGeomLinearUnit(), but that does raise a coding error when the units are not found. We could remove the coding error if that is why you did not use it.

[jufrantz]

Thank you for pointing out UsdMayaUtil::ConvertMDistanceUnitToUsdGeomLinearUnit(). I’ve replaced my local impl, I simply missed the existing one. c733a53

The coding error seems appropriate here: we only call it with the values returned by MDistance::uiUnit() and MDistance::internalUnit(), I believe an error would surface only if a valid enum value is not handled yet by the function.

[pierrebai-adsk]

Needs a few fixes, but otherwise great work.

[jufrantz]

Thank you for the review @pierrebai-adsk
I’m currently out of office but will update the PR ASAP.

[jufrantz]

I’ve addressed the requested changes and fixes.
The decimeter support is unchanged from my initial implementation, but @pierrebai-adsk , if you spot anything I’ve missed, please let me know. Otherwise this PR is ready to merge from my side.

[pierrebai-adsk]

The test testCacheToUsd failed:

11:45:45          self.assertRegexpMatches(self.stage.GetRootLayer().ExportToString(), 'payload = @.*testCacheToUsd/cache.usda')
11:45:45      AssertionError: Regex didn't match: 'payload = @.*testCacheToUsd/cache.usda' not found in '#sdf 1.4.32\n\ndef Xform "CacheParent" (\n    variants = {\n        string animation = "Cache"\n    }\n    prepend variantSets = "animation"\n)\n{\n    variantSet "animation" = {\n        "Cache" {\n            def Xform "cachePrimName" (\n                prepend payload = @W:\\build\\RelWithDebInfo\\test\\lib\\mayaUsd\\fileio\\testCacheToUsdOutput\\testCacheToUsd\\cache.usda@</cachePrimName>\n            )\n            {\n            }\n\n        }\n        "Rig" {\n            def MayaReference "Reference1"\n            {\n                bool mayaAutoEdit = 0\n                string mayaNamespace = "simpleSphere"\n                asset mayaReference = @simpleSphere.ma@\n            }\n\n        }\n    }\n}\n\n'

Something changed that makes the payload have backslashes instead of forward slashes. I don't know if you can easily identify what could have affected this in your changes? I might take a look this afternoon, but you might have an idea.

[jufrantz]

The test testCacheToUsd failed:

11:45:45          self.assertRegexpMatches(self.stage.GetRootLayer().ExportToString(), 'payload = @.*testCacheToUsd/cache.usda')
11:45:45      AssertionError: Regex didn't match: 'payload = @.*testCacheToUsd/cache.usda' not found in '#sdf 1.4.32\n\ndef Xform "CacheParent" (\n    variants = {\n        string animation = "Cache"\n    }\n    prepend variantSets = "animation"\n)\n{\n    variantSet "animation" = {\n        "Cache" {\n            def Xform "cachePrimName" (\n                prepend payload = @W:\\build\\RelWithDebInfo\\test\\lib\\mayaUsd\\fileio\\testCacheToUsdOutput\\testCacheToUsd\\cache.usda@</cachePrimName>\n            )\n            {\n            }\n\n        }\n        "Rig" {\n            def MayaReference "Reference1"\n            {\n                bool mayaAutoEdit = 0\n                string mayaNamespace = "simpleSphere"\n                asset mayaReference = @simpleSphere.ma@\n            }\n\n        }\n    }\n}\n\n'

Something changed that makes the payload have backslashes instead of forward slashes. I don't know if you can easily identify what could have affected this in your changes? I might take a look this afternoon, but you might have an idea.

Yes, I think I know why. In testCacheToUsd I added a temporary subfolder so we could export multiple cache files for the batched-export test cases. Previously the cache went to testCacheToUsd.usda; now it’s testCacheToUsd/<name>.usda. I wrongly use os.path.join to generate this payload path, it generates this wrong asset path on windows.
I look into fixing this now.

[jufrantz]

The test testCacheToUsd failed:

11:45:45          self.assertRegexpMatches(self.stage.GetRootLayer().ExportToString(), 'payload = @.*testCacheToUsd/cache.usda')
11:45:45      AssertionError: Regex didn't match: 'payload = @.*testCacheToUsd/cache.usda' not found in '#sdf 1.4.32\n\ndef Xform "CacheParent" (\n    variants = {\n        string animation = "Cache"\n    }\n    prepend variantSets = "animation"\n)\n{\n    variantSet "animation" = {\n        "Cache" {\n            def Xform "cachePrimName" (\n                prepend payload = @W:\\build\\RelWithDebInfo\\test\\lib\\mayaUsd\\fileio\\testCacheToUsdOutput\\testCacheToUsd\\cache.usda@</cachePrimName>\n            )\n            {\n            }\n\n        }\n        "Rig" {\n            def MayaReference "Reference1"\n            {\n                bool mayaAutoEdit = 0\n                string mayaNamespace = "simpleSphere"\n                asset mayaReference = @simpleSphere.ma@\n            }\n\n        }\n    }\n}\n\n'

Something changed that makes the payload have backslashes instead of forward slashes. I don't know if you can easily identify what could have affected this in your changes? I might take a look this afternoon, but you might have an idea.

Yes, I think I know why. In testCacheToUsd I added a temporary subfolder so we could export multiple cache files for the batched-export test cases. Previously the cache went to testCacheToUsd.usda; now it’s testCacheToUsd/<name>.usda. I wrongly use os.path.join to generate this payload path, it generates this wrong asset path on windows. I look into fixing this now.

I looked at it more closely. It seems payloads had backslashes on Windows even before these changes. What changed is that the regex only matched one component of the asset path, so the slash style didn’t matter. I would propose to change the test to assert on the actual cacheFile, like this:

@@ -317,7 +317,7 @@ class CacheToUsdTestCase(unittest.TestCase):
         if relativePath:
             if self.stage.GetRootLayer().anonymous:
                 self.assertNotIn('payload = @testCacheToUsd/cache.usda', self.stage.GetRootLayer().ExportToString())
-                self.assertRegexpMatches(self.stage.GetRootLayer().ExportToString(), 'payload = @.*testCacheToUsd/cache.usda')
+                self.assertIn('payload = @' + cacheFile, self.stage.GetRootLayer().ExportToString())
                 self.makeRootLayerNotAnonymous()
                 mayaUsd.lib.Util.updatePostponedRelativePaths(self.stage.GetRootLayer())

This passes on linux but I dont have a windows box available.
What to you think @pierrebai-adsk ? I can push this and we see what it gives ?

[jufrantz]

@pierrebai-adsk It looks like the CI failed due to a timeout, it isn’t a code error?

I took the opportunity to push two last-minute commits:

• Minor optimization in mayaUsdMergeToUsd when a single exportOptions is given for multiple objects (avoids redundant parsing). 5a8e921
• A unit test to verify exportOptions parsing with multiple objects (single vs per-object options). f3c9cd0

[pierrebai-adsk]

We will do some QA on it internally before approving

[jufrantz]

I’ve just rebased onto dev and resolved a conflict with testCacheToUsd.py (c7e1d3a).
@pierrebai-adsk let me know if there’s anything else needed from my side for approval.

[seando-adsk]

@jufrantz Sorry the delay. QA has internally tested the changes and given approval. Pierre is currently away so we are waiting for him to come back to finish the code review. Then we'll get this one merged.