CUDA Error in plan_single_js with Streaming Client in Isaac Sim

[OneOneLiu]

Environment Info

1. cuRobo installation mode: docker isaac sim
2. python version: 3.10.14
3. Isaac Sim version (if using): 4.1.0
4. Omniverse streaming client: v103.1
5. GPU: 4x A6000 Ada

---

Issue Details

Hi, I am using cuRobo to generate motion plans for a UR5E robot in Isaac Sim, hosted on a remote server. While the plan_single method works perfectly in the example script, attempting to use plan_single_js for joint-space planning results in the following CUDA error when connected via the Omniverse Streaming Client:

RuntimeError: CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

After a careful investigation, I found that this issue occurs when connecting via the Omniverse Streaming Client. Running the code in normal (local GUI transferred by SSH) mode avoids the error but results in significantly lower FPS (~11 FPS in local GUI vs. ~50 FPS with streaming client).

Steps to Reproduce

1. Make the following changes to the motion_gen_reacher.py example:
   Change a. Start the simulationapp in headless mode by replacing line

   curobo/examples/isaac_sim/motion_gen_reacher.py

   Lines 93 to 99 in 2fbffc3

   	simulation_app = SimulationApp(
   	{
   	"headless": args.headless_mode is not None,
   	"width": "1920",
   	"height": "1080",
   	}
   	)

   with

simulation_app = SimulationApp(
    {
        "headless": args.headless_mode is not None,
        "width": "1920",
        "height": "1080",
        "hide_ui": False,  # add this line to enable ui
    }
)

# For enable streaming
from omni.isaac.core.utils.extensions import enable_extension
enable_extension("omni.kit.streamsdk.plugins-3.2.1")
enable_extension("omni.kit.livestream.core-3.2.0")
enable_extension("omni.kit.livestream.native")

Change b. Use plan_single_js method in the main loop, and use cu_js as both start and goal for testing. Replacing

curobo/examples/isaac_sim/motion_gen_reacher.py

Line 386 in 2fbffc3

result = motion_gen.plan_single(cu_js.unsqueeze(0), ik_goal, plan_config)

with:

result = motion_gen.plan_single_js(cu_js.unsqueeze(0), cu_js.unsqueeze(0), plan_config)

2. Run the example in headless mode:
   omni_python /{path_to_the_example_script}/motion_gen_reacher.py --headless_mode="native"

3. Start the simulation and move the target and you will get the error.

Questions and Suggestions

Is this a known bug?

The error appears specific to using the Omniverse Streaming Client. Could this be a compatibility issue between CUDA Graphs and the streaming setup?

Suggestions for Higher FPS in Local Mode:

How can I achieve similar FPS (~50 FPS) in the local GUI mode as I do with the streaming client?

Workarounds:

Are there recommended settings or updates for improving compatibility between CUDA Graphs and the Omniverse Streaming Client?

Any insights or suggestions would be greatly appreciated.