[Question] Using motion generators along with RL pipeline for moving two different robots

[samueledelloca]

Hi everyone. I would like to achieve the following: managing and moving two different articulations in the same environment for completing a common task.

I have a scene with a UR5e an a humanoid, which should be working together for completing a cooperative task (in this case the screwing of a plastic piece, but the aim of the task is not really important). Just to give you an idea, in reality the scene is similar to this one:

I'm able to separately move both entities using either RL, as explained in the third tutorial series (RL) or motion generators, like the DifferentialIKController explained in the fifth tutorial (controller).

But I'm not able to manage and move both entities together. I would like to move the humanoid using a controller (DifferentialIKController is fine) and to manage the UR5e with RL (by specifying command, actions, rewards, observation ecc.). For simplicity let's say that the humanoid has to behave in a predetermined manner (for example by moving its hands to specific locations) and the UR5e has to learn how to perform his task without interfering with the humanoid (for example by moving its end effector to other locations without colliding with the humanoid arms).

The problem is that, as shown in the tutorials, the RL pipeline is substantially different from the classic simulation one. In the first one I setup the environment and the learning agent (using RslRl and PPO), then I call the learn method and I have no more control over the simulation. Instead, with the classic simulation pipeline, I have control over resetting and stepping the simulation, managing buffers and writing data to the simulation.

I saw different examples, but none of them used motion generators along with RL in the way I need. I don't want to use a state machine as in standalone/environments/state_machine/lift_cube_sm.py because it prevents me from using RL; and neither I want to use DifferentialInverseKinematicsActionCfg because I don't want the RL to influence the humanoid's motion controller, and this would happen by specifying the DifferentialIKController inside an action as in orbit_tasks/manipulation/lift.

I tried to keep the RL pipeline and to use a command for the controller, but I didn't manage to make it work yet and moreover I'm not sure this is the best approach. Any suggestions or examples I can look at?

[zoctipus]

If I understood your setup correctly, It sounds to me that you need to write a custom RlTaskEnv, for the action input that step() function receives. You only keep the UR5e specific command so UR5e is controlled by rl. Then for the missing part of the humanoid Controller command, maybe add a custom logic that add back your predetermined value for humanoid. Then each part of input action goes to each robot's actionManager to produced the desired joint position/effort for the next frame.