How to add wrist camera

[Yingdong-Hu]

Hi, I want to add a wrist-mounted camera for the OpenCabinetDrawer env
I create a config file for the camera

name: "wrist"
position: [0, 0, 1.0]
rotation: [0.9238795, 0, 0.3826834, 0]  # transforms3d.euler.euler2quat(-np.pi/3*k, np.pi/4, 0, 'rzyx')
_include: "default_camera.yml"

I try to mount the camera on right_panda_link8 (maybe other link8 is more appropriate) by adding the following code to _load_camera function in base_env.py:

links = self.agent.robot.get_links()
if cam_info['name'] == 'wrist':
    camera_mount_actor = links[14]

The left image is the world frame image and the right image is my wrist-mounted camera image.
It seems that the camera take a picture from behind the cabinet.
How can I adjust the parameter like position, rotation, and camera_mount_actor to get an appropriate wrist-mounted camera? I have tried a lot but can't get the desired camera.

[tongzhoumu]

Hi @Alxead ,

Can you provide the full script and config files? I can help debug.

[Yingdong-Hu]

wrist_camera_config.zip

The full script and config files are in the zip file.
I add a new config file wrist_camera.yml in ManiSkill/mani_skill/assets/config_files/components
I modify some paramters of default_camera.yml and open_cabinet_drawer.yml, I want to render 256 * 256 images.
As for base_env.py，I mainly modify _load_camera function and get_obs function to add a new mode: "wrist"
I use vis_demo.py to visualize expert policy by setting --obs-mode=wrist

[tongzhoumu]

Hi @Alxead ,

Please use the following camera parameters:

    - name: "wrist"
      position: [0, 0, 0.15]
      rotation: [ 0.70710678,  0.        , -0.70710678,  0.        ] # transforms3d.euler.euler2quat(0, -np.pi/2, 0, 'rxyz')
      _include: "components/default_camera.yml"

It will give you something like this: