Question about how RTAB-MAP iOS works

[takanotume24]

Hello. Thank you for your support.

I have a question about how RTAB-MAP iOS works. I have briefly read your paper. I also read the explanations in your forum.
All questions assume an iPad Pro with a LiDAR camera.

• Q1. When in ARKit Mode (LiDAR is disabled), what kind of information is being obtained from ARKit? My understanding is that in ARKit Mode, the Visual Odometry technology presented in section 3.1.1 of the paper is directly replaced by ARKit. Specifically, I think that in Figure 2, ARKit directly outputs the "Odometory" and "TF" on the right side of the figure. Is this correct?

• Q2. In LiDAR mode, I think that RGB-D images can be acquired, but in this case, are "Odometory" and "TF" directly output by ARKit?
  Or is only the "RGB-D Image" shown in Figure 2 replaced with the RGB-D Image from the iPad Pro's LiDAR camera, and the rest of the algorithm is from the paper?

Sorry if this is a strange question as I don't fully understand it yet. Thank you in advance.

[matlabbe]

Hello!

I think this will answer both questions: whether the LiDAR is enabled or not, ARKit's VIO is used as input odometry of rtabmap. In ROS world, that would correspond indeed to odometry topic and TF odom->base_link.

When LiDAR is used, a RGB-D image is fed to rtabmap (ARKit provides the LiDAR data as a depth image already registered to RGB camera). If LiDAR is not used, only RGB image with corresponding features tracked by ARKit are fed to rtabmap.

cheers,
Mathieu

[takanotume24]

Thank you very much! I understand now.

If LiDAR is not used, only RGB images with corresponding features tracked by ARKit are fed to rtabmap.

If LiDAR is not used, the 3D map built with RTAB-MAP iOS shows some point clouds. Where does this come from, does ARKit add depth information to "the RGB image with corresponding features tracked" even if LiDAR is not used?

[matlabbe]

ARKit provides the 3D position of the features tracked by VIO. We forward those 3D points as "point cloud" to rtabmap, which will create the sparse point cloud. The 3D points are also projected in RGB frame so that rtabmap can use the corresponding 2D features with depth for loop closure transformation estimation.