How to get started with new images?

[Codeguyross]

Hello,

If I wanted to start with fresh set of images to use this code to generate a mesh.ply file, what steps do I need to do to get the new images ready? I'm new to this area of study so any help getting going is appreciated!!

Thanks!

[plutoyuxie]

Hi, the pairs of color images and depth images should be aligned.
You need to caculate the camera pose for each frame. ICP may be helpful.

[YJonmo]

Hi, the pairs of color images and depth images should be aligned.
You need to caculate the camera pose for each frame. ICP may be helpful.

I used Kinect Fusion to align the depth images and got the camera pose form there. But when I use TSDF with the pose and depth it just doesn't work.

[Viky397]

Hi! I have my .pcd point clouds saved, and I use ICP to calculate the poses. However, running tsdf.py, the output mesh looks quite odd and elongated. Any tips would be welcome, thank you!

[YJonmo]

Maybe try to scale your pose info, i.e., divide it by a 2 or 3 ..

[Viky397]

Hello. I tried dividing by 2 or 3 but this just makes the output mesh look more sparse, and doesnt help with the oddly elongated output shape.

[YJonmo]

Did you divide the translation?
Only the translation should be divided or multiplied by a scaling coefdicient.
Keep in mind, the x axis is rightward, y is downward and z is outward.

[Viky397]

I used the following to get the transformation:

reg_p2p = o3d.registration.registration_icp(
            source, target, threshold, trans_init,
            o3d.registration.TransformationEstimationPointToPoint())

And then I save the following to a .txt file, and divide by a scaling factor:
reg_p2p.transformation/3

[YJonmo]

I guess the reg_p2p.transformation is a 4*4 transformation matrix?

You only need to divide the last column of this matrix by a coefficient. The last column is a the translation:
reg_p2p.transformation[0:3,-1] = reg_p2p.transformation[0:3,-1]/Coeff

In my case, I chose changes in the movement when it was no rotation. Then I focused on finding the correct dividing coefficient.

[Viky397]

Hm still seems very odd. This is the resulting mesh.ply.

[YJonmo]

It is hard to say what is going on just by looking at the image.
Also give some off set to your depth values. For example all the depth values should be added by some constant value. If you maximum depth is 255, then add 50 to the value of each pixel and see how it looks like. Play with the numbers.

[Ademord]

@Viky397 did you find a solution to this?

[Viky397]

@Ademord nope. I switched to Kimera-Semantics instead. Very easy repo to work with in ROS :)

[Ademord]

@Viky397 ahhh im working in unity so i need to do this by hand technically :(

[rayryeng]

I managed to get this working with my own custom dataset. I did some further research and some implementation tests, specifically on how the poses should be laid out. There are two coordinate frames that we have to keep in mind if we want to use our own custom data. This, of course, assumes that the camera frame and depth frame are aligned together.

Local Coordinate Frame

The dataset is from the RGB-D 7 Scenes Dataset from Microsoft - specifically, we are using the RedKitchen scene. This scene was captured using a KinectV1 sensor, with the x and y directions defined as increasing right and upwards, while the z direction is increasing by moving outwards. However, the documentation mentions that the depths are defined with -z, so the look direction is going into the screen, but the z axis is defined as moving outwards for positive values.

Judging from a quick glance at the source code, this is already taken into account, so we do not need to negate the depth values when providing our own depth maps. Just remember that they are unsigned 16-bit and each element is defined in millimetres.

World Coordinate Frame

The poses defined in each of the text files are the camera-to-world transforms, so when we transform the pixels into the local coordinate frame using the depth maps, we perform an additional transform to bring the pixels with respect to the world coordinate frame using the pose defined for the frame. The convention of the coordinate frame is using the OpenCV/COLMAP coordinate frame where x is increasing to the right, y is increasing downwards and z is increasing into the screen.

      z
     /
    +---x
    |
    |
    y

If you have poses defined in a different coordinate frame, make sure you post-process your poses before you write them to file. The format of the data in the data directory is enumerated such that for each frame, there is a triplet of data you need - the camera frame, the depth image and the pose of the camera image. The pose is a 4 x 4 rigid transform such that the top 3 x 3 elements define the rotation and last column, except the last row, define the translation vector:

[r11 r12 r13 t1]
[r21 r22 r23 t2]
[r31 r32 r33 t3]
[ 0   0   0   1]

Once you have the correct poses, you can run the demo. Just change the data directory accordingly. One final thing you need to take account of is when you're displaying the data. Meshlab and Open3D both use the OpenGL coordinate frame where the y direction increases upwards and the z direction increases outward:

   y
   |
   +---x
  /
 z

Therefore, when you are extracting the mesh and point cloud, you will need to edit the source to negate the y and z coordinates of the vertices for the point cloud and mesh, as well as the normals for the mesh:

Mesh Extraction

Inside the fusion.py file, search for the get_mesh method inside the TSDFVolume class and make the following change. First, add a use_opengl=False input parameter that will toggle whether you want to output the 3D coordinates for the mesh and normals in the OpenGL coordinate frame. After, add the necessary logic to negate the y and z coordinates of the vertices and normals should we desire it.

    # Marching cubes
    verts, faces, norms, vals = measure.marching_cubes(tsdf_vol, level=0, method='lewiner')
    verts_ind = np.round(verts).astype(int)
    verts = verts*self._voxel_size+self._vol_origin  # voxel grid coordinates to world coordinates

    if use_opengl:
      # Negate the y and z coordinates of the vertices
      verts[:, 1] = -verts[:, 1]
      verts[:, 2] = -verts[:, 2]

      # Negate the y and z components of the normals
      norms[:, 1] = -norms[:, 1]
      norms[:, 2] = -norms[:, 2]  

Point Cloud Extraction

Search for the get_point_cloud method in the TSDFVolume class and make the same use_opengl=False edit in the signature, followed by negating the y and z coordinates for the output vertices.

    # Marching cubes
    verts = measure.marching_cubes(tsdf_vol, level=0, method='lewiner')[0]
    verts_ind = np.round(verts).astype(int)
    verts = verts*self._voxel_size + self._vol_origin

    if use_opengl:
      # Negate the y and z coordinates of the vertices
      verts[:, 1] = -verts[:, 1]
      verts[:, 2] = -verts[:, 2]

After all of this, I was able to successfully use this to do TSDF fusion on my own dataset. Here's an output example viewed in MeshLab - it's a synthetic creation of an Arizona desert: