Potential TSDF Value Error in Submap Merging During mergeSubmapAintoB

[KuangHaofei]

Dear Authors,

Thanks for your great work!

I am using your code on my own dataset. I have a problem with submap merging.

When merging two submaps in the LayerManipulator::mergeSubmapAintoB function, the resulting TSDF values in the target submap (Submap B) appear incorrect in my case, causing artifacts in the generated mesh. For example, after merging, certain parts of the mesh fly far from their expected position, as shown in the figure below, you can see the chair before and after merging (I also plot the wall as a reference).

Before merge: TSDF values in Submap A and Submap B appeared reasonable.

After merge: TSDF values in Submap B are unexpectedly shifted or far from their original positions.

I tried to debug, I found the problem is caused by the following:
https://github.com/ethz-asl/panoptic_mapping/blob/3926396d92f6e3255748ced61f5519c9b102570f/panoptic_mapping/src/map_management/layer_manipulator.cpp#L117C1-L123C8

Is there any way to solve the problem?

You can download my dataset if you want to reproduce the problem, the link is:
https://drive.google.com/file/d/1LhWlI_e8SGQSu7Nj_tdwiJuVS89dtbS-/view?usp=sharing

Please let me know if you need more information!
Thanks!

[KuangHaofei]

Hi,

There are some updates regarding the problem.

I tried visualizing all the blocks in Rviz, as shown below:

It seems that when block A merges into block B, the position of block A appears in the wrong location. It feels like there might be an issue with block allocation during the merging process.

I hope this information is helpful to you!

This issue currently affects most of my datasets, so I sincerely hope the author can help resolve this problem!

Thank you very much!

[Schmluk]

Hi @KuangHaofei

Thanks for reaching out and your interest!

Few things that might be happening are:

• Unfortunately, the merging functionality was never brought to complete generality, so there are a few assumptions in the current implementation.
• The merging should never move things around on its own, and expects the initial grids to be grid aligned and of equal parametrization. To get there there are functionalities to transform, interpolate, and otherwise modify grids. In the default implementation of Panoptic Mapping all grids are actually aligned by construction.
• It could be that the resolutions are different for your maps. Since fusion happens by index I would strange things expect to happen, such as submaps being scaled (in global coordinates, thus also moving around)

Hope this helps, let us know if you have more questions or you found a solution to your issue!

[KuangHaofei]

Hi @Schmluk

Thanks for your reply!

The problem is temporarily solved by using the same resolution for all submaps, like this:

small_instance_voxel_size: 0.02
medium_instance_voxel_size: 0.02
large_instance_voxel_size: 0.02

However, this solution loses the advantage of multi-resolution.
Since the two submaps being merged belong to the same category, their resolutions should already be identical. Therefore, this issue shouldn’t have occurred.

Anyway, I will debug based on this information and hopefully find a proper solution.

Thank you so much, and Merry Christmas!

[Schmluk]

Hi @KuangHaofei

Happy to hear this narrowed down your problem! Few further comments:

• Yes, submaps being merged should have the same resolution already in the standard setting, this might be good to check what is happening.
• You can fuse submaps of different resolutions by interpolating one submap to the other (i.e. for each voxel center of Submap_B you can find the field value at this position using trilinear interpolation from Submap_A). Note that 'upsampling' a discrete field may introduce some extra discretization inaccuracies, should not be major though in most cases.

Hope this helps and much success with your project!