Support for realtime camera image? ROS Integration

[DavidYaonanZhu]

Hi,

Thanks for this amazing work.

I wonder, wether you provide support for tutorials about how to integrate with robot cameras in real-time.
Just like the example you showed by using the HSR robot.

I could also contribute to this matter.

[zubair-irshad]

Hi @DavidYaonanZhu, Please feel free to open up a PR as we don't have support for this currently. For HSR robot, apologies if we didn't mention this, but the demo was done offline i.e. first we collected rgb-d image from HSR robot and then ran our model on it. One could use ROS to stream rgb-d images and could run it online.

I found this other implementation which ran CenterSnap (our previous work) on an OAK-D and RealSense cameras. One thing worth noting here is that you might need model finetuning on cameras other than the one NOCS captured its real images from. We tried finding that camera but couldn't find it on the market. We made one minor adjustment, we warped the images from HSR robot slightly to make its intrinsics similar to the NOCS Real camera intrinsics before feeding it to ShAPO model for inference. Hope it helps!

[DavidYaonanZhu]

Thanks a lot for your answer and suggestion.

Let me try this.

[DavidYaonanZhu]

Hi @zubair-irshad

May I ask which intrinsic parameters from the HSR camera that you adjusted to make it looks like the NOCS Real camera?

[zubair-irshad]

I don't think those intrinsics matter so long as you warp the image coming out of your custom camera to match the image with NOCS real camera intrinsics. Here is a small gist I wrote that might help you. reproject function is important here and we feed the output RGB depth coming out of this function to our model in the demo results. I will update this with FAQ in the Reamde for now. Hope it helps and feel free to open up a PR if this works or you find an elegant solution. The best solution as I wrote above is to finetune the model on custom data but this works too.

[DavidYaonanZhu]

@zubair-irshad

Ok, thanks!
Let me try this out.