Slow point cloud creation with D435

[nbubis]

Required Info
Camera Model	D435
Operating System & Version	Ubuntu 22.04
Kernel Version (Linux Only)	5.15.148-tegra
Platform	Jetson Orin
SDK Version	v2.54.2
Language	python

Issue Description

Creation of a point cloud from the depth image seems to take quite a long time on the D435. Following the code in the examples:

for pipeline_idx, pipeline in enumerate(self.pipelines):
    frame = pipeline.wait_for_frames()
    frame_list.append(frame)

for pipeline_idx, frame in enumerate(frame_list):
    depth_frame = frame.get_depth_frame()
    t0 = time.time()
    pointcloud = rs.pointcloud().calculate(depth_frame).get_vertices()
    pointcloud = np.asanyarray(pointcloud).view(np.float32).reshape(-1, 3)
    print("Pointcloud calculation time: ", time.time() - t0)

On the L515, the time printed (to calculate the pointcloud) is less than 2 msec. However, using the D435, the time is roughly 50(!) msec for creating the pointcloud. I'm using only the depth camera (color is turned off), with a 640p resolution.

How can I speed up the pointcloud processing time?

[MartyG-RealSense]

Hi @nbubis Point cloud generation can be a processing-intensive task because the calculations are performed on the computer's processing CPU rather than the camera hardware. On Jetson boards, the processing can be accelerated by installing the librealsense SDK with support for CUDA enabled, so that the processing work is offloaded from the CPU onto the Jetson's graphics GPU.

There are two approaches to enabling CUDA support in librealsense.

1. Install the SDK from dedicated Jetson packages.

Or:

2. Compile the SDK from source code with CMake and include the flag -DBUILD_WITH_CUDA=TRUE in the CMake build instruction.

Instructions for both a package installation and source code build on Jetson can be found at the link below.

https://github.com/intelRealSense/librealsense/blob/master/doc/installation_jetson.md

[nbubis]

@MartyG-RealSense - What I find curious is that the times for the L515 are less than a 25th of the time for the D435. Arguably, the 515 is taking in only 320p, but that should account for a x4 speed up, not x25! I'm also comparing to a hand rolled python loop on other cameras on the same machine, there is no reason for the calculation to take longer than 10 msec even at higher resolutions.

Should I be using rs.rs2_deproject_pixel_to_point instead?

[MartyG-RealSense]

The performance of the pc.calculate and rs2_deproject_pixel_to_point methods of pointcloud generation should be very similar, except that pc.calculate is a little more accurate when an aligned depth-to-color pointcloud is being generated.

The L515 lidar depth camera model's hardware can process 23 million depth points per second, so there may be an advantage there over the D435 stereo depth camera.

You could try the rs.rs2_deproject_pixel_to_point Python pointcloud script at #11744 to see how its speed compares to your pc.calculate script.