[BUG] IMU stops publishing in 2^32 microsecond intervals

[mrjogo]

Describe the bug

Regularly (but not 100% of the time...), when running the camera_node for long periods of time, the IMU topic (/camera_name/imu/data) will stop publishing for 2^32 microseconds (~4295 seconds, or 71.5 minutes) and then resume. The pause always starts a multiple of 2^32 us from the previous pause, and always lasts for 2^32 us. If left long enough, it will stop and resume multiple times.

I tried to isolate the cause to only certain parameters or sets of parameters, but I wasn't able to find a pattern, mostly because it sometimes doesn't freeze for a few hours (see below for parameters that do reliably fail). It SEEMS like a "good" parameter set never freezes, and a "bad" parameter set will always (eventually) freeze, but I didn't test enough and there were plenty of other uncontrolled variables on my system to be sure.

Here is the output from a monitoring node I created:

[camera_node-1] [INFO] [1757992702.086844391] [wrist_camera]: Camera ready!
[topic_stall_monitor.py-3] [INFO] [1757992702.252753213] [topic_stall_monitor]: Subscribed to /wrist_camera/imu/data [sensor_msgs/msg/Imu]
[topic_stall_monitor.py-3] [INFO] [1757992702.278115079] [topic_stall_monitor]: /wrist_camera/imu/data: learned expected period ~ 0.002s from 10 intervals
[topic_stall_monitor.py-3] [WARN] [1758005590.166395012] [topic_stall_monitor]: /wrist_camera/imu/data: no message for 5.437s (threshold 5.000s, expected period ~0.002s)
[topic_stall_monitor.py-3] [INFO] [1758009879.722961523] [topic_stall_monitor]: /wrist_camera/imu/data: resumed after gap 4294.995s (expected ~0.002s)

I did not check if any of the image or depth topics freeze as well. The fact that it's toggling at 2^32 intervals suggests some sort of timer wraparound or overflow that's not accounted for somewhere.

Running with DEPTHAI_DEBUG=1 doesn't seem to give any additional information, as the output looks the same, whether the node is frozen or not. Here's an example of the output right around when a freeze occurs (4294.967):

[camera_node-1] [19443010D18E772700] [1.3.3] [4294.941] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4294.941] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4294.971] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4294.971] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.004] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.004] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.041] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.041] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.071] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.071] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.104] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.104] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.141] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.141] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.171] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.171] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.204] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.204] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.241] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.241] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.271] [StereoDepth(4)] [debug] Post-processing filters: '1'
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.271] [StereoDepth(4)] [debug] Executing median+lut hardware stage
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.299] [system] [info] Memory Usage - DDR: 133.71 / 333.26 MiB, CMX: 2.38 / 2.50 MiB, LeonOS Heap: 38.15 / 81.70 MiB, LeonRT Heap: 4.42 / 39.87 MiB / NOC ddr: 1348 MB/s
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.299] [system] [info] Temperatures - Average: 58.46C, CSS: 59.36C, MSS 57.44C, UPA: 58.94C, DSS: 58.08C
[camera_node-1] [19443010D18E772700] [1.3.3] [4295.299] [system] [info] Cpu Usage - LeonOS 73.00%, LeonRT: 31.20%

Minimal Reproducible Example

Launch camera_node with the following config.yaml and leave running for a number of hours:

/**:
  ros__parameters:
    camera:
      i_pipeline_type: Depth
      i_publish_tf_from_calibration: true
      i_tf_base_frame: wrist_camera
    pipeline_gen:
      i_enable_imu: true
    stereo:
      i_align_depth: true
      i_board_socket_id: 2
      i_right_rect_publish_topic: true
      i_left_rect_publish_topic: true
      i_output_depth: true
    imu:
      i_enable_rotation: false
      i_enable_mag: false
      i_message_type: IMU
      i_gyro_freq: 400
      i_acc_freq: 400
      i_sync_method: LINEAR_INTERPOLATE_ACCEL

Expected behavior

Topic doesn't stop publishing.

Pipeline Graph

19443010D18E772700_pipeline.json

Attach system log

• Provide output of log_system_information.py: log_system_information.json
• Which OS/OS version are you using? Ubuntu 24.04
• Which ROS version are you using? 2
• Which ROS distribution are you using ? Jazzy
• Is depthai-ros built from source or installed from apt? apt
• Is depthai/depthai-core library installed from rosdep or manually? rosdep
• Please include versions of following packages - apt show ros-$ROS_DISTRO-depthai ros-$ROS_DISTRO-depthai-ros ros-$ROS_DISTRO-depthai-bridge ros-$ROS_DISTRO-depthai-ros-msgs ros-$ROS_DISTRO-depthai-ros-driver

Package: ros-jazzy-depthai
Version: 2.30.0-1noble.20250424.110737
Priority: optional
Section: misc
Maintainer: Adam Serafin <adam.serafin@luxonis.com>
Installed-Size: 68.2 MB
Depends: libc6 (>= 2.38), libgcc-s1 (>= 4.2), libopencv-core406t64 (>= 4.6.0+dfsg), libopencv-imgproc406t64 (>= 4.6.0+dfsg), libstdc++6 (>= 13.1), libopencv-dev, libusb-1.0-0-dev, nlohmann-json3-dev, ros-jazzy-ros-workspace
Homepage: https://www.luxonis.com/
Download-Size: 24.9 MB
APT-Manual-Installed: no
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: DepthAI core is a C++ library which comes with firmware and an API to interact with OAK Platform

Package: ros-jazzy-depthai-ros
Version: 2.11.2-1noble.20250814.103406
Priority: optional
Section: misc
Maintainer: Adam Serafin <adam.serafin@luxonis.com>
Installed-Size: 44.0 kB
Depends: ros-jazzy-depthai, ros-jazzy-depthai-bridge, ros-jazzy-depthai-descriptions, ros-jazzy-depthai-examples, ros-jazzy-depthai-filters, ros-jazzy-depthai-ros-driver, ros-jazzy-depthai-ros-msgs, ros-jazzy-ros-workspace
Download-Size: 7,200 B
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: The depthai-ros package

Package: ros-jazzy-depthai-bridge
Version: 2.11.2-1noble.20250814.084604
Priority: optional
Section: misc
Maintainer: Adam Serafin <adam.serafin@luxonis.com>
Installed-Size: 604 kB
Depends: libc6 (>= 2.38), libgcc-s1 (>= 3.3.1), libopencv-calib3d406t64 (>= 4.6.0+dfsg), libopencv-core406t64 (>= 4.6.0+dfsg), libopencv-imgcodecs406t64 (>= 4.6.0+dfsg), libopencv-imgproc406t64 (>= 4.6.0+dfsg), libstdc++6 (>= 13.1), libboost-dev, libopencv-dev, ros-jazzy-camera-info-manager, ros-jazzy-composition-interfaces, ros-jazzy-cv-bridge, ros-jazzy-depthai, ros-jazzy-depthai-ros-msgs, ros-jazzy-ffmpeg-image-transport-msgs, ros-jazzy-image-transport, ros-jazzy-rclcpp, ros-jazzy-robot-state-publisher, ros-jazzy-ros-environment, ros-jazzy-sensor-msgs, ros-jazzy-std-msgs, ros-jazzy-stereo-msgs, ros-jazzy-tf2, ros-jazzy-tf2-geometry-msgs, ros-jazzy-tf2-ros, ros-jazzy-vision-msgs, ros-jazzy-xacro, ros-jazzy-ros-workspace
Download-Size: 174 kB
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: The depthai_bridge package

Package: ros-jazzy-depthai-ros-msgs
Version: 2.11.2-1noble.20250814.080551
Priority: optional
Section: misc
Maintainer: Adam Serafin <adam.serafin@luxonis.com>
Installed-Size: 2,730 kB
Depends: libc6 (>= 2.14), libgcc-s1 (>= 3.3.1), libpython3.12t64 (>= 3.12.1), libstdc++6 (>= 11), ros-jazzy-fastcdr, ros-jazzy-builtin-interfaces, ros-jazzy-geometry-msgs, ros-jazzy-rclcpp, ros-jazzy-rosidl-default-generators, ros-jazzy-sensor-msgs, ros-jazzy-std-msgs, ros-jazzy-vision-msgs, ros-jazzy-ros-workspace
Download-Size: 204 kB
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: Package to keep interface independent of the driver

Package: ros-jazzy-depthai-ros-driver
Version: 2.11.2-1noble.20250814.101717
Priority: optional
Section: misc
Maintainer: Adam Serafin <adam.serafin@luxonis.com>
Installed-Size: 3,525 kB
Depends: libc6 (>= 2.38), libconsole-bridge1.0 (>= 1.0.1+dfsg2), libgcc-s1 (>= 3.3.1), libopencv-core406t64 (>= 4.6.0+dfsg), libopencv-imgproc406t64 (>= 4.6.0+dfsg), libstdc++6 (>= 13.1), libtinyxml2-10 (>= 10.0.0), ros-jazzy-ament-cmake-auto, ros-jazzy-camera-calibration, ros-jazzy-cv-bridge, ros-jazzy-depthai, ros-jazzy-depthai-bridge, ros-jazzy-depthai-descriptions, ros-jazzy-depthai-examples, ros-jazzy-depthai-ros-msgs, ros-jazzy-diagnostic-msgs, ros-jazzy-diagnostic-updater, ros-jazzy-ffmpeg-image-transport-msgs, ros-jazzy-image-pipeline, ros-jazzy-image-transport, ros-jazzy-image-transport-plugins, ros-jazzy-pluginlib, ros-jazzy-rclcpp, ros-jazzy-rclcpp-components, ros-jazzy-sensor-msgs, ros-jazzy-std-msgs, ros-jazzy-std-srvs, ros-jazzy-vision-msgs, ros-jazzy-ros-workspace
Download-Size: 750 kB
APT-Sources: http://packages.ros.org/ros2/ubuntu noble/main amd64 Packages
Description: Depthai ROS Monolithic node.

Additional context

• Camera is OAK-D-SR
• ros2 bag was also running and collecting the IMU data (which is how I first noticed the gap)

[Serafadam]

Hi, could you change i_sync_method to COPY and see if the issue still occurs?

[mrjogo]

That does seem to fix the issue (I'm trying with a more complex config file as well just to be sure). Why is that the case?

[Serafadam]

Seems like the error is somewhere in the interpolation function (interpolation is done on host here), when using COPY method the data is copied straight from DAI message without any external buffering.