Introduce True Zero-Copy Publish/Subscribe IPC to Autoware!! 🎉 (New Middleware Coexisting with ROS 2)

[sykwer]

TL;DR;

• TIER IV has developed Agnocast, a novel True Zero-Copy middleware that can be easily integrated into Autoware.
• Unlike IceOryx, it supports zero-copy communication for all message types, including unsized message types.
• This discussion covers its integration into Autoware, performance improvements, and future deployment considerations.

Agnocast repo: https://github.com/tier4/agnocast

---

Autoware mitigates the overhead of Pub/Sub communication between nodes by using ComponentContainer to host multiple nodes within the same process.However, not all nodes can currently be consolidated into a single process, leading to significant inter-process communication (IPC) costs, especially when transferring large messages such as PointCloud2.
From a fault isolation perspective, it is preferable to place each node in a separate process, making the long-term goal to phase out the use of ComponentContainer.To achieve a one-node-per-process architecture while minimizing IPC overhead, zero-copy communication between processes is essential.

Existing zero-copy middleware solutions include IceOryx (https://iceoryx.io/), which can be integrated seamlessly when using CycloneDDS without requiring changes to the technology stack (https://github.com/ros2/rmw_cyclonedds/blob/rolling/shared_memory_support.md). However, IceOryx only supports zero-copy communication for message types with static size. Since Autoware extensively relies on unsized sized message types (e.g., messages containing std::vector fields), integrating IceOryx poses significant challenges.

A zero-copy middleware suitable for Autoware must satisfy all of the following requirements:

1. Support zero-copy communication for all message types, including unsized message types.
2. Require minimal modifications to existing Autoware code.
3. Allow selective zero-copy communication between specific nodes without disrupting the existing communication mechanism for other nodes.
4. Ensure that inter-host node communication remains fully compatible with the existing communication mechanism.

While IceOryx meets all requirements except the first, no existing middleware—either in production or in research—fully satisfies all these criteria. Some prior research has explored solutions addressing the first requirement, but no middleware has yet achieved comprehensive compliance with all four requirements, even at the research level.

To address these challenges, TIER IV has developed Agnocast (https://github.com/tier4/agnocast), a zero-copy middleware that fully satisfies all four requirements. Agnocast supports "True Zero-Copy Communication" by eliminating both serialization and deserialization, ensuring optimal efficiency. In this discussion, we will introduce the integration of Agnocast into Autoware and explore any overlooked concerns that may need further consideration.

Agnocast Software Stack

Agnocast consists of three main components:

• Client Library (agnocastlib)
• Heap Runtime (agnocast_heaphook)
• Kernel Module (agnocast_kmod)

As illustrated in the diagram, Agnocast operates independently of the ROS 2 software stack (rclcpp ~ rmw middleware). This independence allows it to function agnostically alongside any rmw middleware (e.g., any DDS), regardless of the specific implementation chosen. agnocast_heaphook is a dynamically injected library, inserted via LD_PRELOAD, that hooks heap-related functions such as malloc and free. agnocast_kmod serves as the central manager for Agnocast’s key data structures and is accessed via system calls from agnocastlib.

Required Modifications to Integrate Agnocast into Autoware

To apply Agnocast to Autoware, the following changes and operations are necessary:

• Modifications to Autoware's source code. Only the smart pointer type wrapping the message type and the publish/subscribe functions need to be modified. For non-ComposableNode implementations, the Agnocast-specific executor must be used within the main function.
• Modifications to Autoware's launch files. Setting environment variables such as LD_PRELOAD and applying the Agnocast-specific executor.
• Installation of dependencies (e.g., apt install ros-humble-agnocast)
• Insertion of the kernel module (e.g., insmod /path/to/agnocast.ko)

Agnocast provides a smart pointer called agnocast::ipc_shared_ptr, which manages the lifetime of message objects transferred between processes. The only required change for publisher and subscription functions is to update their namespace from rclcpp to agnocast.

In the launch file, environment variables such as LD_PRELOAD must be set to apply agnocast_heaphook to processes participating in Agnocast communication. Additionally, for ComposableNode, the Agnocast-specific executor must be applied within the launch file.

When the publisher and subscription are rewritten using Agnocast's API, the corresponding communication edge achieves True Zero-Copy—regardless of whether the nodes are in the same process or different processes. For all other communication edges, whether on the same host or across different hosts, the existing ROS 2 communication stack (e.g., DDS) remains operational.This means that communication edges can be incrementally transitioned to True Zero-Copy, starting with the most performance-critical bottlenecks. Currently, "ROS 2 Publish → Agnocast Subscription" is not supported, but it may be supported in the future.

The dependency packages will be released so that they can be installed using commands such as apt install ros-humble-agnocast (WIP: ros2-gbp/ros2-gbp-github-org#712, ros/rosdistro#44641). agnocast_heaphook and agnocast_kmod will be released via a PPA and will be installed automatically along with agnocastlib. For agnocast_kmod, it should either be explicitly inserted using the insmod command or configured to be automatically inserted when the Linux system starts.

Eliminate Side Effects of Agnocast Integration

As explained in the previous section, the modifications required to integrate Agnocast into Autoware are minimal. However, to further facilitate the initial integration, Agnocast will be integrated so that Autoware can be built without any dependency on Agnocast by default. In other words, it will be possible to build Autoware without installing ros-humble-agnocast. When using the existing build commands, the generated build artifacts will remain unchanged. This is achieved by introducing macros, as shown in the example code.

When the environment variable ENABLE_AGNOCAST is set, these macros will expand to Agnocast's API and be used during the build process.

export ENABLE_AGNOCAST=1
rosdep install -y --from-paths src --ignore-src --rosdistro $ROS_DISTRO
colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release

The environment variable enforcement can be overridden using CMake options.

colcon build --cmake-args -DUSE_AGNOCAST=ON ...
colcon build --cmake-args -DUSE_AGNOCAST=OFF ...

Additionally, the followings are considered:

• Communication across different Docker containers can also be achieved with True Zero-Copy.
• CARET (measurement tool) remains fully functional, as tracepoints have been integrated.
• Various ROS 2 commands (e.g., ros2 topic list) have been extended to support Agnocast communication.
• etc...

Deployment Experiments and Evaluation Results at TIER IV

TIER IV conducted deployment experiments and performance evaluations of Agnocast in the following environments. In all cases, Agnocast was confirmed to function correctly:

• Autoware Universe v0.41.2
• Pilot-Auto Cargo (Reference Design)
• Pilot-Auto Robo-Taxi (Reference Design)

This section shares the results of applying Agnocast to Autoware Universe v0.41.2. The applied code can be found in the following branch: https://github.com/veqcc/autoware/tree/test/agnocast

To target communication bottlenecks, the following topics were selected for Agnocast integration:

• /sensing/lidar/*/pointcloud_before_sync
• /perception/obstacle_segmentation/pointcloud

Since rviz subscribes to /perception/obstacle_segmentation/pointcloud, communication with rviz continued using the existing DDS transport. The following results show the reduction in lo network traffic when applying Agnocast only to the /sensing/lidar/*/pointcloud_before_sync topic. (Applying Agnocast to /perception/obstacle_segmentation/pointcloud did not contribute to traffic reduction due to rviz's presence.). Left: before Agnocast applied, Right: after Agnocast applied.
⚠️ Note: The rosbag replay speed is x0.2, meaning that the measurements reflect a 1/5 scale of actual traffic.

Applying Agnocast to the /sensing/lidar/*/pointcloud_before_sync topic alone reduces lo traffic by 2/3. As Agnocast is applied to more topics, lo traffic will approach zero.

Next, to verify that communication latency is nearly zero on edges where Agnocast is applied, we examine the latency measurements for the /perception/obstacle_segmentation/pointcloud topic.

Next, we examine the improvement in Top LiDAR Preprocessing response time due to the zero communication latency of /sensing/lidar/top/pointcloud_before_sync. Left: before Agnocast applied, Right: after Agnocast applied.
The response time has improved by an average of approximately 4ms. As seen in the Response Time Details, this improvement is attributed to the near-zero communication latency of /sensing/lidar/top/pointcloud_before_sync after applying Agnocast.

Applying Agnocast to topics that transfer large messages significantly reduced overall system memory consumption. Additionally, in processes where Agnocast was applied, DDS-related CPU usage was observed to decrease.

While this effect is not noticeable when applying Agnocast to only a few topics, full system-wide adoption would make Autoware startup instant. Currently, Autoware takes a long time to launch due to the DDS Discovery Protocol, whose message complexity scales with the square of the number of participants and the first power of the number of edges. As Agnocast is applied to more topics, the number of edges decreases, gradually reducing startup time.

Conference Paper and Document

A paper on Agnocast is accepted to ISORC 2025 and will be published in May. currently under review and will be shared here once published.

Additionally, the documentation is being written and will be shared here upon completion.
The release for Agnocast installation via apt install is currently in progress.

[sykwer]

In the coming days, Pull Requests will be submitted to the Autoware repositories to apply Agnocast to the following two topics.

• /sensing/lidar/*/pointcloud_before_sync
• /perception/obstacle_segmentation/pointcloud

[mitsudome-r]

Thanks, this looks interesting, but I have few questions:

• Initial integration seems to require changes in both source code and launch files. What happens if you only modify source code and does not update launch file? (Is it going to show some kind of error that is obvious, or does it fall back to normal non-zero-copy DDS communcation?)
• Is this solution also shared to ROS 2 Community to have design review? Ideally, I would prefer to keep the details about middleware design to be aligned with ROS 2 community.
  • For example, ROS 2 provides RMW_IMPLEMENTATION environment variable to allow users to switch between different middleware implementation (including iceoryx), without forcing the modification in source code. Are these kind of implementation and design considered?

[sykwer]

Initial integration seems to require changes in both source code and launch files. What happens if you only modify source code and does not update launch file?

As stated in the post description, the default build command (colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release) produces exactly the same build artifacts as before. Therefore, regardless of how the launch file is written, it will continue to work without any issues.

If you build with Agnocast Enabled while keeping the launch file unchanged, Autoware will terminate with an error message prompting you to modify the launch file for Agnocast.

Is this solution also shared to ROS 2 Community to have design review? Ideally, I would prefer to keep the details about middleware design to be aligned with ROS 2 community.

As stated in the post description, Agnocast is not an RMW implementation but a middleware that exists independently and in parallel with the ROS 2 technology stack. Therefore, it is unrelated to the RMW_IMPLEMENTATION environment variable. Regardless of which middleware is set for RMW_IMPLEMENTATION, Agnocast will function alongside it without any issues.

I also want to collaborate with the ROS community, but I do not intend to undergo a design review in the near future. However, In the long term, I aim to promote the adoption of Agnocast within the ecosystem.

[esteve]

@sykwer this looks really cool. It'd be good to reach out to the ROS 2 community as soon as possible to obtain feedback. I agree with @mitsudome-r that aligning ourselves with the ROS 2 community will benefit us in the long term, especially with maintaining the project and hopefully influence the design of rclcpp as well to avoid any unnecessary maintenance overhead.

[xmfcx]

For large static-sized messages, how does it compare to iceoryx?

[sykwer]

This is a fundamental evaluation of communication latency among Agnocast, IceOryx, and CycloneDDS, conducted as part of our paper under review. In a no-load environment, we placed the Publisher Node and Subscriber Node in separate processes and plotted the time from when a message was published to when the subscription callback began execution. The visualization shows how latency changes as the message size varies between 1 KB, 10 KB, 100 KB, and 1 MB.

IceOryx achieves zero-copy communication for the static-sized message type, whereas copies occur for unsized message types. Therefore, we considered IceOryx with unsized messages and IceOryx with static-sized messages of the same size as separate comparison targets.

As a result, Agnocast was the only option where latency remained constant as message size increased when handling unsized message types. At a message size of 1 MB, IceOryx had a communication latency close to 1.0 ms, whereas Agnocast remained below 0.1 ms.

Since Autoware's PointCloud2 topics can exceed 10 MB, the latency gap is expected to widen even further.

[sykwer]

For static-sized message types, as shown in the figure, IceOryx also exhibits constant communication latency. In the case of Agnocast, its IPC mechanism is the same for both static-sized and unsized message types, meaning it demonstrates the same latency characteristics for static-sized message types as depicted in the figure.

(We have many other fundamental evaluation results, but since they contain substantial content from our paper under review, we will explain them directly in a meeting.)

[xmfcx]

Wow! From this analysis, it looks like that Agnocast is faster than even the static sized iceoryx!

Btw, if the analysis is still ongoing, might be a good idea to compare with iceoryx2 too: https://github.com/eclipse-iceoryx/iceoryx2 as it seems to have even lesser latency.

[elBoberido]

@xmfcx with the v0.5 release, iceoryx2 supports fully dynamic arrays. You just need to specify the number of elements and if there is enough free memory available, iceoryx2 will give you such an array without you having to do anything :)

If you want to know more, just start a new discussion here or at eclipse-iceoryx/iceoryx2. I don't want to hijack this discussion thread :)

[mitsudome-r]

Notes from a separate call among @xmfcx, @sykwer, and @mitsudome-r

• There are no intention of integrating Agnocast natively with ROS 2.
• What happens if you set intra_process_comms with component container? Any performance difference?
  • It will use intra_process_comms instead of agnocast. It probably won't have much difference in performace, but we can run as a isolated node with agnocast.
• Users would have to use agnocast cli (ros2 topic list_agnocast) to list up the topics that are published under Agnocast API
• Agnocast currently does not support ROS2 Publisher -> Agnocast Subscriber.
  • This is okay for TIER IV usecase, but required by OSS ecosystem.
• What are supported QoS?
  • History, Depth, Reliability, Durability are supported.
  • liveliness, deadline are not supported
    
    

Discussion Points:

• how well is agnocast going to be maintained?
• Any long term plan about merging into ROS Community?

[mitsudome-r]

After today's discussion, I'm personally fine with introducing agnocast into Autoware.

However, I want to keep the feature turned off by default until it supports "ROS 2 publisher" -> "Agnocast Subscriber" communication (especially if we plan to apply this across various nodes). This is because we never know how users use Autoware, and we don't want users to get stuck with not being able to understand why it cannot send their topics to a specific Autoware nodes. I would like to make sure that people turn on the feature consciously with understanding of the current limitations.

[sykwer]

https://github.com/orgs/autowarefoundation/discussions/5861

The slides used in the Software Working Group (3/11).
20250311 AWF Software WG.pdf

[elBoberido]

Hi @sykwer

iceoryx dev here :)

First, congratulations to Agnocast :)

It's always interesting to see the different unique approaches taken to shared memory communication. I skimmed over the documentation, however have a few questions and I hope it’s alright if I ask them:

• Are you supporting fully dynamic types like std::vector or just a dynamic array as payload, like in the example?
• Does one need to have free virtual address space at address 0x40000000000?
• Do all applications have access to the same shared memory?
• Do you plan to support other OSes than Linux?
• Are you aiming for certification, e.g. ASIL-B/D?
• How do you handle back-pressure when the publisher is faster than the subscriber?

Currently, we are developing iceoryx2 at ekxide. Similar to you, we also know too well of the limitations of iceoryx classic and we try to tackle all of them with iceoryx2. For example, inceoryx2 does not require a central daemon and since the v0.5 release from last December, we fully support dynamic arrays. We do not yet support fully dynamic types like std::vector but have ideas on how to make it work. We haven't had the time to dedicate to it due to other features taking priority, like: request-response, blackboard pattern and Python support. Since iceoryx2 has a modular design, it would be possible to integrate your allocation strategy into iceoryx2 seamlessly. This could bring the benefit of your dynamic allocator to our wide rage of supported platforms.

We are currently also working on a rmw_iceoryx2 and the next release is in the works. With that release, we will support unbounded messages via serialization. The overhead of serialization is noticeable however our latest numbers are below 0.4ms for 1MB. I guess you are using it via CycloneDDS, right?

It would be great to exchange ideas in the interest of pursuing the best solutions.

Btw, if you like, we could also help you to add iceoryx2 to your benchmark.

Oh, and we can also just move this discussion to your repo or to eclipse-iceoryx.

[sykwer]

Thanks for your questions!

Are you supporting fully dynamic types like std::vector or just a dynamic array as payload, like in the example?

We support true zero-copy transfer (eliminating serialization/deserialization) for all ROS 2 message types, including fully dynamic types like std::vector, without any modifications. All objects are transferred between processes as they are.

Does one need to have free virtual address space at address 0x40000000000?

The current implementation assumes that the virtual address space at 0x40000000000 is conventionally available. However, we plan to support dynamically determining this range in the near future.

Do all applications have access to the same shared memory?

No, only the subscriber process can read a specific heap region of the publisher process in a read-only manner.

Do you plan to support other OSes than Linux?

We have no immediate plans to support other OSes, but we would like to determine development priorities through discussions with the OSS community.

Are you aiming for certification, e.g. ASIL-B/D?

We have no immediate plans for certification, but we have not made a decision for the long term.

How do you handle back-pressure when the publisher is faster than the subscriber?

We are not entirely sure of the intent behind your question, but our system allows configuring QoS similar to ROS 2's QoS settings.

[sykwer]

We do not yet support fully dynamic types like std::vector but have ideas on how to make it work.

@elBoberido
I'm very interested in that idea! Autoware needs to support true zero-copy communication while handling all native C++ types, including std::vector, without modifications. Can all ROS 2 message types truly be transmitted with true zero-copy communication (with no serialization nor deserialization) without any modifications?

With that release, we will support unbounded messages via serialization. The overhead of serialization is noticeable however our latest numbers are below 0.4ms for 1MB. I guess you are using it via CycloneDDS, right?

We'd like to eliminate both serialization and deserialization in IPC while achieving true zero-copy communication without any modifications to ROS 2 message objects.

Btw, if you like, we could also help you to add iceoryx2 to your benchmark.

Thanks! We will add iceoryx2 for our benchmark.

[sykwer]

Notably, the DDS discovery protocol has become a bottleneck, so we also have a long-term goal of moving away from the DDS-based communication stack.

[elBoberido]

Thanks for your questions!

Are you supporting fully dynamic types like std::vector or just a dynamic array as payload, like in the example?

We support true zero-copy transfer (eliminating serialization/deserialization) for all ROS 2 message types, including fully dynamic types like std::vector, without any modifications. All objects are transferred between processes as they are.

Cool

Does one need to have free virtual address space at address 0x40000000000?

The current implementation assumes that the virtual address space at 0x40000000000 is conventionally available. However, we plan to support dynamically determining this range in the near future.

Also cool

Do all applications have access to the same shared memory?

No, only the subscriber process can read a specific heap region of the publisher process in a read-only manner.

This is something I would like to know more about. But it has to wait until I'm less busy with other tasks.

Do you plan to support other OSes than Linux?

We have no immediate plans to support other OSes, but we would like to determine development priorities through discussions with the OSS community.

This might be an option for collaboration. We already have a platform abstraction for multiple OSes. But I'm not sure how much it would fit your design since you also need a kernel module.

Are you aiming for certification, e.g. ASIL-B/D?

We have no immediate plans for certification, but we have not made a decision for the long term.

What we've learned is that one has to think about this early on, since it influences the API design and when it's done late, it limits the solution space.

How do you handle back-pressure when the publisher is faster than the subscriber?

We are not entirely sure of the intent behind your question, but our system allows configuring QoS similar to ROS 2's QoS settings.

Yeah, it's related to the QoS. I guess I need to check that. It's basically if a slow subscriber is able to receive, e.g. the latest three samples, even if the subscriber is much slower than the publisher.

We do not yet support fully dynamic types like std::vector but have ideas on how to make it work.

@elBoberido
I'm very interested in that idea! Autoware needs to support true zero-copy communication while handling all native C++ types, including std::vector, without modifications. Can all ROS 2 message types truly be transmitted with true zero-copy communication (with no serialization nor deserialization) without any modifications?

I thinks it goes into a similar direction than what you will do with dynamically determining the address range. But we approach the problem from the opposite end since we are aiming for ASIL-D from the beginning. It might be possible to still be able to use ROS 2 std_msgs but that would be limited to QM software. I think for the safety critical subset the stl types like std::vector cannot be used, since they have undefined behavior and use exceptions. Here, one would need to diverge from std_msgs.

With that release, we will support unbounded messages via serialization. The overhead of serialization is noticeable however our latest numbers are below 0.4ms for 1MB. I guess you are using it via CycloneDDS, right?

We'd like to eliminate both serialization and deserialization in IPC while achieving true zero-copy communication without any modifications to ROS 2 message objects.

Like mentioned above. For QM software this would be fine but at least for ASIL-D this would not work. But one can use these types https://github.com/ZhenshengLee/ros2_shm_msgs.

Btw, if you like, we could also help you to add iceoryx2 to your benchmark.

Thanks! We will add iceoryx2 for our benchmark.

Cool. Thanks.

Notably, the DDS discovery protocol has become a bottleneck, so we also have a long-term goal of moving away from the DDS-based communication stack.

Yeah, the DDS discovery is one of the two big pain points we hear when people complain about DDS. The other one is the complex setup. So moving away from DDS will most probably make your users happy.

[sykwer]

A conference paper on Agnocast has been accepted to ISORC 2025 and will be published in May.

[veqcc]

@mitsudome-r
cc @sykwer
Hi, I have a question about Agnocast and autoware.core.

Currently, I'm working on applying Agnocast to the following two large topics in autoware.universe:

• /sensing/lidar/*/pointcloud_before_sync
• /perception/obstacle_segmentation/pointcloud

However, I found that some related functions and subscribers have been move to autoware.core recently.
For example, autoware_motion_velocity_planner in autoware.core subcribes /perception/obstacle_segmentation/pointcloud.

So the questions is: "Is it possible to introduce Agnocast dependency into autoware.core?".

[mitsudome-r]

I might want to see them working on Universe first.
We have been seeing some issues with agnocast like this since we just introduced it to Autoware. I would like to make sure that they work more stably before introducing it to core.

[veqcc]

Thank you for your quick response.

I might want to see them working on Universe first.
We have been seeing some issues with agnocast like #5974 since we just introduced it to Autoware. I would like to make sure that they work more stably before introducing it to core.

It really makes sense, and I basically agree to this idea.
But I have several concerns on it:

• If we introdude Agnocast to a package (e.g. pointcloud_preprocessor ), then the migration of the package from universe to core is blocked?
• Our first target topics of Agnocast is the aforementioned two pointcloud topics. However, applying Agnocast to them is not desiable since some of them are already migrated to autoware.core. Then, should we apply Agnocast to other topics in autoware.universe so that we can confirm that Agnocast acutually works fine with Autoware?

[mitsudome-r]

If we introdude Agnocast to a package (e.g. pointcloud_preprocessor ), then the migration of the package from universe to core is blocked?

The nodes from pointcloud_preprocessor was reimplemented in Core and the original code remains in Universe, we can apply agnocast to the one in universe first.

[mitsudome-r]

Our first target topics of Agnocast is the aforementioned two pointcloud topics. However, applying Agnocast to them is not desiable since some of them are already migrated to autoware.core. Then, should we apply Agnocast to other topics in autoware.universe so tha we can confirm that Agnocast acutually works fine with Autoware?

If it's okay with you, let's start with /sensing/lidar/*/pointcloud_before_sync which can be done within Universe first. Like I said, once we confirm that we can use it without any issue and seems to work stably for a while, then I'm okay to use agnocast on Core. I just don't want to make Autoware Core depend on something that is unstable.

[veqcc]

If it's okay with you, let's start with /sensing/lidar/*/pointcloud_before_sync which can be done within Universe first.

Agreed. Thank you for your advice 👍

[sykwer]

Agnocast Adoption Tracker

• feat(autoware_cuda_pointcloud_preprocessor): agnocast support autoware_universe#10812
• feat(hesai): agnocast support tier4/nebula#335

[sykwer]

Academic Paper Tracker

• Agnocast Paper preprint: https://arxiv.org/abs/2506.16882