Integrate Multi sensor fusion model 'BEVFusion' into Autoware

[cyn-liu]

Description

BEVFusion (paper) (code), an efficient and generic multi-task multi-sensor fusion framework. It unifies multi-modal features in the shared bird's-eye view (BEV) representation space, which nicely preserves both geometric and semantic information.

Purpose

In the existing camera_lidar perception mode of Autoware, lidar and cameras use multiple individual models for object detection, and then the detection results of the two are fused through a rule based fusion module.

In contrast, BEVFusion unifies multi-modal features in the shared bird's-eye view (BEV) perspectives for 3D object detection. In another word, BEVFusion dose all the staff above in one single model.

Further more, this is not only simplify the perception pipeline of multi-sensor fusion mode, but also continuously improve the accuracy of 3D object detection through iterative model and algorithms.

Based on above fact, we plan to integrating BEVFusion into Autoware for 3D object detection based on multi-view images and LiDAR.

Possible approaches

We plan to integrate BEVFusion algorithm based on this repo TensorRT deployment solution of BEVFusion provide by NVIDIA.

To Do List

• Wrap the original TensorRT deployment solution of BEVFusion code as a vendor ROS2 package (library package) and maintain it outside Autoware with different CIs settings and code standards.
• Create autoware_tensorrt_bevfusion package in Autoware.universe and call the bevfusion_vendor library for 3D object detection inference.
• Retrain BEVFusion model with TIER4 dataset to verify its generalization performance, upload the retrain tutorial to AWF official document site at the same time

Others

The license for BEVFusion's training code is Apache-2.0, which is compatible with autoware.

What worry me is that the license of BEVFusion's deployment code is MIT, we plan to integrate nvidia's "Lidar_AI_Solution" into autoware's repo as a separate vendor package following MIT license. Is this acceptable to AWF? Will that bring legal risk to AWF? @kminoda

[knzo25]

@cyn-liu
As a hobby, as I already performed the integration of bevfusion in autoware. In the past we had also tried to do it, but got stopped due to how NVIDIA released their bevfusion implementation (shared library and copyright).

https://github.com/knzo25/bevfusion_ros2

My integration directly uses weights trained using mmdet3d or similar, and uses either mmcv's spconv or traveller59's one as TensorRT plugins. The before mentioned repo contains only mmcv's implementation since I am still cleaning travellers59's.

Regarding licenses, both are Apache, so there would be no problem:
https://github.com/open-mmlab/mmcv
https://github.com/traveller59/spconv

As to why I haven't sent a PR yet, I am waiting for autoware's TensorRT update, which should happen soon.

[knzo25]

Regarding 2), I don't really have a say in that, maybe autoware's architects have a more defined opinion. However, as explained previously, I don't feel particularly happy linking against something I can neither control (fix) nor know its actual implementation (black box).

I will post any updates from our side to this issue

[liuXinGangChina]

Got you point ， kenzo-san

BY “ autoware's architects ” ， do you mean mitsudomi-san？

happy hacking！

心刚

[knzo25]

Yes, I meant people like @mitsudome-r and @xmfcx !

[liuXinGangChina]

Thanks kenzo-san， i will discuss my proposal with them and get back to you soon

Have a nice day！

心刚

[knzo25]

Today I had some time so pushed and cleaned most of the code (there will be minor changes to comply with autoware's code standards, bu other than camera synchronization aspects, it is mostly done).
https://github.com/knzo25/bevfusion_ros2

Wrote some benchmarks and it think using NVIDIA's spconv implementation will not make it much faster. As they mention, when using CL the bottleneck is the camera backbone