[Proposal] Integration of deep learning based trajectory prediction

[ahsan155]

Motivation

Reliable prediction of the trajectories of human agents around the AD system is vital for the planning of safe and efficient trajectories. The current implementation in Autoware is a relatively limited rule-based approach that covers only a narrow range of possible scenarios. In recent years, ML-based prediction models have shown promising results in various different trajectory prediction challenges.
Integrating a sophisticated motion prediction module in Autoware will allow the ego vehicle to safely navigate in complex traffic scenarios.

Possible key advantages over rule based prediction:

1. Various implementations of transformer architectures (e.g. MTR: 2024 WOD Motion Prediction Challenge - 1st Place - MTR v3.pdf] have been highly successful in predicting the trajectory of road users in e.g. the Waymo Open Dataset Motion Prediction Challenge (Challenges Overview – Waymo Open Dataset).
2. These models can produce plausible trajectory predictions over a wide range of possible scenarios, including highly complex intersection settings as well as free-space locomotion of pedestrians.
3. The horizon of reliable predictions of these models is longer than the existing rule based prediction.
4. Most models provide a distribution of possible trajectories, allowing for an estimation of confidence and subsequent risk assessment for motion planning.

Model input (identical to current implementation):

Name	Type	Description
~/perception/object_recognition/tracking/objects	autoware_perception_msgs::msg::TrackedObjects	tracking objects without predicted path.
~/vector_map	autoware_map_msgs::msg::LaneletMapBin	binary data of Lanelet2 Map.
~/perception/traffic_light_recognition/traffic_signals	autoware_perception_msgs::msg::TrafficLightGroupArray	rearranged information on the corresponding traffic lights

Model output

Name	Type	Description
~/output/objects	autoware_perception_msgs::msg::PredictedObjects	Tracked objects with distribution of predicted paths.
Further auxiliary signals

The autoware_perception_msgs::msg::PredictedObjects message type will need to be extended to include more than one predicted trajectory.

Subsequent handling of the predicted trajectories:

• Continuous risk assessment for early detection of possible collisions.
• Planning of safe paths in the vicinity of other agents.
• Modules in planning component affected by trajectory:
  • Avoidance module (dynamic): This module is under development. It can be assisted by motion prediction. Predicted trajectory can be used to stop the ego vehicle or trigger lane change to go around an obstacle.
  • Lane change module: considers predicted trajectory to change lane and keep a safe distance from other vehicles. Specifically, predicted trajectories of target lane vehicles are needed to determine if there is enough gap in between to change lanes.
  • Behavior path planner (high level): depends on predicted trajectories to generate safe path and drivable areas. Trigger other modules such as lane change when necessary.
  • Collision checker: verifies the safety of the planned path against predicted object positions.

Integration potential

A simple way to integrate would be to replace the existing map based prediction module. The existing module input is similar to what our model uses which are object tracking history and lanelet map.

Tracking data of a particular vehicle with unique id can be retrieved from TrackedObjectKinematics message which contains pose with covariance and twist with covariance features. Vehicle position and orientation history can be placed in a buffer. Once enough history is collected for an agent, the model will start making predictions.

Also, our model uses a lanelet map for the purpose of calculating vehicle distance from lane boundaries and collecting possible routes a vehicle can take from a certain pose. Vehicle to a lane boundary distance can be easily calculated if lanes are defined as polylines. Multiple nearest lane point distance can be taken from a vehicle and averaged to get an estimated distance. For retrieving possible routes from a certain vehicle pose, lane center points of the lane that contains the vehicle and lane center points of connected lanes can be gathered. If possible route information can’t be collected in this manner then we can have predefined route information saved for later retrieval. We can drive around in the chosen map and save the route information a vehicle can follow from different poses of the map.

Each of the ten future coordinates that our model outputs can be set as Pose with geometry_msgs/msg/Pose type. Each of these Pose can be inserted into the Pose[] array in autoware_perception_msgs/msg/PredictedPath type. Modules in planning will be able to access this trajectory array as it will be published.

Architecture questions

• The model we are proposing takes the past five positions and predicts a trajectory for ten timestep in the future. The difference between each timestep is 0.1 second. For model training, data was collected in carla simulation sync mode. The simulation speed was 10hz. Consistent time difference between each of the five historical data points is critical because the model relies on a temporal structure. Data collection rate by real world hardware must be consistent with simulation settings in which model training data was collected.

  • Do I need to adjust insertion of historical data in the data buffer depending on the publish rate of TrackedObjects message?
  • Does the current map based prediction have any preprocessing step that creates historical data with consistent time difference?
  • Is there any recommended prediction horizon for downstream modules to get the most benefit out of prediction?
  • Object tracking can be impacted by sensor noise or occlusion which creates gap in data. How will prediction handle this gap?

• It was observed in simulation that autoware struggles to estimate vehicle future trajectory in intersections. Behavior velocity planner has intersection as a sub-component. A purpose of this sub-component is to check collisions with upcoming vehicles. Assuming current map based prediction works with vehicles with constant velocity, will this sub-component benefit from our prediction model that can account for speed change.
  

• How to add a distribution of possible predicted trajectories to the PredictedObjects message type? Options: Provide a set of the N most probable trajectories (including their weights) or provide a distribution of possible trajectories.

• What is a good way to integrate a distribution over future positions of each predicted agent into downstream components such as trajectory planning and collision avoidance?

[xmfcx]

Related:

• ML-based Motion Prediction #6348

• feat(perception): add support of ML-based prediction with MTR #6777

  • If you want to know some examples of using predicted path from perception output in planning, you can check dynamic obstacle avoidance and intersection modules.
  • https://autowarefoundation.github.io/autoware.universe/main/planning/behavior_path_planner/autoware_behavior_path_dynamic_obstacle_avoidance_module/#determination-of-lateral-dimension
  • https://autowarefoundation.github.io/AWSIM-Labs/main/GettingStarted/QuickStartDemo/
  • https://autowarefoundation.github.io/autoware-documentation/main/design/autoware-architecture/planning/

[mitsudome-r]

cc. @ktro2828

[technolojin]

Hi.
This is Taekjin Lee, developer of the perception components.

I may can answer some of questions.

Do I need to adjust insertion of historical data in the data buffer depending on the publish rate of TrackedObjects message?

autoware_multi_object_tracker supports constant time interval mode (setting a parameter enable_delay_compensation to True). However, to absorb system fluctuation of overall pipeline latency, the frequency is not so strict. It will fluctuate within 10% of interval.

Does the current map based prediction have any preprocessing step that creates historical data with consistent time difference?

Short answer: NO
Long answer: It has historical data accumulation, but it is for predict path decision, not for consistent time difference.

Is there any recommended prediction horizon for downstream modules to get the most benefit out of prediction?

It depends on user requirement. It has configuration parameter prediction_time_horizon.

Object tracking can be impacted by sensor noise or occlusion which creates gap in data. How will prediction handle this gap?

Current rule-based prediction has no requirement of consist time interval.

How to add a distribution of possible predicted trajectories to the PredictedObjects message type?

PredictedObjects -> PredictedObject -> PredictedObjectKinematics -> PredictedPath[]
The predicted object message has member predicted_paths. It is a vector can store multiple predicted paths.

[mitsudome-r]

@ahsan155 Here's the slide that @ktro2828 shared today:
2025_04_09 ML-based Prediction Status.pdf