FPS Drops Low When Running Autoware on Jetson and AWSIM on PC Simultaneously

[banban1029]

I run Autoware on Jetson AGX Orin and AWSIM on a Ubuntu 22.04.

The AWSIM runs smoothly with 60 FPS when alone. However, when I run Autoware on Jetson and connect them with the PC via ROS2 (Cyclone DDS Unicast), the FPS of AWSIM drops to 0~1 hz. The car cant drive autonomously. Why does the FPS drop like this?

environment

• awsim lab-v1.3.1(branch) - Ubuntu22.04-pc
• autoware-awsim-stable-update(branch) - Jetson Orin AGX jetpack6.0

method

https://autowarefoundation.github.io/AWSIM-Labs/main/GettingStarted/QuickStartDemo/

1. ./awsim
2. ros2 launch autoware_launch e2e_simulator.launch.xml map_path:=$HOME/{マップの場所}/autoware_map/nishishinjuku_autoware_map vehicle_model:=sample_vehicle sensor_model:=awsim_labs_sensor_kit launch_vehicle_interface:=true

status

senscing topic down 1fps???
**
**

ros2 topic hz /

• ros2 topic hz /localization/kinematic_state

  average rate: 0.534
  	min: 1.842s max: 1.900s std dev: 0.02891s window: 2
  average rate: 0.494
  	min: 1.842s max: 2.335s std dev: 0.22018s window: 3
  average rate: 0.503
  	min: 1.842s max: 2.335s std dev: 0.20268s window: 4

topic from awsim

• /awsim/ground_truth/localization/kinematic_state

  average rate: 15.905
  	min: 0.001s max: 0.230s std dev: 0.06928s window: 18
  average rate: 12.169
  	min: 0.001s max: 0.659s std dev: 0.13172s window: 26
  average rate: 10.635
  	min: 0.001s max: 0.786s std dev: 0.15894s window: 40

• /awsim/ground_truth/vehicle/pose/clock

  WARNING: topic [/awsim/ground_truth/vehicle/pose/clock] does not appear to be published yet

• /control/command/control_cmd

  The message type 'autoware_control_msgs/msg/Control' is invalid

• /control/command/emergency_cmd

  The message type 'tier4_vehicle_msgs/msg/VehicleEmergencyStamped' is invalid

• /control/command/gear_cmd

  The message type 'autoware_vehicle_msgs/msg/GearCommand' is invalid

• /control/command/hazard_lights_cmd

  The message type 'autoware_vehicle_msgs/msg/HazardLightsCommand' is invalid

• /control/command/turn_indicators_cmd

  The message type 'autoware_vehicle_msgs/msg/TurnIndicatorsCommand' is invalid

• /sensing/camera/traffic_light/camera_info

  average rate: 0.540
  	min: 1.839s max: 1.865s std dev: 0.01326s window: 2
  average rate: 0.506
  	min: 1.839s max: 2.226s std dev: 0.17667s window: 3
  average rate: 0.516
  	min: 1.815s max: 2.226s std dev: 0.16827s window: 4

• /sensing/camera/traffic_light/image_raw

  average rate: 0.133
  	min: 2.368s max: 12.724s std dev: 5.17822s window: 2
  average rate: 0.132
  	min: 2.368s max: 12.724s std dev: 4.22800s window: 3
  average rate: 0.157
  	min: 2.368s max: 12.724s std dev: 4.20948s window: 4

• /sensing/gnss/pose

  average rate: 0.169
  	min: 5.833s max: 5.985s std dev: 0.07591s window: 2
  average rate: 0.169
  	min: 5.833s max: 5.985s std dev: 0.06301s window: 3
  average rate: 0.169
  	min: 5.833s max: 5.985s std dev: 0.05603s window: 4

• /sensing/gnss/pose_with_covariance

  average rate: 0.168
  	min: 5.893s max: 6.008s std dev: 0.05765s window: 2
  average rate: 0.168
  	min: 5.893s max: 6.008s std dev: 0.04735s window: 3
  average rate: 0.167
  	min: 5.893s max: 6.047s std dev: 0.05980s window: 4

• /sensing/imu/tamagawa/imu_raw

  average rate: 4.432
  	min: 0.008s max: 0.717s std dev: 0.22428s window: 7
  average rate: 5.588
  	min: 0.008s max: 0.717s std dev: 0.17007s window: 15
  average rate: 4.821
  	min: 0.008s max: 0.780s std dev: 0.20923s window: 18

• /sensing/lidar/left/pointcloud_raw

  average rate: 1.516
  	min: 0.458s max: 1.054s std dev: 0.27926s window: 3
  average rate: 1.341
  	min: 0.458s max: 1.054s std dev: 0.28397s window: 4
  average rate: 1.121
  	min: 0.458s max: 1.477s std dev: 0.38735s window: 5

• /sensing/lidar/left/pointcloud_raw_ex

  average rate: 0.932
  	min: 0.530s max: 1.616s std dev: 0.54273s window: 2
  average rate: 1.258
  	min: 0.511s max: 1.616s std dev: 0.47405s window: 4
  average rate: 1.064
  	min: 0.511s max: 1.616s std dev: 0.51363s window: 5

• /sensing/lidar/right/pointcloud_raw

  average rate: 0.683
  	min: 0.526s max: 2.727s std dev: 0.92733s window: 3
  average rate: 0.908
  	min: 0.526s max: 2.727s std dev: 0.84529s window: 5
  average rate: 0.785
  	min: 0.526s max: 2.727s std dev: 0.86218s window: 6

• /sensing/lidar/right/pointcloud_raw_ex

  average rate: 0.766
  	min: 0.544s max: 2.069s std dev: 0.76269s window: 2
  average rate: 1.071
  	min: 0.454s max: 2.069s std dev: 0.61507s window: 5
  average rate: 0.783
  	min: 0.454s max: 2.996s std dev: 0.95193s window: 6

• /sensing/lidar/top/pointcloud_raw

  average rate: 1.176
  	min: 0.609s max: 1.091s std dev: 0.24121s window: 2
  average rate: 1.399
  	min: 0.533s max: 1.091s std dev: 0.22014s window: 4
  average rate: 1.272
  	min: 0.533s max: 1.091s std dev: 0.24332s window: 5

• /sensing/lidar/top/pointcloud_raw_ex

  average rate: 1.156
  	min: 0.784s max: 1.021s std dev: 0.11072s window: 3
  average rate: 1.294
  	min: 0.578s max: 1.021s std dev: 0.14633s window: 5
  average rate: 1.340
  	min: 0.522s max: 1.021s std dev: 0.15549s window: 7

• /vehicle/status/control_mode

  The message type 'autoware_vehicle_msgs/msg/ControlModeReport' is invalid

• /vehicle/status/gear_status

  The message type 'autoware_vehicle_msgs/msg/GearReport' is invalid

• /vehicle/status/hazard_lights_status

  The message type 'autoware_vehicle_msgs/msg/HazardLightsReport' is invalid

• /vehicle/status/steering_status

  The message type 'autoware_vehicle_msgs/msg/SteeringReport' is invalid

• /vehicle/status/turn_indicators_status

  The message type 'autoware_vehicle_msgs/msg/TurnIndicatorsReport' is invalid

[xmfcx]

This is most probably related to DDS and overall network configuration settings.

Make sure you have set up this:

• https://autowarefoundation.github.io/autoware-documentation/main/installation/additional-settings-for-developers/network-configuration/dds-settings/
  • You might need to configure this both on sim and autoware pc.
  • This will make it run on localhost, configure the related parts accordingly for multi machine setup.

And then configure this:

• https://autowarefoundation.github.io/autoware-documentation/main/installation/additional-settings-for-developers/network-configuration/multiple-computers/

[banban1029]

I've done cyclon-dds, so I gained

` sudo systemctl status multicast-lo.service
○ multicast-lo.service - Enable Multicast on Loopback
Loaded: loaded (/etc/systemd/system/multicast-lo.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Mon 2024-12-23 16:05:18 JST; 4min 21s ago
Process: 4140571 ExecStart=/usr/sbin/ip link set lo multicast on (code=exited, status=0/SUCCESS)
Main PID: 4140571 (code=exited, status=0/SUCCESS)
CPU: 2ms

Dec 23 16:05:18 banban systemd[1]: Starting Enable Multicast on Loopback...
Dec 23 16:05:18 banban systemd[1]: multicast-lo.service: Deactivated successfully.
Dec 23 16:05:18 banban systemd[1]: Finished Enable Multicast on Loopback.`

but fps drops. Is there any way to check?
I don't know idea whether DDS is accurated.

[rt-2pm2]

I had a similar problem, low fps but with both AWsim and Autoware on the same machine.

The AWSim was running at 60 fps alone, but was dropping to 1-2 fps once Autoware started.
As suggested by @xmfcx . it was a problem with the DDS. I was using a configuration found on the Eclipse site and not the one suggested by the Autoware devs. Once I fixed that, I started getting a negligible fps drop.

In your case you say:

Dec 23 16:05:18 banban systemd[1]: Starting Enable Multicast on Loopback... Dec 23 16:05:18 banban systemd[1]: multicast-lo.service: Deactivated successfully. Dec 23 16:05:18 banban systemd[1]: Finished Enable Multicast on Loopback.`

The loopback interface is used when everything is on the same machine. If you have two machines, you should probably either select the ethernet interface you use to connect to the LAN, or let the system autoselect it as suggested by @xmfcx.

My 2c

[rt-2pm2]

@banban1029, Any update regarding this?

I tried a distributed configuration similar to the one reported by the author of the issue and I am experiencing the same problem.

I configured the Cyclone DDS as reported in the documentation and modified the xml for the case of multiple machines.

I made some tests and it seems that the problem is indeed network related.
I run a ros2 topic hz /sensing/camera/traffic_light/image_raw on the processing machine running Autoware, and the simulator on the other machine experiences a noticeable drop in the framerate.
I don't have any issue if I listen to other lighter topics.

Monitoring the bandwidth, while listening to the camera, I see that the stream occupies about half of the available bandwidth (~400-500 Mbps). Is a 1Gbps LAN sufficient for a distributed setup or is it necessary to have a 10Gbps?

My configuration is composed as follow:

1. Simulation Machine (AWSim Lab) with i7-4790K, 32GB RAM, and a 4060Ti
2. Autoware Machine (Ampere Altra Development Platform) with the 64 Core Arm CPU and a 4060Ti

Let me know if you have any ideas regarding the issue

[rt-2pm2]

@xmfcx I saw that you made some experiments on the distributed simulation setup. You used a 10Gbps network between the machines. I would be curious to know whether that is a strict requirement to avoid the issues seen here?

Regards