[Unreal Engine Crash unpredictable when training DRL]

[changpowei]

Hardware/Software Information

• CPU: Intel® Xeon(R) Silver 4116 CPU @ 2.10GHz × 24
• GPU: NVIDIA Corporation TU102 [TITAN RTX]
• RAM: 64GB
• OS: ubuntu 20.04
• python: 3.8
• aisim version: 1.3.0
• Unreal Engine version: 4.24.3-0+++UE4+Release-4.24
• Environment: AirSimNH

Error screenshot

• I am using RL to train a drone to avoid obstacle in AirSimNH environment. It seems working perfectly over whole day (around 50000 steps in 4000 episodes), but it suddenly crash with following error log.

[saihv]

Looking at the log, it seems like you're running out of GPU memory. I would recommend running watch -d -n 0.5 nvidia-smi in another terminal and monitor the GPU memory usage.

[changpowei]

@saihv Thanks for replying, I will keep watching.

[changpowei]

Hi @saihv
I found that the GPU memory usage of AirSimNH is keep increasing. It started from about 900MB and continues to accumulate to 1700MB so far. What could be the cause of this situation? How to save memory usage?

By the way, I use lidar sensor to gather distance information.

    	"Sensors": {
    		"Lidar_down": {
    			"SensorType": 6,
    			"Enabled": true,
    			"Range": 10,
    			"NumberOfChannels": 1,
    			"RotationsPerSecond": 10,
    			"PointsPerSecond": 200,
    			"X":0, "Y":0, "Z": 0,
    			"Yaw": 0, "Pitch": 0, "Roll": 0,
    			"VerticalFOVUpper": -89,
    			"VerticalFOVLower": -91,
                    	"HorizontalFOVStart": 0,
                    	"HorizontalFOVEnd": 360,
    			"DataFrame": "SensorLocalFrame"
    			},
    		"Lidar_front_down": {
    			"SensorType": 6,
    			"Enabled": true,
    			"Range": 10,
    			"NumberOfChannels": 1,
    			"RotationsPerSecond": 10,
    			"PointsPerSecond": 1000,
    			"X":0, "Y":0, "Z": 0,
    			"Yaw": 0, "Pitch": 0, "Roll": 0,
    			"VerticalFOVUpper": -44,
    			"VerticalFOVLower": -46,
                    	"HorizontalFOVStart": -5,
                    	"HorizontalFOVEnd": 5,
    			"DataFrame": "SensorLocalFrame"
    			},
    		"Lidar_left": {
    			"SensorType": 6,
    			"Enabled": true,
    			"Range": 10,
    			"NumberOfChannels": 1,
    			"RotationsPerSecond": 10,
    			"PointsPerSecond": 2000,
    			"X":0, "Y":0, "Z": 0,
    			"Yaw": 0, "Pitch": 0, "Roll": 0,
                    	"HorizontalFOVStart": -92,
                    	"HorizontalFOVEnd": -88,
    			"DataFrame": "SensorLocalFrame"
    			},
    		"Lidar_front_left": {
    			"SensorType": 6,
    			"Enabled": true,
    			"Range": 10,
    			"NumberOfChannels": 1,
    			"RotationsPerSecond": 10,
    			"PointsPerSecond": 2000,
    			"X":0, "Y":0, "Z": 0,
    			"Yaw": 0, "Pitch": 0, "Roll": 0,
                    	"HorizontalFOVStart": -47,
                    	"HorizontalFOVEnd": -43,
    			"DataFrame": "SensorLocalFrame"
    			},
    		"Lidar_right": {
    			"SensorType": 6,
    			"Enabled": true,
    			"Range": 10,
    			"NumberOfChannels": 1,
    			"RotationsPerSecond": 10,
    			"PointsPerSecond": 2000,
    			"X":0, "Y":0, "Z": 0,
    			"Yaw": 0, "Pitch": 0, "Roll": 0,
                    	"HorizontalFOVStart": 88,
                    	"HorizontalFOVEnd": 92,
    			"DataFrame": "SensorLocalFrame"
    			},
    		"Lidar_front_right": {
    			"SensorType": 6,
    			"Enabled": true,
    			"Range": 10,
    			"NumberOfChannels": 1,
    			"RotationsPerSecond": 10,
    			"PointsPerSecond": 2000,
    			"X":0, "Y":0, "Z": 0,
    			"Yaw": 0, "Pitch": 0, "Roll": 0,
                    	"HorizontalFOVStart": 43,
                    	"HorizontalFOVEnd": 47,
    			"DataFrame": "SensorLocalFrame"
    			}

looking forward to your reply, Thanks a lot!

[wangwwno1]

Hello, apart from the GPU memory please also check the configuration files used by the environment.
If the format of settings.json is incorrect, it would raise similiar error.
Plus, please check the training code as it occupy over 21GB, periodically dump unused cache(especially for PyTorch) would help.

[changpowei]

Hi @wangwwno1 ,

I set the fixed GPU memory size when running the training code, so it occupy the 21GB of memory.

When I modify the code to dynamic adjust the memory size when training, it will reduce the usage of memory as following figure.
But unfortunately, it still face the same problem.

Thanks for replying, but I still have couple questions here.

1. What do you mean the settings.json is incorrect? Is it possible to start the AirSimNH environment if the settings.json is incorrect?

2. I currently use the official AirSimNH environment. Could it be a conflict between AirSimNH environment and local device(ubuntu 20.04 + nvidia driver 455.45.01 + cuda 11.0 + cudnn 8.0.4)?

3. I found the distance information captured by distance sensor is incorrect. It can only measure the distance in front of drone, despite the setting.json file has been modified.

[wangwwno1]

@changpowei

1. Sorry for giving misleading infos, I have checked the error msg and I think this might not your case. Still, incorrect in format (e.g. additionall comma in the end of config) or value DOES cause the original AirSim environment crash during the initialization.
2. I'm not familiar with the AirSimNH so I'm not sure about that.
3. This issue might related to the direction of sensors. The Roll, Pitch and Yaw setting in other sensors are all set to 0, that means these sensors are facing towards the FRONT direction of the drone. Try it with different values.