Performance impact of removing camera helper on Android

[j20001970]

This is rather a casual testing than a formal report, as it turns out that I'm dumb and lazy at perform proper profiling.

Camera helper is deprecated since GDMP v0.5 in favour of CameraServer-based methods from Godot Engine, and is planned to be disabled by default in next release. But we need to know that how it will affect the performance when alternative ways to access camera frames are employed.

Since we know that desktop platforms (Linux, Windows, macOS) are generally powerful enough for copying larger pixel data, and the camera frames from webcams (should) have no orientation problems, so replacing camera helper based on OpenCV with Godot camera feeds that use platform-specific API for accessing webcams would be a no-problem if not considering the benefits of removing the dependencies of OpenCV video module, but that's not the case for mobile platforms (Android, iOS), especially on Android that the orientation of camera sensors are device-specific, and we have to rotate the camera pixel data according to the orientation of both camera sensor and device screen to obtain a frame with proper rotation, which would impact performance if not done cleverly.

For camera helper on Android, we use MediaPipe java code that convert external texture into GLES texture with the options of performing rotation and y-flip, all are done using GPU acceleration.

By removing camera helper, we will definitely lose the optimizations we already have before passing camera frame data into MediaPipe vision tasks on Android, but in order to achieve standardized ways to access camera frames on Godot Engine, and reduce the burden of platform-specific code that GDMP carries without too much performance loss, some testing need to be done for comparison before camera helper is removed.

Setup

In this testing, we used Android Emulator 35.5.10 using Intel x86_64 Atom System Image with API level 35 and 4 CPU cores, running Godot Engine v4.4.1 with GDMP commit 02a6585, and compare camera helper with CameraServerExtension addon, a Godot v4.4 GDExtension plugin that provide additional platforms support to CameraServer. As of CameraServerExtension version 2025.04.06, camera frame rotation on Android is done using rotate_90 and rotate_180 from Godot Image class, while very straightforward, this is the least performant way as it uses CPU to manipulate pixel data, and JPEG format is used to provide camera frames, which might introduce additional CPU overhead for decompression.

The setup is run on hardware with Intel Core i5-12500 processor and 32GiB system memory, running Arch Linux with kernel version 6.15.3-arch1-1. Note that a desktop is usually more powerful than a mobile device in processing power, any performance differences in the result below might be more magnificent when running on actual phone or tablet.

Evaluation method

GDMP Demo project is used to measure the average time delta between two consecutive _camera_frame calls from VisionTask.gd, in microseconds (usec), with 10,000 samples in total. The actual vision task inference is skipped since we only care about the camera frame part of performance.

var frame_count := 0
var prev_tick := 0
var avg_frame_delta := 0.0

func _camera_frame(image: MediaPipeImage) -> void:
	frame_count += 1
	var t := Time.get_ticks_usec()
	if prev_tick == 0:
		prev_tick = t
	avg_frame_delta += ((t - prev_tick) - avg_frame_delta) / frame_count
	prev_tick = t
	if frame_count % 100 == 0:
		print("%d: %f" % [frame_count, avg_frame_delta])

Result

Using camera helper with resolution 640x480, we got 33420.931800 usec on average, while using camera feed with the same resolution have 33419.214100 usec average time. So far so good, since VGA is still a reasonable resolution for vision tasks that CPU can handle, but mobile devices with less powerful processor might still perform worse.

Using camera helper with resolution 1280x720 got 33421.685500 usec on average, which is about the same as 640x480. But for camera feed with 1280x720 we got 50164.849700 usec, and this is running on a desktop platform. It would struggle on most mobile devices unless it is very high-end.

How about pulling GPU frames to system memory? Since camera helper on Android create GPU images, we wanted to know if converting the data to CPU also takes considerable amount of time. Turns out it only took 33418.607500 usec on average with an additional image.convert_to_cpu() call for each 1280x720 camera frame from camera helper. Even if the memory speed of mobile devices is usually not as good as desktop, it might still be worth rotating camera frames on GPU, then pull them to system memory.

Since CameraServerExtension uses CPU to rotate camera frames, what happen if no rotation is required? We measured the average time between two 1280x720 camera frames using camera feed, with the device orientation that do not need image rotation, we got 33711.571800 usec on average as the result. Considering the data is in JPEG format and decompression is performed for each frame, we can confirm that image rotation on CPU is the most critical part for the performance of camera feed being worse than camera helper, somewhat expected but testing it in advance is always good before the complete removal of camera helper.

Conclusion

Without camera helper, performance on reading camera frames will depend on how camera feed drivers are implemented. With current CameraServerExtension addon, accessing larger camera frames on Android may be inefficient since rotating image frames is done on CPU. Note that official Godot Engine will support camera feeds on more platforms in future releases, which may be more performant than the current CameraServerExtension addon.