Android Lag (Face Landmark Detection) and Potential Memory Leak

[garrett-tuer]

Plugin Version or Commit ID

v0.16.1

Unity Version

6000.0.33f1

Your Host OS

Windows 11 Enterprise

Target Platform

Android

Target Device

Galaxy A53

[Windows Only] Visual Studio C++ and Windows SDK Version

No response

[Linux Only] GCC/G++ and GLIBC Version

No response

[Android Only] Android Build Tools and NDK Version

Build Tools: 34.0.0
NDK: 23.1.7779620

[iOS Only] XCode Version

No response

Build Command

None. I'm using the pre-built/setup "all" project.

Bug Description

Running the Face Landmark Detection sample scene on Android, the face landmark detection is very slow. The on-screen annotation lags behind actual movement by multiple seconds, and the lag increases over time as the app runs. This issue does not occur when running in the Editor on Windows or a built application for Windows. This issue occurs regardless of whether I build with the inference mode set to CPU or GPU.

I added a small timing test to the code based on my understanding of the FaceLandmarkerRunner class: By my understanding, the timestamp sent to the OnFaceLandmarkDetectionOutput function is the same timestamp sent via taskApi.DetectAsync in the Run method. Thus, the difference between the current time within OnFaceLandmarkDetectionOutput and the provided timestamp would give the time (or an approximation of the time) that it took to detect the face landmarks of the image matching that timestamp. Based on this premise, I logged the difference between GetCurrentTimestampMillisec() and timestamp in OnFaceLandmarkDetectionOutput to see what the "processing time" was for each image. Similar to the increasing lag I'm seeing in the built app, the "processing time" increases with each image.

private void OnFaceLandmarkDetectionOutput(FaceLandmarkerResult result, Image image, long timestamp)
    {
      long timeDiff = GetCurrentTimestampMillisec() - timestamp;
      Logger.Log("[TIME TEST] Processing time: " + timeDiff);
      _faceLandmarkerResultAnnotationController.DrawLater(result);
    }

Please let me know if my understanding of this issue and the timestamp sent to OnFaceLandmarkDetectionOutput is correct. If my simple time test is not a valid way to measure/approximate frame-by-frame processing time, please let me know what would be a good alternative, or at least where I could start looking for a solution. Any help or additional information would be greatly appreciated. Thank you!

Steps to Reproduce the Bug

1. Open the MediaPipeUnityPlugin-all project in Unity 6000.0.33
2. Add the code change shown above
3. Add Android Logcat to the project for Android debugging
4. Set the build platform to Android
5. Set the default scene (or only scene) to the Face Landmark Detection sample scene
6. Set the build as a Development build
7. Build and run the app
8. When the scene opens, make sure the camera is set to one pointing at your/a face
9. Filter the Android log for "[TIME TEST]" and check the changes in processing time over time.

Log

MediaPipeUnityAndroidLog.txt

Screenshot/Video

No response

Additional Context

LOG NOTES

1. The time stamps up until 06-26 04:21:20.110 are for the back camera with no face detected. After that, I switched to the front camera where my face was detected.
2. At 06-26 04:21:48.580, I started closing the app, which is why the processing times start decreasing after that point.

[homuler]

What's the resolution setting of your camera?
Also,

1. Is there any difference when you set ImageReadMode to CPU or GPU?
2. When you set the running mode to VIDEO, does it affect lag or other performance aspects?

[garrett-tuer]

To clarify what I mean by lag: The image annotation is lagging more and more behind the displayed camera/video image. The camera feed/video plays at the correct speed.

What's the resolution setting of your camera?

I was using the default camera resolution (1280 X 720) in my initial test. I just tested with the lowest available resolution (176 X 144) and the issue was still present. The lag built up slower in the latter case.

Is there any difference when you set ImageReadMode to CPU or GPU?

Not really. I was using CPU, but switching didn't seem to make much difference. I tested running each for about 20 seconds and checking the lag rate. The processing time in CPU increases at a rate of about 145 milliseconds per second of runtime while in GPU, the rate is about 158 milliseconds per second of runtime. That seems a little strange to me since I'd expect GPU processing to be faster. The timing is based on the Log message timestamp and reported processing time in the debug message I added.

CPU_Timestamps.txt
GPU_Timestamps.txt

EDIT: Added timestamp files

Correction: I'd mixed up files and was looking at a file from a run where I had used GPU as the preferred inference mode in the AppSettings, not one where I'd used GPU as the ImageReadMode. With GPU as the ImageReadMode in the configuration menu, the lag seems to level off at about 1100 ms. It has some spikes here and there, but it tends to return to that amount of delay between image submission and the detection output callback.

GPU_Timestamps_Fixed.txt

When you set the running mode to VIDEO, does it affect lag or other performance aspects?

I tested with a video and saw the same issue. Similar to the above test, the higher resolution of the video (1920 X 1080) caused the lag to build up faster than in the camera-based tests.

[homuler]

The sample app provides 3 different settings for both ImageReadMode and RunningMode, and their behavior varies. ImageReadMode doesn't seem to have much impact, so you can ignore that. However, I would like you to check the RunningMode (you can change them from the bottom menu while the app is running).

MediaPipeUnityPlugin/Assets/MediaPipeUnity/Samples/Scenes/Face Landmark Detection/FaceLandmarkDetectionConfig.cs

Lines 20 to 22 in 4fbe60f

	public ImageReadMode ImageReadMode { get; set; } = ImageReadMode.CPUAsync;
	public Tasks.Vision.Core.RunningMode RunningMode { get; set; } = Tasks.Vision.Core.RunningMode.LIVE_STREAM;

Specifically, VIDEO doesn't mean the input is from a video —it refers to a mode with that name. Unlike the default mode, it operates synchronously (not via callbacks). Therefore, you may need to change where you insert log for measurement purposes.
Since the main thread is blocked until the result is returned, inference for multiple frames won't run in parallel, and you should be able to measure the inference time accurately.

P.S. If possible, I’d like you to use a profiler instead of logs to check whether there are any methods that are taking a long time to execute (That said, if the native processing is what's slow, there's a high chance you won’t be able to identify anything).

[garrett-tuer]

Ah, sorry for the misunderstanding, and thank you for the explanation. I think I assumed the VIDEO running mode was only for video files and the LIVE_STREAM running mode was required for live webcam feeds.

Switching to using a live camera feed with RunningMode.VIDEO, the processing time stayed pretty consistent at 115-120 ms for the default resolution (1280 X 720) and around 110-115 ms for the lowest available resolution (176 X 144). Those times are based on a similar timing log for the RunningMode.VIDEO case in FaceLandmarkerRunner.Run():

case Tasks.Vision.Core.RunningMode.VIDEO:
  long timestamp = GetCurrentTimestampMillisec();
  if (taskApi.TryDetectForVideo(image, timestamp, imageProcessingOptions, ref result))
  {
    long timeDiff = GetCurrentTimestampMillisec() - timestamp;
    Logger.Log("[TIME TEST] VIDEO processing time: " + timeDiff);
    _faceLandmarkerResultAnnotationController.DrawNow(result);
  }
...

P.S. If possible, I’d like you to use a profiler instead of logs to check whether there are any methods that are taking a long time to execute (That said, if the native processing is what's slow, there's a high chance you won’t be able to identify anything).

I need to talk to someone in my IT department about whether I can get my computer set up for Android profiling, as the prerequisites listed here would require their assistance and may not be allowed under our security practices. I will update after talking to them.

[homuler]

I think I've found the cause.
If the processing for the previous frame hasn't finished, there's logic to ignore new input images. In the case of FaceLandmarker, it seems to check a proto option to determine whether to perform this logic, but this plugin didn't set that option.

I suspect the following code is directly responsible, but I haven't been able to identify the issue even after looking at the actual output graph.

MediaPipeUnityPlugin/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Tasks/Core/TaskInfo.cs

Lines 28 to 82 in 4fbe60f

	public CalculatorGraphConfig GenerateGraphConfig(bool enableFlowLimiting = false)
	{
	if (string.IsNullOrEmpty(taskGraph) || taskOptions == null)
	{
	throw new InvalidOperationException("Please provide both `task_graph` and `task_options`.");
	}
	if (inputStreams?.Count <= 0 || outputStreams?.Count <= 0)
	{
	throw new InvalidOperationException("Both `input_streams` and `output_streams` must be non-empty.");
	}
	if (!enableFlowLimiting)
	{
	return new CalculatorGraphConfig()
	{
	Node = {
	BuildConfigNode(taskGraph, taskOptions, inputStreams, outputStreams),
	},
	InputStream = { inputStreams },
	OutputStream = { outputStreams },
	};
	}
	var throttledInputStreams = inputStreams.Select(AddStreamNamePrefix);
	var finishedStream = $"FINISHED:{Tool.ParseNameFromStream(outputStreams.First())}";
	var flowLimiterOptions = new CalculatorOptions();
	flowLimiterOptions.SetExtension(FlowLimiterCalculatorOptions.Extensions.Ext, new FlowLimiterCalculatorOptions()
	{
	MaxInFlight = 1,
	MaxInQueue = 1,
	});
	return new CalculatorGraphConfig()
	{
	Node = {
	new CalculatorGraphConfig.Types.Node()
	{
	Calculator = "FlowLimiterCalculator",
	InputStreamInfo = {
	new InputStreamInfo()
	{
	TagIndex = "FINISHED",
	BackEdge = true,
	},
	},
	InputStream = { inputStreams.Select(Tool.ParseNameFromStream).Append(finishedStream) },
	OutputStream = { throttledInputStreams.Select(Tool.ParseNameFromStream) },
	Options = flowLimiterOptions,
	},
	BuildConfigNode(taskGraph, taskOptions, throttledInputStreams, outputStreams),
	},
	InputStream = { inputStreams },
	OutputStream = { outputStreams },
	};
	}

[homuler]

@garrett-tuer Does setting UseStreamMode = true resolve this issue?

diff --git a/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Tasks/Core/BaseOptions.cs b/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Tasks/Core/BaseOptions.cs
index 6f33530c..f150d479 100644
--- a/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Tasks/Core/BaseOptions.cs
+++ b/Packages/com.github.homuler.mediapipe/Runtime/Scripts/Tasks/Core/BaseOptions.cs
@@ -85,6 +85,7 @@ namespace Mediapipe.Tasks.Core
       {
         ModelAsset = modelAsset,
         Acceleration = acceleration,
+        UseStreamMode = true,
       };
     }
   }

If the issue is not resolved, please let me know whether the same problem occurs with other samples, such as Pose or Hand.
At this point, I suspect that when inference doesn't finish within one frame, inference processes for multiple input images are running in parallel, which may be causing the inference time to keep increasing.

[garrett-tuer]

It looks like that fixed it! Now the inference times are more stable and don't progressively increase. Instead, they're kind of all over the place between about 15 ms and 130 ms (average ~73 ms) using the default settings:

I assumed the disparity was based on whether Mediapipe considered a face tracked or not, and therefore whether the detector needed to be run again. I changed the MinTrackingConfidence from 0.5 to 0.25 in the FaceLandmarkDetectionConfig, and that gave a tighter grouping (average ~37 ms):

We'd ideally like to be able to process images at 30 FPS, so I'll keep tweaking the settings on my end. If you have any suggestions for improving the processing time, I'd be grateful, but that's outside the scope of this issue, so I think the issue can be considered resolved.

Thank you for looking into this and finding a solution quickly!

[homuler]

Thank you for checking. In practice, it's likely that the changes will be made in different places, so I'll probably share it again once it's released.

To improve performance, you could consider:

1. Disabling unnecessary output options (e.g., blendshapes, matrix)
2. Changing the delegate (i.e., CPU, GPU, TPU) — in some environments, CPU may be faster than GPU
3. Reducing the input image size to the lowest level that still maintains sufficient accuracy

However, for more precise knowledge on this topic, I recommend asking in the official MediaPipe community.

[garrett-tuer]

Thank you for the suggestions!

It looks like I spoke too soon on the issue being resolved. The issue seemed fixed, but when I reopened the app with ADB running, the increasing processing times returned. I'd been using a fresh install (from Build and Run) during each of my previous tests, so it looks like the fix only works on the initial run of the app. I closed and reopened the app a few times and it consistently showed the error.

I tested clearing the cache and the app data, as well as uninstalling the app then reinstalling it using Unity. Clearing the cache didn't seem to make a difference, but clearing all app data reset it to that initial state where things seemed to be fixed. Unfortunately, subsequently closing and reopening the app makes the issue appear again. Uninstalling and reinstalling the app had the same effect as clearing all app data.

[homuler]

Does the issue reproduce on the first launch after a clean install without the modifications in this thread?

[garrett-tuer]

Huh. No, it does not. That's strange. I could've sworn I tested it off of a fresh install before opening this issue.

So I guess the UseStreamMode edit above didn't actually do anything.

I tested the Hand and Pose landmarker scenes, and those do not have the same issue. Only the Face landmarker seems to have the issue.

[homuler]

Do you know how long a single inference with PoseLandmarker takes?
At this point, there's a hypothesis that the slowdown might be due to parallel processing.
So I'd like to know whether the issue occurs even when the processing time exceeds one frame (maybe around 33ms).
If it does take longer than 33ms, there's also a possibility that the issue lies in the implementation of FaceLandmarker in the MediaPipe itself.

[Lucky1403]

Actually I am facing to use the application(.apk) which I built using the Mediapipe.unitypackage version 0.14.1 on my android device.
Unity Version: 2022.3.50f1
Can you help me with that step by step ?
It would be very helpful for me.