How to integrate AR Foundation (ARCore) camera feed with MediaPipe Unity Plugin (v0.16.1)?

[2323Furi]

Plugin Version or Commit ID

v0.16.1

Unity Version

6000.0.52f1

Your Host OS

Windows 11

Target Platform

Android

Description

Hello Homuler,

I’m currently working on a Unity AR project and would like to use your MediaPipe Unity Plugin (v0.16.1) together with AR Foundation.

Here’s my setup:

• Unity 6.0
• AR Foundation 6.1.1
• ARCore XR Plugin 6.1.1
• MediaPipe Unity Plugin v0.16.1 (by Homuler)
• Target device: Android (Samsung Galaxy A55 5G)
• Template: AR Mobile (using AR Camera for plane detection)

Problem:
When I use AR Foundation alone, the AR camera works correctly and detects planes.
When I add the MediaPipe Unity plugin for hand tracking, the AR plane detection still works,
but hand tracking does not detect anything.

It seems that both AR Foundation and MediaPipe are trying to access the camera separately.
I tried combining their camera scripts into one so that they share the same camera,
but it still doesn’t detect hands.

Question:
What is the correct way to feed the AR Foundation (ARCore) camera frame into MediaPipe,
so that I can use AR plane detection and MediaPipe hand tracking at the same time?

Thank you so much for your amazing work on this plugin

Code to Reproduce the issue

No response

Additional Context

This is the showcase for now https://drive.google.com/drive/folders/1Nhh8k5bTBwKOeXSG120kTZMeK-qhba6P?usp=drive_link

[homuler]

Please note that the sample code is not designed for reuse. It attempts to read the input image from the camera ( WebCamTexture ), which in your case is probably already being used by ARCamera.

First, please read the tutorial to learn how to use the Task API.
To integrate the Task API with ARCamera, you need to create an Image instance from the camera’s output.
The code in #343 (comment) is a bit old, but if you replace ImageFrame with Image, it should work in a very similar way.