v0.16.3

Bug Fixes

• prevent crash in Holistic Landmarker when face is lost (#1418) (41886bd) by @NullClone

Full Changelog: v0.16.2...v0.16.3

v0.16.2

Bug Fixes

• Workaround for compilation errors on macOS (#1404) (4504bd8)

Build System

• MediaPipe v0.10.22 (#1357) (4fbe60f)
• Drop support for Ubuntu 20.04 (#1408) (f81cc1d)
• Upgrade the Android NDK version to support 16KB page size (#1407) (f7d125f)
• Upgrade Unity to support Android 16KB page sizes (#1409) (1c0aee7)

Full Change Log: v0.16.1...v0.16.2

v0.16.1

Features

• implement GestureRecognizer (#1352) (606bac5)

Bug Fixes

• Destroy textures to fix memory leak (#1337) (ef8d2c0) by @ishigurobot
• Fix coordinate transformation process in FaceLandmarker (#1355) (aa61073) by @malaybaku

Full Changelog: v0.16.0...v0.16.1

v0.16.0

Starting from this release, the OpenCV version has been upgraded. In particular, on Android, the application will not work unless libstdc++_shared.so is explicitly included in the APK, so please be aware of this.
For more details, please refer to the README.

Additionally, I have added a tutorial on how to use the Task API. Since an API reference is not yet available, please refer to the documentation comments via an editor or other means for now.

Lastly, HolisticLandmarker is implemented, but it appears that it is not officially supported yet. In fact, its I/F differs from other Tasks. While it should work fine on Desktop, using it on mobile is still quite challenging.

⚠️ BREAKING CHANGES

• update OpenCV version to 4.10.0 (#1295) (d1e81f0)
• MediaPipe v0.10.20 (#1290) (cfd83a5)

Features

• create Image/ImageFrame from Texture2D (#1329) (e85d987)
• drop Unity 2021 support (#1288) (4bf760d)
• implement HolisticLandmarker (#1315) (da27ade)
• move ImageTransformationOptions to Packages (#1327) (04b46f4)
• sample: implement ImageReadMode (#1307) (bc0aff2)
• support EDGETPU_NNAPI (#1309) (808fbb1)

Bug Fixes

• acquire a lock in a callback to avoid data race (#1328) (acbb6ed)
• Resolve landmark jitter by adjusting frame copy timing (#1311) (0e04cb8) by @DenchiSoft
• acquire a lock when drawing annotations (#1305) (d80f33a)
• sample: keep running CalculatorGraph if another window takes focus (#1301) (f010367)
• build failure on Linux (#1287) (eeaf5fb) by @AndrejsK

Build System

• disable bitcode since it's deprecated (#1293) (dd46334)
• include generated protobuf .cs files (#1310) (a3e30c0)
• Unity 6 (#1299) (524455c)

Full Changelog: v0.15.0...v0.16.0

v0.15.0

⚠️ BREAKING CHANGES

• remove ResourceManager base class (#1210)
• move Logger from Mediapipe.Unity to Mediapipe (#1208)
• drop Unity 2020 support (#1207)

Features

• implement ImageSegmenter (#1225) (7cfa666)
• implement ObjectDetector (#1220) (e007dd7)
• read the input Texture on GPU (#1245) (1dd8fef)
• remove legacy solutions (#1233) (a38f0d6)
• Task API with GPU Image (#1242) (e9b1fe4)

Bug Fixes

• dispose callback arguments properly (#1244) (efb103a)
• workaround for #1196 (#1206) (1a9b193)

Build System

• change the default AssetLoader type (#1247) (ab78d5c)

Full Changelog: v0.14.4...v0.15.0

v0.14.4

Features

• Upgrade MediaPipe to v0.10.14 in #1200

Bug Fixes

• NullReferenceException occurs when rendering multiple face landmarks by @Eunji-new in #1175

Build Systems

• Update the Unity version (2022.3.34f1) in #1202

Full Changelog: v0.14.3...v0.14.4

v0.14.3

Temporarily, the Android library is built only for arm64 (armv7 is excluded). This is due to the convenience of building the newly added AudioClassifier. For those who want to build for armv7, please refer to #1143 (comment).
If you need a fat aar, either build individually for each platform or exclude the AudioClassifier from the build (e.g. python build.py build --android arm64 --android_fat_apk_cpu=armeabi-v7a,arm64-v8a --solutions=face_mesh).

Features

• implement AudioClassifier (#1137) (3753fdf)
• implement Packet.At (#1136) (d397429)

Bug Fixes

• memory leaks when the PoseLandmarker runs with segmentation mask enabled (#1166) (019a655)
• sample: the sync mode fails when trying to get the ImageFrame result (#1142) (b6961e5)

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself using MediaPipeUnityPlugin-all-stripped.zip or MediaPipeUnityPlugin-all.zip, which contain the iOS library.

v0.14.1

Bug Fixes

• PoseLandmarker returns only one pose if numPoses > 1 (#1125) (73554f2)
• sample: Android build error (#1127) (77183c6)

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself using MediaPipeUnityPlugin-all-stripped.zip or MediaPipeUnityPlugin-all.zip, which contain the iOS library.

v0.14.0

Yesterday, I declared the deprecation of the generic Packet, but embarrassingly, I'm going to resurrect it. Well, to be precise, I will rather make the Packet added in the previous release generic (Packet<TValue>). This will allow us to regain the type safety sacrificed in the previous release.

Since I'm removing the APIs deprecated in v0.13.1, transitioning from v0.12.0 will involve more breaking changes. Here's how the syntax will differ:

// v0.12.0
StringPacket stringPacket = new StringPacket("Hello World!", new TimeStamp(0));
var value = stringPacket.Get();
stringPacket.GetInt(); // compile error

var intPacket = new IntPacket();

// v0.13.1
Packet stringPacket = Packet.CreateStringAt("Hello World!", 0);
var value = stringPacket.GetString();
stringPacket.GetInt(); // runtime error

var intPacket = new Packet();

// v0.14.0
Packet<string> stringPacket = Packet.CreateStringAt("Hello World!", 0);
var value = stringPacket.Get();
stringPacket.GetInt(); // compile error

var intPacket = new Packet<int>();

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself. MediaPipeUnityPlugin-all-stripped.zip also contains the iOS library.

⚠️ BREAKING CHANGES

• strongly-typed Packet, once again (#1120)
• remove deprecated APIs (#1119)

See CHANGELOG for more details.

v0.13.1

This is a release after six months, and please be aware that this version includes relatively significant changes.

ATTENTION: For some reason, the iOS framework is no longer included in the unitypackage, so if you need the iOS library, please use the tarball version or build it by yourself.

Summary

• Upgraded MediaPipe to version v0.10.9
• Implemented Task API (FaceDetector, FaceLandmarker, HandLandmarker, PoseLandmarker)
• Status API has been made internal
• StatusOr<T> API has been removed
• Packet<T> is deprecated and non-generic Packet is introduced

Details

Features

• Upgraded MediaPipe to version v0.10.9
• Implemented Task API (FaceDetector, FaceLandmarker, HandLandmarker, PoseLandmarker)

Sample code can be found under Assets/MediaPipeUnity/Samples/Scenes/Tasks. Note that there's a known bug in HandLandmarker sample where handedness is reversed for Right/Left.

At this stage, the mechanism to share GlContext with MediaPipe in the Task API is not yet implemented. Therefore, it is necessary to copy the input image on the CPU, even on Android.

Breaking Changes

• Status API has been made internal
• StatusOr<T> API has been removed

Specifically, APIs that used to return Status will no longer do so (void), and APIs that used to return StatusOr<T> will now return the value directly. If the status is not OK, a BadStatusException will be thrown.
To migrate, please remove calls to Status.AssertOk and StatusOr<T>.Value. If you haven't checked Status before, it will now throw an exception. However, I think there should be no problem if the compilation step succeeds.

// graph.StartRun().AssertOk();
graph.StartRun(); // throws BadStatusException if the status is not OK

Background

As an excuse, let me provide some background information.

Originally, this plugin was created with the goal of allowing people familiar with the C++ MediaPipe API to use MediaPipe in C# with a similar syntax. Therefore, for APIs that return abseil::Status in C++, it was designed to return Status in C# as well (I would like to add that there was an intention to slack off from writing documentation by doing so).

However:

• In the latest MediaPipe, the documentation for the Python API is rather more comprehensive.
• Status probably exists due to Google's unique circumstances, not wanting to throw exceptions, and there is no rationale to return Status in C#.
• After all, calling AssertOk can throw an exception.
• Marshalling StatusOr is a cumbersome process.

Given these circumstances, at this timing, I decided to abolish them.
Status itself is still internally present as it is necessary for communicating with MediaPipe and handling errors.

Deprecation

As a forewarning for future breaking changes, Packet<T> has been deprecated.
Instead, please use a non-generic Packet.
This Packet is implemented to be similar to the interface of packet_creator and packet_getter in the Python API.

// before
var stringPacket = new StringPacket("Hello World!", new TimeStamp(0));
var intPacket = new IntPacket(0);
var value = stringPacket.Get();

// after
var stringPacket = Packet.CreateStringAt("Hello World!", 0);
var intPacket = Packet.CreateInt(0);
var value = stringPacket.GetString();

As a trade-off, because the data type inside Packet can no longer be statically determined, the OutputStream API is no longer able to return a value. Instead, an OutputStream API that returns Packet has been added (I apologize for the lack of documentation at the moment).

// before
var multiFaceLandmarksStream = new OutputStream<NormalizedLandmarkListVectorPacket, List<NormalizedLandmarkList>>(calculatorGraph, "multi_face_landmarks");
if (multiFaceLandmarksStream.TryGetNext(out var multiFaceLandmarks, false)) // may block the main thread
{
    // ...
}

// after
var multiFaceLandmarksStream = new OutputStream(calculatorGraph, "multi_face_landmarks");
var result = await multiFaceLandmarksStream.WaitNextAsync(); // won't block the main thread
if (result.packet != null)
{
    var multiFaceLandmarks = packet.GetProtoList(NormalizedLandmarkList.Parser);
    // ...
}

For usage, refer to the sample app code, etc.

Background

Here is another somewhat excuse-like background.

Packet<T> was created to somehow introduce type safety into Packet. For example, without type checking, you can call an API that retrieves a string when the data inside is an int (and this API call causes a crash on Windows). However, there were the following issues:

• No guarantee of consistency between C++ and C# types, so incorrect APIs could be still called
• Increased redundancy in specifying type parameters
• Some API calls Reflection API internally, and their calls are heavy

Especially the last issue was significant, and as the Task API required calling that API frequently, I migrated to a non-generic API.

Miscellaneous

Finally, there is no impact on the plugin itself, but the sample app's configuration method has been changed.
Previously, settings were in the Bootstrap Component but have now moved to AppSetting.asset.
assetLoaderType defaults to Local, and if not changed, it will result in an error when executed on a real device, so please be careful .

It's been lengthy, but those are the major changes.
Since there are many changes, it would be helpful if you could report any bugs you find.

The documentation is not up to date, but I will address it when the motivation arises.
For those who want to use the Task API, I have tried to write documentation comments, so please refer to them.

P.S. The absence of v0.13.0 is simply because it has been a while since the last release, and I made a mistake in the release process, so please don't worry about it.

Other CHANGELOG

Features

• build Image/ImageFrame from Texture2D (#1083) (bcce0e9)
• sample: drop non-blocking sync mode (#1110) (589b335)
• sample: remove Start Scene (#1017) (059b0b2)

Bug Fixes

• proper autofit image if camera is rotated (#1043) (2d2863e) (by @Dender46)

See CHANGELOG for more details.