AR face tracking for both iOS and Android

I'm looking for good solutions or ideas for AR face tracking in Unity for both iOS and Android. My ideal solution would be one API for both platforms, but could have separate solutions if needed.

I need to be able to place 3D models (like hats, sunglasses, necklace, etc..) on a user's head, face, and neck, with masking, while tracking position/rotation, and running at a decent framerate. My client is basically asking for Snapchat AR, at least the model placement part, dont really need the face lens mesh part, except for maybe masking. I believe Snapchat's solution is a custom low level version of OpenCV.

From my quick research, and correct me if I'm wrong or missing something, here are my options:

iOS:

• Vuforia does not currently support face tracking.
  
• Use ARKit plugin, but face tracking only supported on iPhone X or greater with a truedepth camera. Examples: Face tracking and masking example
  
• Use OpenCV plugin, wider device support, but may have low performance/framerate unless paired with a low level camera solution, like NatCam. Examples: Face tracking example, Face masking example

Android

• Vuforia does not currently support face tracking.
• ARCore does not currently support face tracking.
• Use OpenCV plugin

Anyone else seen solutions for this in Unity? What is the best way to achieve face tracking in Unity?

 

Obvious open-source alternatives include:

- DLib (C++ lib, or you can buy a unity plugin that bridges to it.)
- Firebase MLKit face recognition. Has ObjC/Java API which would be relatively trivial to bridge to Unity.

The performance both of these options (both in terms of framerate and detection robustness to e.g. head rotation) is nowhere near parity with ARKit, Snapchat lenses, etc.

 

I've tried the demo from - https://www.xzimg.com - on a non-iPhone X and a mac and it looks quite good.

On the asset store I've seen, but haven't tried - https://assetstore.unity.com/packages/tools/animation/face-detection-pro-114410

If you find other options, please let us know.

 

Prolly mlkit arkit is your best bet. Use the LWR/arfoundation technique so mlkit can pull from camera texture

https://github.com/Unity-Technologies/arfoundation-samples/tree/lwrp_support/Assets/LWRPSupport

You need a machine learning model basically

 

I would recommend you check this step-by-step guide to building an AR app for iOS. The author in the simple terms explains how to get started with the ARKiT SDK and guides you through the whole process of creating an AR app with all necessary code snippets.

 

I am trying to use the xzimg one as well, but with no success.. would you mind sharing some tips with me? basically not even by just opening their project sample I can manage to open the device webcam :-(

 

Since ARFoundation 3.0 is out, is it now possible to use face tracking on Android with ARF3.0 ?

 

A few of my colleagues were having a similar discussion and they mentioned using AR Foundation and point clouds or planar surface detection to configure similar functions on a 3d image. I think that's something you can look into, especially since AR in mobile apps is trending in the industry. I think specifically for Android, that could work. Good luck!

 

Hello dustin_red .

If you are still interested in such a solution, take a look at Banuba's Unity AR SDK which works for both iOS and Android operating systems, supports the face tracking functionality on the Unity engine apps and allows you to implement virtual try-ons of the accessories you've mentioned. In addition to the latter, there is also a possibility to use their face filters as well as avatars and emojis if you're ever going to need one.

Best,

Alexei.

 

I would recommend using AR Foundation.
It wraps native ARKit and ARCore solutions so you'll get native performance and tracking quality.

 

All the above answers are indeed useful and good to take into the consideration for developing AR face tracking feature of android and iOS application, I have been into the tech market for 5+ years and I have worked with numerous mobile/web and software application design and development companies who taught me the tech vulnerabilities of various techs, such as AR/VR, IoT, AI, and Chatbot.

Although, an ideal case would be hiring leading iOS app developers for iOS mobile application AR tech integration and for android, finding an authentic android app development firm that could leverage AR tech sufficiently in the android application would be the right decision.

Furthermore, after researching a bit I found some resources for your development approach, and that can play an imperative role in your AR face tracking feature development for android and iOS.

1). https://developers.google.com/ar/develop/unity/augmented-faces
2). https://developer.apple.com/documentation/arkit/tracking_and_visualizing_faces
3). https://www.banuba.com/blog/how-to-choose-face-tracking-sdk-for-android-ios
4). https://www.raywenderlich.com/5491-ar-face-tracking-tutorial-for-ios-getting-started

Let me know how these blogs you use in your application development, and if you any further questions similar to the above one, I would love to help.