Feature Request ARkit facetracking with truedepth?

hello @timbokoppers,

MARS Supports several versions of ARFoundation https://docs.unity3d.com/Packages/com.unity.mars-ar-foundation-providers@1.0/manual/index.html

So yes, you can use FaceTracking from ARKit in MARS by using ARFoundation on your MARS project.

 

Hello @timbokoppers,

If you are talking about face tracking in the editor, generally, then yes MARS Provides that; if you are talking specifically about face mesh simulation in the editor then no, MARS doesnt have that.

 

Ah alright, Blendshape info is something we do not simulate on the editor (and is also not recorded). But you ultimately can use Landmarks https://docs.unity3d.com/Packages/com.unity.mars@1.0/manual/Landmarks.html

 

Hi there!

I wanted to build on @jmunozarUTech's response (thanks!). Blendshapes in particular are a tricky subject in today's AR landscape. As far as I know, ARKit is the only platform to use that particular set of blendshapes. As a result, if you author content to use them it will likely only work on Apple devices with TrueDepth cameras. It won't, for example, work on Android devices or other potential platforms which support face tracking with head pose estimation, but do not provide the same blendshapes as ARKit. It's feasible to try and "translate" blendshape values if they are similar enough, but things break down quickly as the different representations facial expressions diverge.

Furthermore, as a way of responding to facial expressions, blendshapes can be a bit limiting. They are not all directly mapped to expressions (like smile, frown, wink, etc.) although they can be used this way. They are very good for deforming a mesh that has been authored specifically with the platform's configuration in mind, but not very good at providing general-purpose anchor points for locations on the face. For example, blendshapes do not help me place a mustache on the user's upper lip, because I still need the face mesh to locate the "staring point" for the mouth open/close expression.

In MARS, we expose a set of landmarks and expressions which we intend to be a "minimum subset" that all platforms should support. In our ARFoundationFaceTrackingProvider, we utilize the platform-provided mesh to produce these landmarks and expression values based on the position of specific triangles that we know to correspond to facial features. If you want to make googly eyes or fake eyebrows, landmarks are a great way to drag and drop content onto specific facial features. You can read more about this in our documentation:

https://docs.unity3d.com/Packages/c...acking.html#placing-digital-content-on-a-face

As a final note, these landmarks are included in the face recordings that come with MARS, and can therefore be simulated in the Editor.

For expression data on Android, we use a set of position ranges for these landmarks to do a basic estimation, however this approach does not work for all faces and we do not recommend its use for production apps. On iOS with truedepth, we actually use blendshape data to calculate the expressions, which is a bit more reliable. The expression system is best suited to future integrations with SDKs that are specifically intended to provide expression/emotion information.

If your intent is to create an "animoji" type character based on ARKit blendshapes, this is unfortunately not a workflow that we support in MARS. You are able to access the blendshape data through the AR Foundation session that we use to implement face tracking in MARS, but we do not expose blendshapes through our MARS data types at this time.

We released a code sample back in 2018 which may help you iterate on characters using ARKit blend shapes. https://github.com/Unity-Technologies/facial-ar-remote

Hope this helps!

 

Correct. It is based on the old ARKit plugin. It shouldn't be too hard to update to the latest AR Foundation version, but I have not tried this. I will try to take some time soon to take a stab at this and report back.

 

OK! That was easier than I thought it would be. I've created an `update` branch on that repo that sets it up as a UPM package and integrates with AR Foundation.

I've also set up an integration project for easy set up here https://github.com/Unity-Technologies/Facial-AR-Remote-Project

 

MARS has built in face recordings (with video and tracking), hope that is helpful.

 

> Could your explain maybe what really change with the previous version?

All that's changed in this new version was to upgrade to a newer version of Unity (2019.4) and use AR Foundation. The networking code is mostly unchanged, but I fixed a bug where you could accidentally leave out certain blendshapes if the settings weren't configured correctly.