Other Mars Face tracking questions

Hello @jiraphatK,

Short answer is No, we dont support blenshapes; the large answer (from @mtschoen 's own words) from this Thread:
https://forum.unity.com/threads/arkit-facetracking-with-truedepth.903548/#post-6523851


Blendshapes in particular are a tricky subject in today's AR landscape. As far as I know, ARKit is the only platform to use that particular set of blendshapes. As a result, if you author content to use them it will likely only work on Apple devices with TrueDepth cameras. It won't, for example, work on Android devices or other potential platforms which support face tracking with head pose estimation, but do not provide the same blendshapes as ARKit. It's feasible to try and "translate" blendshape values if they are similar enough, but things break down quickly as the different representations facial expressions diverge.

Furthermore, as a way of responding to facial expressions, blendshapes can be a bit limiting. They are not all directly mapped to expressions (like smile, frown, wink, etc.) although they can be used this way. They are very good for deforming a mesh that has been authored specifically with the platform's configuration in mind, but not very good at providing general-purpose anchor points for locations on the face. For example, blendshapes do not help me place a mustache on the user's upper lip, because I still need the face mesh to locate the "staring point" for the mouth open/close expression.

In MARS, we expose a set of landmarks and expressions which we intend to be a "minimum subset" that all platforms should support. In our ARFoundationFaceTrackingProvider, we utilize the platform-provided mesh to produce these landmarks and expression values based on the position of specific triangles that we know to correspond to facial features. If you want to make googly eyes or fake eyebrows, landmarks are a great way to drag and drop content onto specific facial features. You can read more about this in our documentation:

https://docs.unity3d.com/Packages/c...acking.html#placing-digital-content-on-a-face

As a final note, these landmarks are included in the face recordings that come with MARS, and can therefore be simulated in the Editor.

For expression data on Android, we use a set of position ranges for these landmarks to do a basic estimation, however this approach does not work for all faces and we do not recommend its use for production apps. On iOS with truedepth, we actually use blendshape data to calculate the expressions, which is a bit more reliable. The expression system is best suited to future integrations with SDKs that are specifically intended to provide expression/emotion information.

If your intent is to create an "animoji" type character based on ARKit blendshapes, this is unfortunately not a workflow that we support in MARS. You are able to access the blendshape data through the AR Foundation session that we use to implement face tracking in MARS, but we do not expose blendshapes through our MARS data types at this time.

We released a code sample back in 2018 which may help you iterate on characters using ARKit blend shapes. https://github.com/Unity-Technologies/facial-ar-remote

Hope this helps!

 

hello @jiraphatK,

For the easy to follow guide on face landmarks you can check our guide in https://docs.unity3d.com/Packages/com.unity.mars@1.2/manual/FaceTracking.html

For a more in depth guide about landmarks and how to create your own custom landmarks you can check https://docs.unity3d.com/Packages/com.unity.mars@1.2/manual/Landmarks.html

With regards to expressions you could check the FaceExpressionAction and there we have a set of basic expressions.

Let us know if you have any other questions.