Bug Face Expression Action for eye open/close not working

Hi there

As the title explains, I am unable to get either of the eye's Close or Open events to trigger when running on an Android device.

It works fine when simulated in the editor's Device View, but not after building to the device.

The smile engaged/disengaged Expression Actions work however.

My project can be located here:
https://github.com/Petrie65/unity-mars-build-fail
It is an empty project with MARS that swaps out game objects on the eyes/month Landmarks based on Face Expression Actions.

Any advice would be appreciated. Thanks

 

Hi @jmunozarUTech

Thanks for getting back to me. I have done as you suggested. I also had to remove mars-ar-foundation-providers from the package manager as there were now duplicate scripts. Unfortunately after building these changes, it seems that MARS is not working at all on the device.

I made a script listen to MARSFaceManager FaceUpdated events to see what values MARS has for the facial expressions. I can manipulate all the values with my face on-device, except for the eyes - they are frozen on 0. But when I run it in the Editor then it is clearly detecting the values.

 

Ok, thank you I was able to move the package so that I could modify the

ARCoreFacialExpressionSettings

and successfully build it to device.

I have tried a bunch of different configurations, but I am still unable to get the

LeftEyeClose 

or

RightEyeClose 

to reflect anything other than 0.

Something else worth mentioning; in order to get the Android build to succeed, I had to remove the

Unity Content Manager

and

Unity MARS Companion Core

preview packages from the project. I don't think this should affect the core functionality though (for reference https://forum.unity.com/threads/il2...tandard-version-2-0-0-0.1041868/#post-7331191)

 

Hi @jmunozarUTech

I've been trying to get this working, but unfortunately made no progress. Do you perhaps have some advice on how I could further debug this?

I am really baffled by what could cause this. Especially since the rest of the expressions are working. I'd imagine the method used for detecting eyes closed/open would use on the same tech stack that allows recognition of the rest of the expressions.

Do you maybe know if other MARS users have successfully built this to an Android device? I would think so because it seems like a pretty common use case.

On a related note, I also tried deploying to an iPhone 6S Plus and could only see the rear-facing camera with no face-tracking functionality. I believe that is because it doesn't have a forward-facing depth camera, so it would only work on an iPhone X or newer, is that right?

 

Thank you for looking into it.

Today we deployed to iPhone X and it worked, including the eye expressions. The same build is still not working for the iPhone 6S Plus though, I suppose some older-generation devices might not be able to run it.

We also tried 2 different Android platforms - so in total, we have tried a device from Huawei, Samsung, and Xiaomi. All with the same issue.

For more context, the relevant versions we are using are the following:

• Unity 2020.3.6f1 (LTF)
• AR Foundation, AR Subsystems, ARCore, ARKit, AR Face Tracking - 4.1.7
• Mars, Mars AR Foundation Providers, Mars Nav Mesh - 1.3.1

 

Thanks @amydigiov

I'll be looking forward to the 1.4 release.

 

May I know the roadmap when 1.4 will be release to fix on the captioned issue? Thank you!

 

Hello,
I've updated to 1.4.1 and can confirm that files such as "ARCoreFacialExpressionSettings" are in a more accessible location to modify and that eyes closed detection is "working" for Android.
It is finicky, however. If the user's device is tilted back or forward, it alters the values in which it thinks the eyes are closed. Currently looking into using the device orientation to offset this value.

If anyone who finds this thread is wondering how i'm finding the eye's closed float value, here's some code to get you started:

Code (CSharp):

1. public class FacialFeaturesReader : MonoBehaviour, IUsesFaceTracking, IUsesCameraOffset
2.     {
3.         [SerializeField] private TextMeshProUGUI dbgEyesText = null;
4.  
5.         IProvidesFaceTracking IFunctionalitySubscriber<IProvidesFaceTracking>.provider { get; set; }
6.         IProvidesCameraOffset IFunctionalitySubscriber<IProvidesCameraOffset>.provider { get; set; }
7.  
8.         private void OnEnable()
9.         {
10.             this.SubscribeFaceAdded(OnFaceAdded);
11.             this.SubscribeFaceUpdated(OnFaceUpdated);
12.             this.SubscribeFaceRemoved(OnFaceRemoved);
13.         }
14.  
15.         private void OnDisable()
16.         {
17.             this.UnsubscribeFaceAdded(OnFaceAdded);
18.             this.UnsubscribeFaceUpdated(OnFaceUpdated);
19.             this.UnsubscribeFaceRemoved(OnFaceRemoved);
20.         }
21.  
22.         private void OnFaceAdded(IMRFace face)
23.         {
24.             Debug.Log(face.Expressions.Count + " expression found for facial tracker");
25.         }
26.  
27.         void OnFaceUpdated(IMRFace face)
28.         {
29.              if (dbgEyesText != null)
30.              {
31.                  dbgEyesText.text = $"Left Eye: {face.Expressions[MRFaceExpression.LeftEyeClose]}" +
32.                      $"\n\nRight Eye: {face.Expressions[MRFaceExpression.RightEyeClose]}";
33.              }
34.         }
35.  
36.         private void OnFaceRemoved(IMRFace face)
37.         {
38.  
39.         }
40.     }