Search Unity

Question MARS eye movement tracking

Discussion in 'Unity MARS' started by xhkong, Dec 3, 2020.

  1. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    Is there a way to track eye movement in MARS? I'm trying to replace user's eyes with 3D eyes object, and have them move the same way user move eyes, is this doable in MARS?
     
  2. mtschoen

    mtschoen

    Unity Technologies

    Joined:
    Aug 16, 2016
    Posts:
    194
    Hi there!

    Yes, this is possible with MARS. You use the Face Mask template, and attach your content to the `LeftEye` and `RightEye` landmark transforms. You can read more about this workflow here https://docs.unity3d.com/Packages/c...acking.html#placing-digital-content-on-a-face

    Please bear in mind that platform support for eye tracking is somewhat limited. ARCore does not provide rotation data for the eyes, and we do not currently apply blendshape data from ARKit to our landmarks, which means that the eyes on iOS will also not rotate. It is possible to create a custom face tracking provider that applies rotation to the eye landmarks, but this would require some coding effort on your part.

    Hope this helps!
     
  3. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    Thanks for the quick reply. Eye-rotation is what I'm trying to get. Can you elaborate a bit more on now can I get access to the eye-rotation data from iOS in MARS?
     
  4. mtschoen

    mtschoen

    Unity Technologies

    Joined:
    Aug 16, 2016
    Posts:
    194
    You're welcome. :)
    The next reply might not be so quick. I think I can get this working with a MonoBehaviour that subscribes to the `OnMatchUpdate` event from a `MatchAction` on a face proxy, but it'll take some time to test it out and ensure it works. Inside of this callback, you can get access to the face subsytem from the active ARFaceManager, cast it to an ARKit Face Subsystem, and then get the blendshape values of the face you matched against. There are 4 blendshape values per-eye which you can use to drive the x/y rotation of the eye.

    To be honest, the best way for this to happen is for us to update the MARS AR Foundation Providers package to incorporate this into the MARS landmarks directly. We will try to incorporate this into a future release. In the meantime, I'll see if I can get you a workaround that you can drop into your project once I get it working.
     
  5. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    Thanks so much! That would be great! I'm looking forward to this. Any ETA I can expect on this?