Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. We have updated the language to the Editor Terms based on feedback from our employees and community. Learn more.
    Dismiss Notice
New Forum User Notice Update to the Unity Editor Software Terms

Question Face Landmark Detection

Discussion in 'AR' started by marsinator, Dec 13, 2022.

  1. marsinator

    marsinator

    Joined:
    Apr 15, 2022
    Posts:
    15
    Can someone help me with detecting face landmarks in Unity AR?
     
    marsinator, Dec 13, 2022
    #1

  2. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    818
    andyb-unity, Dec 13, 2022
    #2

  3. marsinator

    marsinator

    Joined:
    Apr 15, 2022
    Posts:
    15
    No problem with detecting faces, I struggle to detect face landmarks
     
    marsinator, Dec 13, 2022
    #3

  4. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    818
    Keep reading the link I posted above?

     
    andyb-unity, Dec 13, 2022
    #4

  5. marsinator

    marsinator

    Joined:
    Apr 15, 2022
    Posts:
    15
    I read this, but i want to detect the mouth region. More specific the landmarks that are in this region in order to determine if the mouth is open or closed and so on..
     
    marsinator, Dec 13, 2022
    #5

  6. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    818
    These samples demonstrate all the face tracking techniques available in ARCore and ARKit. On the ARKit side, mouth tracking is straightforward using Blend Shapes.

    On ARCore, you would have to do some processing yourself based on the vertices of the Face Mesh, as ARCore's Augmented Faces feature doesn't specifically handle mouth tracking for you: https://developers.google.com/ar/develop/augmented-faces
     
    andyb-unity, Dec 13, 2022
    #6

  7. marsinator

    marsinator

    Joined:
    Apr 15, 2022
    Posts:
    15
    I need to do it on Android so i have to use ARCore. I got the OpenCV and the DlibFaceLandmarkDetector Assets but have problems to get it to work.

    But about the face mesh. Does every vertex have an specific index? That way i would just have to loop through the vertices list.
    Do you know which vertices specify the mouth region?
     
    marsinator, Dec 13, 2022
    #7

  8. andyb-unity

    andyb-unity

    Unity Technologies

    Joined:
    Feb 10, 2022
    Posts:
    818
    I am not certain, but I would wager that the vertices are always reported in the same order. I have not done this experiment but it seems possible that you could identify the mouth vertices.

    I definitely can't help you with OpenCV, but good luck on your app!
     
    andyb-unity, Dec 14, 2022
    #8

  9. marsinator

    marsinator

    Joined:
    Apr 15, 2022
    Posts:
    15
    Do you know how to draw for example a circle at a specific vertex and display this on the device?
     
    marsinator, Dec 14, 2022
    #9