Search Unity

Modifying face mesh from Facekit

Discussion in 'AR' started by DigitalBeach, Mar 18, 2019.

  1. DigitalBeach

    DigitalBeach

    Joined:
    Jan 17, 2015
    Posts:
    37
    Has anyone been able to manipulate the face mesh via shader from the ARKit Face system?

    By default Unity puts the video on the background and just overlays the mesh. To distort the face mesh we would need to use the camera feed as a texture and calculate the UVs based on the camera projection. Then could distort the mesh in the shader.

    Though I am not sure exactly how to set up the shader for the UV projection.
     
    DigitalBeach, Mar 18, 2019
    #1

  2. DigitalBeach

    DigitalBeach

    Joined:
    Jan 17, 2015
    Posts:
    37
    Made some progress on this. By default the video is set up on the YUVMaterial and is handled by the YUVShader. You can set it up so the material isn't applied to the camera and instead set it on your face mesh. Then customize the shader to do the projection.

    I don't have the projection math right yet but should be doable.
     
    DigitalBeach, Mar 18, 2019
    #2

  3. DigitalBeach

    DigitalBeach

    Joined:
    Jan 17, 2015
    Posts:
    37
    Got it really close by modifying the YUV_Shader to do a projection. The scale in X is slightly off so I suspect the screen aspect ration is not being accounted for give I am not using the DisplayTransform in this.

    Attached is shader
     

    Attached Files:

    DigitalBeach, Mar 19, 2019
    #3

  4. waldgeist

    waldgeist

    Joined:
    May 6, 2017
    Posts:
    388
    Sounds good! Did you optimize this further?
     
    waldgeist, Nov 24, 2021
    #4