Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Question Face Capture: Jaw bone movement?

Discussion in 'Virtual Production' started by dgoyette, Oct 20, 2022.

  1. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    3,996
    I've just tried out Face Capture on a model I have. The model has standard blend shapes, which work well with face capture. However, my model also has a Jaw bone, which needs to be moved up/down as the mouth opens and closes. I don't see a way to control that behavior with Face Capture.

    Is there some approach to doing so? Or is jaw movement something that might come in a future update?
     
  2. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    516
    I don't have a answer for the questions, but I saw use of bones in some Blender characters people created recently too. They used bones near each eyebrow, allowing bone animation to do some quite expressive face contortions.

    Seems pretty hard to generalize though. Blendshapes really became normalized when the Apple ARKit set of shapes came out. It provided a degree of standardization, which I don't think is there with facial bones at this stage.
     
  3. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    3,996
    It seems like maybe what I should be doing is get the bones to move via blend shapes as well. I've just been looking into this. It seems that I can associate bone transform changes to blend shape value changes. So, for my "Mouth_Open" blend shape, I can also make the jaw move. So, I'll be looking into whether that's the solution here.
     
  4. ScottSewellUnity

    ScottSewellUnity

    Unity Technologies

    Joined:
    Jan 29, 2020
    Posts:
    20
    @dgoyette Yes, it is easiest to create a script that moves the bones based on the blend shapes of the mesh. Ideally the script should apply the pose between LateUpdate and rendering to avoid any latency.
     
    dgoyette likes this.
  5. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    3,996
    Thanks. I put together a little script to do this. There were a few blend shapes that could potentially move the jaw bone, so the script supports multiple blend shapes each having some impact on the bone. Just including it here in case it's useful to anyone.

    Code (CSharp):
    1. using System;
    2. using System.Collections.Generic;
    3. using System.Linq;
    4. using UnityEngine;
    5.  
    6. [ExecuteInEditMode]
    7. public class LeoBoneController : MonoBehaviour
    8. {
    9.     public SkinnedMeshRenderer BodyRenderer;
    10.     public Transform JawBone;
    11.     public List<BonePositionBlender> JawBlenders;
    12.     public Vector3 JawBoneOffsetLowerLimit;
    13.     public Vector3 JawBoneOffsetUpperLimit;
    14.  
    15.     private Dictionary<string, int> _blendShapeIndices;
    16.  
    17.     private Vector3 _initialJawLocalPosition;
    18.     private bool _jawPositionInitialized;
    19.  
    20.     private void LateUpdate()
    21.     {
    22.         if (!_jawPositionInitialized)
    23.         {
    24.             InitializeJawPosition();
    25.         }
    26.         else
    27.         {
    28.             var stuff = JawBlenders.Select(jb => BodyRenderer.GetBlendShapeWeight(_blendShapeIndices[jb.BlendShapeName]) / 100 * jb.FullPositionOffset).ToList();
    29.             Vector3 jawOffset = JawBlenders.Select(jb => BodyRenderer.GetBlendShapeWeight(_blendShapeIndices[jb.BlendShapeName]) / 100 * jb.FullPositionOffset)
    30.                 .Sum()
    31.                 .Clamp(JawBoneOffsetLowerLimit, JawBoneOffsetUpperLimit);
    32.  
    33.             JawBone.transform.localPosition = _initialJawLocalPosition + jawOffset;
    34.         }
    35.     }
    36.  
    37.     private void InitializeJawPosition()
    38.     {
    39.         if (BodyRenderer == null || JawBone == null)
    40.         {
    41.             return;
    42.         }
    43.  
    44.         _blendShapeIndices = new Dictionary<string, int>();
    45.         for (int blendShapeIndex = 0; blendShapeIndex < BodyRenderer.sharedMesh.blendShapeCount; blendShapeIndex++)
    46.         {
    47.             _blendShapeIndices[BodyRenderer.sharedMesh.GetBlendShapeName(blendShapeIndex)] = blendShapeIndex;
    48.         }
    49.  
    50.         _initialJawLocalPosition = JawBone.localPosition;
    51.         _jawPositionInitialized = true;
    52.     }
    53.  
    54.     [Serializable]
    55.     public class BonePositionBlender
    56.     {
    57.         public string BlendShapeName;
    58.         public Vector3 FullPositionOffset;
    59.  
    60.     }
    61. }
    62.  
    63. public static class Vector3Util
    64. {
    65.     public static Vector3 Sum(this IEnumerable<Vector3> self)
    66.     {
    67.         var s = self.ToList();
    68.         Vector3 sum = Vector3.zero;
    69.         foreach (var v3 in self)
    70.         {
    71.             sum += v3;
    72.         }
    73.  
    74.         return sum;
    75.     }
    76.  
    77.     public static Vector3 Clamp(this Vector3 self, Vector3 low, Vector3 high)
    78.     {
    79.         return new Vector3(Mathf.Clamp(self.x, low.x, high.x),
    80.             Mathf.Clamp(self.y, low.y, high.y),
    81.             Mathf.Clamp(self.z, low.z, high.z));
    82.     }
    83. }
    84.  
     
    markbeiline and marc_tanenbaum like this.