Search Unity

Fuse motion capture and eye tracking into joint HumanPose

Discussion in 'Animation' started by Peter-K, May 9, 2018.

  1. Peter-K

    Peter-K

    Joined:
    Apr 12, 2017
    Posts:
    3
    I want to retarget both tracked eye movements and body motion to one humanoid avatar.

    The motion capture system's plugin provides a stream of updates to a HumanPose which I merely copy to the Player Prefab via HumanPoseHandler.SetHumanPose within my retargeting script's Update method.

    The eye tracking device provides the current fixation point within the scene. To use this position, I decided to add an empty AnimatorController with IK pass enabled. In OnAnimatorIK I use the eye tracker's fixation position as the Animator LookAtPosition with the LookAtWeights set to only move the eyes. This works like a charm, but unfortunately completely overwrites the HumanPose set previously in the Update method resulting in the character being in the fetus-like pose just like when setting up a humanoid character in the Editor.

    Is there any way to use Mecanim's built-in IK system without overwriting the motion capture input?
     
    Last edited: Jul 23, 2018
  2. theANMATOR2b

    theANMATOR2b

    Joined:
    Jul 12, 2014
    Posts:
    7,790
    Does adding this to an animation layer - set to additive - solve the overwrite problem?
    Base layer mocap, new additive layer eye tracking data.
     
  3. Peter-K

    Peter-K

    Joined:
    Apr 12, 2017
    Posts:
    3
    Whereas body motion is set in the Update() method by copying muscle values from an invisible character that receives the motion capture data stream, the Animator Controller is only used to set the avatar LookAtPosition in OnAnimatorIK(). Neither using AnimationBlendMode to "Additve" nor using an AvatarMask prevents overriding the animation.

    Previously, I manually converted the gaze vectors to muscle values which felt like an overly bloated solution. I've started experimenting with the AimConstraint recently added to Unity and might get away with that.
    I'll keep you updated, thanks for the suggestion!
     
    theANMATOR2b likes this.