Search Unity

automatic lip sync animation in unity3D

Discussion in 'Animation' started by Akshea, Mar 12, 2014.

  1. Akshea

    Akshea

    Joined:
    Nov 6, 2013
    Posts:
    10
    Hi,
    I have an audio file and a text file for the audio. And i have generated the phonemes for the audio file as the text(.txt) file. With help of this phoneme text file., how could i animate my 3d character model. Is there any tool that takes the audio and phoneme file as input and produce the corresponding lipsync animation for the character in unity3D. How could i acheive this. Please help me. I am newbie to unity3D. Are is there a way to do lipsync programmatically?
    Thanks in advance.
     
  2. TonyLi

    TonyLi

    Joined:
    Apr 10, 2012
    Posts:
    12,706
    Since you have the audio, use an audio analysis animation tool like FaceFX or iClone. There are also second-tier products like Magpie and Dark Voices. You'll need a model that animates the basic phonemes. If you've already generated the phonemes in time-sync with the audio in the text file you write about, then you might be able to roll your own solution that animates the phonemes in the file in sequence. But tools like FaceFX are nice because they inject other facial animations like random blinking that makes the effect more realistic.

    If you didn't have audio, you could look at FaceFX's text-to-speech.

    Another option is FacePlus, which records your face in realtime.
     
  3. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    1,399
    We recently released a 2D/3D Sprite/BlendShape based lip sync and eye movement system called SALSA with RandomEyes. It processes audio in real time and does not require manual phoneme mapping. It handles lip sync, random or programmatic eye control, eye tracking, and offers an event system for advanced implementations. Our soon to be released 1.1 update will add unlimited BlendShape-based emotional expressions to the mix for 3D characters.
    http://forum.unity3d.com/threads/242135-SALSA-with-RandomEyes-1-0-0-RELEASED
     
    Last edited: May 5, 2014
  4. RichardKain

    RichardKain

    Joined:
    Oct 1, 2012
    Posts:
    1,261
    I'm working on something similar to what's being requested. Unfortunately, it's not quite done yet so I can't really recommend it. I will post more about it when it is available.