Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

automatic lip sync animation in unity3D

Discussion in 'Animation' started by Akshea, Mar 12, 2014.

  1. Akshea


    Nov 6, 2013
    I have an audio file and a text file for the audio. And i have generated the phonemes for the audio file as the text(.txt) file. With help of this phoneme text file., how could i animate my 3d character model. Is there any tool that takes the audio and phoneme file as input and produce the corresponding lipsync animation for the character in unity3D. How could i acheive this. Please help me. I am newbie to unity3D. Are is there a way to do lipsync programmatically?
    Thanks in advance.
  2. TonyLi


    Apr 10, 2012
    Since you have the audio, use an audio analysis animation tool like FaceFX or iClone. There are also second-tier products like Magpie and Dark Voices. You'll need a model that animates the basic phonemes. If you've already generated the phonemes in time-sync with the audio in the text file you write about, then you might be able to roll your own solution that animates the phonemes in the file in sequence. But tools like FaceFX are nice because they inject other facial animations like random blinking that makes the effect more realistic.

    If you didn't have audio, you could look at FaceFX's text-to-speech.

    Another option is FacePlus, which records your face in realtime.
  3. Crazy-Minnow-Studio


    Mar 22, 2014
    We recently released a 2D/3D Sprite/BlendShape based lip sync and eye movement system called SALSA with RandomEyes. It processes audio in real time and does not require manual phoneme mapping. It handles lip sync, random or programmatic eye control, eye tracking, and offers an event system for advanced implementations. Our soon to be released 1.1 update will add unlimited BlendShape-based emotional expressions to the mix for 3D characters.
    Last edited: May 5, 2014
  4. RichardKain


    Oct 1, 2012
    I'm working on something similar to what's being requested. Unfortunately, it's not quite done yet so I can't really recommend it. I will post more about it when it is available.