Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only. On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live. Read our full announcement for more information and let us know if you have any questions.
    Dismiss Notice
  2. Dismiss Notice

Question Narrative Game - Lipsync issue

Discussion in 'Timeline' started by deadlycrow, Feb 18, 2023.

  1. deadlycrow


    Feb 10, 2014
    So we encounter the next problem:

    We have a character thats just standing, smoking a cigarette while looping on an idle animation, also we have a system that plays dialogues and make the character move his mouth with a function that checks the dB of the audio input of the dialogue and then moves a float parameter of a blendtree inside of the mecanim of the character to make him talk.
    The problem is, Timeline is overriding the default mecanim animation, so the movement of the mouth of the character is absolutely lost. So far ive only figured out 2 ways of solving this issue:

    1) Forget about timeline and do it old school just mecanim animations, with animation functions to control the sounds, effects, etc. (NOT IDEAL)

    2) Forget about mecanim animator and do everything in Timeline by hand, meaning, i will have to create a different timeline file for each dialogue option available for the character, assigning by hand each audioclip and controlling also by hand whenever the character opens his mouth to play an override clip of the mouth lipsync animation (its a generic animation of the character moving his mouth, which in our previous method guided by the audiofile dB amplitude kinda faked more or less correctly the illusion of the character talking, i know, its not very decent, but it is what we can afford as an indie studio).

    Now, solution number 1 is not ideal at all because its actually the only serious option for production to create a huge bunch of cutscenes that require characters being animated, sounds, VFXs, hiding objects, camera transitions, etc.
    But option number 2 is even less ideal, because we already had a system in place that could automatically do that job and it was flawless. With option 2 the amount of workload ahead is totally out of proportion for our deadlines and our budget.

    Short version: Any ideas? how to solve the problem of lets say a character hammering nails in a construction site (a timeline scene, with animations, sounds, vfx, etc), BUT having his face dynamically animated depending on dialogue lines (with an additional face animation to play).
    Last edited: Feb 18, 2023
  2. akent99


    Jan 14, 2018
    What I did for the mouth was create 2 components that worked together. I don't think the same component can be controlled by two things - Animator will stomp on your changes. So I created one component that I control with animation clips to control the mouth positions via a Timeline (I actually did facial expressions which included the mouth), then created a second component for adjusting mouth shapes for talking. Behind the scenes the two components worked together (one read from the other one) and merged the data to do the final result which is what adjusted the blendshape strengths. That is, I wrote code to mix two sets of animated properties to overcome the problem.

    I did find I had to be careful - combining a smile and talking did not always work well, so I added in weights for the two components rather than just add them.