Free tool for 2D Lip Sync Animation

Hey there,

I created a simple tool for making 2D characters animate their mouths to audio. It consists of a custom component that sits on a Sprite Renderer and switches the sprite and a custom Timeline Track to control the sprite.

I leverage the fantastic Rhubarb tool to auto-create lip sync from a given audio file.

Here's a video of the process, skip to 1.25 for the result.

https://github.com/gilescoope/rhubarb-timeline

Let me know what you think!

 

How do you handle delays in audio playback for mobile platforms?

I am very much into Lip-Synced animations, and Rhubarb does seem like a pretty good tool. The phoneme timing looks to be a bit limited, but for a smaller-scale project, it is a huge time saver. (hence it's use in the pixel-art heavy Thimbleweed Park) Having a Unity plug-in for Rhubarb makes a lot of sense.

 

Hey there, I didn't realise that this was an issue. In any case the tool simply provides a Playable Track in Timeline in which the timings can be tweaked as you like.

So any timing issues you have with this on mobile would be the same as for any Timeline animation with audio. What would one usually do?

Yeah the results aren't perfect, but are a lot better than the other Unity tools I've come across. It can also be used as a good starting point to tweak the final phoneme animations from.

 

I actually released one of those other Unity tools. Sadly, I wasn't able to maintain it, and it dropped from the Asset Store. I wasn't charging anyway, so it wasn't that big a deal. I'm always interested to see new solutions crop up. And if any of my past experience can help, so much the better.

The issue with mobile comes with the decoding of compressed audio sources. If your speech audio is stored as an uncompressed WAV file, there is no issue. But the vast majority of users aren't going to keep speech files uncompressed, especially for distribution on mobile. If your file is a compressed MP3 or OGG file, the decoding on mobile causes a slight delay when you first start playing it. This delay also won't be consistent, different devices might have a greater or lesser delay based on their processing power. If you start playing an animation driven by the engine at the same time as a compressed audio file, the animation will run normally while the audio gets the slight delay, throwing the two out of sync and causing a noticeable discrepancy.

On desktop you usually don't see this, as even average or low-power desktop and laptop machines have enough processing juice to avoid the audio decompression delay. It is a technical hurdle that has to be addressed, though, and don't be surprised if users start to ask you about it.

I haven't played around with the Timeline much yet, but it seems a very appropriate place to put a lip-synced animation. Most uses of lip-syncing are in pre-scripted animation sequences. The solution I used stored the data in generated Animation Clips. I rigged it up to use either Shape Keys for 3D models or Sprites for 2D animations.

You might try looking at the possibility of playing the animations back dynamically based on the current position of the playing audio file. It requires a bit more coding, but it is one of the better ways to insure that the animation and audio remain in sync with each other.

 

Ah that's interesting thanks for the detailed reply. Given that I use Timeline I think it doesn't make much sense for me to make compensations for timing on mobile in this plugin. I would be interested to hear if people have come across and/or solved this issue though.

Edit:
Ok it seems if you are playing Audio in a Timeline you should set the DirectorUpdateMode to DSPClock. This will keep the animation in sync with the audio.

https://docs.unity3d.com/ScriptReference/Playables.DirectorUpdateMode.html

 

Nice! Good find. That makes it a lot easier. It makes sense that they would add a feature like that to Timeline-based animation playback.

 

I'm not sure how ambitious you are about this project, but I do have a further suggestion. The CMU Sphinx and PocketSphinx libraries are open-source. If you wanted to, you could include those libraries as native plug-ins, and reference them directly instead of going through Rhubarb. While Rhubarb does a lot of the results parsing for you, it also limits itself to only ten or so shapes. More in-depth facial animation requires more subtlety, and I know for a fact that CMU Sphinx has support for all 39 standard phonemes for the English language.

 

Hi there, thanks for the tips.

It's unlikely I take this small project much further as the original main project I did this for was canned. I merely wanted to share what I had done so it didn't go completely unused and that others finding the Asset Store Lip Sync solutions lacking could have a better base to work from.

I am happy of course to maintain this, update along with Rhubarb and integrate pull requests or for others to take what I've done and develop it into something much more fully-featured.

 

please make more mouth example ,, your sprite is complex imean ,,why the A is closed mouth , dang its so cconfusing ,, i am very bad ,,,,,,,,, i mean i dont know how to replicate the mouth with my own style

 

Does this work in 3D>?