[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

Hi AiryKai,

Are you getting any errors in the console when you try to start AutoSync? Make sure you have a US or UK English language pack installed and that your audio file is an uncompressed PCM .wav file. If none of that fixes it, you'll just need to wait for the 1.0 update in March - It will have a completely new version of AutoSync that solves all these problems.

This is strange. Can you show me what your LipSyncData file looks like on the emotions tab? Or best of all, if you can send a project file that recreates the problem to contact@rogodigital.com so that I can look into it.

 

No, no errors. Unfortunately this does not solve the problem. OK, I'll see how it will work in version 1.0 when it's released.

I tried to use your examples audio and data files. There you have already configured emotions. But it's just not working for me.

 

LipSync 0.61 is now live!
This update finally brings the Gestures system, allowing Mecanim animations to be triggered as part of LipSync dialogue clips.
​

Here is the changelist, as usual:

0.61
Features
- Gestures System - allows the triggering of standard Mecanim animations from LipSync files.
- Added two new properties to BlendSystems: blendRangeHigh and blendRangeLow, which allow blend systems to redefine the min/max slider amounts.

Fixes
- Fixed NullReferenceException when deleting a BoneShape with no bone defined.
- Fixed bones not updating when animated by Mecanim or Legacy Animation.
- Fixed bug where loading a new file in the clip editor wouldn't regenerate the waveform.
- Fixed bug that caused errors when attempting to play emotions that weren't in the emotion cache.
- Fixed bug where locking position/rotation of a BoneShape would have no effect.
- Fixed bug where clips shorter than 1 second would not generate waveforms.

Changes
- Improved Project Settings window to simplify adding/changing emotions and gestures.

 

That's a shame, don't worry - there shouldn't be too long to wait for the next update!

I'm really not sure what the problem is then, you're sure you've definitely got the character's emotion poses set up as well as the phoneme ones?

 

Hello there! Great asset. I've been enjoying it for a while. Thank you!

Just one question. Now that I tested it and it works amazing, I added a lot more voices to be autosynced. I have over 500 voiced that I want autosynced. Do I have to do them one by one or is there a way to select all my character voices and have the plugin autosync them with one click? It takes a lot of time draging and dropping the audio into the plugin, selecting autosync from the menu and then saving the file when there are a lot of audio files.

Thanks again

 

Just posting a quick note that the Dialogue System for Unity's LipSync Support package has been updated for LipSync 0.61. You can get the latest package on the Dialogue System Extras page.

 

Thanks, Tony!

Thanks!

Yep, I understand it's a bit cumbersome right now. That's the only method at the moment, but I'm adding an AutoSync batch processing function to LipSync 1.0, which will process a load of audioClips at once and save them out.
It should be released in early March - hopefully that's not too long to wait!

 

Eh a bit too late for my deadline but I'm sure we will use this later on in new games as well so it's very nice to hear.

One suggestion though. I would advertise the eye controller better on your asset page. I know it says it on the feature image and somewhere in the description but it gets lost while reading other features. Some people might not watch the videos or read through the description/images thoroughly. Maybe I'm just too careless but once I found out that was also included after purchasing I was very happy.

 

Hello, I just found this asset and it seems it can be very useful for our current project. However, I had a question.
What does the "upgrade price" mean? Does it mean that, if I buy it now I will have to buy the non-beta version later, or are you referring to dot-releases like "if you have version 1.something, version 2.something will cost the upgrade price"?

 

Yeah, fair enough - I'm redoing all the branding images for version 1.0, so I'll keep that in mind.

Thanks! The upgrade price is for people who already own the standalone version of Eye Controller, as it's included with LipSync. All updates for LipSync will be free, at least throughout 0.x and 1.x - I'm not specifically planning a LipSync 2 just yet, but if it did happen it would probably be a separate product with an upgrade price.

 

Greetings!

I really really like LipSync! I hope that the auto-syncing will work with the new version. Got no luck running SAPI.

But I've got a problem. I'd like to save a phoneme preset to copy to another character in my scene. But it only imports the blendshapes without the bones. I need the jaw bone to correct some modell issues, blendshapes won't work there.

Any possible solution for my issue?

 

Hello again!

Maybe a stupid question but does the phoneme detection tool have a language specific setting? Is it optimized for the English language or does it not matter since it's only interpreting sounds? I'm using the tool with Turkish audio and I sometime come across a situation where few phonemes are marked on the wavelength where a bunch of different sounds can be heard. Is this common with all audio or does the software having difficulty separating certain sounds from others?

Thanks again.

 

Yes, presets only store the blendshapes and their weights. Boneshapes have a transform associated with them, which wouldn't be the same one if it was applied to a different character. I'm making a lot of improvements to the preset system soon, and I plan on finding a way to make this work better then.

In the meantime, you can make it work (not necessarily perfectly) by adding the following blocks of code to LipSyncEditor.cs:

Code (CSharp):

1.  
2. newPhoneme.bones = new List<BoneShape>(ps.bones.Count);
3.  
4. for (int bone = 0; bone < newPhoneme.bones.Count; bone++)
5. {
6.     newPhoneme.bones[bone] = new BoneShape();
7.     newPhoneme.bones[bone].neutralPosition = ps.bones[bone].neutralPosition;
8.     newPhoneme.bones[bone].neutralRotation = ps.bones[bone].neutralRotation;
9.     newPhoneme.bones[bone].endPosition = ps.bones[bone].endPosition;
10.     newPhoneme.bones[bone].endRotation = ps.bones[bone].endRotation;
11.     newPhoneme.bones[bone].lockRotation = ps.bones[bone].lockRotation;
12.     newPhoneme.bones[bone].lockPosition = ps.bones[bone].lockPosition;
13. }
14.  

That should go between newPhoneme.weights = new List<float>(ps.weights); and preset.phonemeShapes.Add(newPhoneme); somewhere around line 1150, and then again in the same place in the LoadPreset() function, around line 1260.

Code (CSharp):

1.  
2. newEmotion.bones = new List<BoneShape>(es.bones.Count);
3.  
4. for (int bone = 0; bone < newEmotion.bones.Count; bone++)
5. {
6.     newEmotion.bones[bone] = new BoneShape();
7.     newEmotion.bones[bone].neutralPosition = es.bones[bone].neutralPosition;
8.     newEmotion.bones[bone].neutralRotation = es.bones[bone].neutralRotation;
9.     newEmotion.bones[bone].endPosition = es.bones[bone].endPosition;
10.     newEmotion.bones[bone].endRotation = es.bones[bone].endRotation;
11.     newEmotion.bones[bone].lockRotation = es.bones[bone].lockRotation;
12.     newEmotion.bones[bone].lockPosition = es.bones[bone].lockPosition;
13. }
14.  

Then that should go shortly after the previous block, in the matching sections for Emotions (around 1170 and 1280). You'll need to re-add the bone transform to each shape when you apply the preset though.

 

Don't worry, it's not a stupid question - I'm not 100% sure myself! The SAPI phoneme detection does seem to work better with English dialogue, and it often refuses to work at all when using other system languages, but it should still work, at least on a basic level, with other languages that use the same sounds. There could be a problem with other sounds that aren't used in English, which it might fail to recognise.

SAPI is being replaced in 1.0 with a system based on CMU Sphinx, which does have support for other languages available, so I'll be looking into making AutoSync support them after that is released.

 

Just got your package after a long time subscribing to this thread

Playing with it, I've had multiple problems running the autosync . I have this audio from MP3 converted to uncompressed .wav. Ran the autosync and it pops up in a fraction of a second and close down. I opened your sample gettysburg.wav file in audacity, and paste my audio clip into it, save, ran autosync and voila. Somehow it works. Then I closed Unity, reopen the project and try to run autosync with the same audio file and the problem occurs again. Trying my earlier trick with a new audio file also doesn't work anymore.

Note I imported into an empty project and just added my audio files for testing. When it works it's great, but my tries have been more failures than successes.

 

Hi Ikazrima,
That's interesting that it worked at one point - it rules out SAPI having a problem with your PC. What is the name of the folder you have the audioclips in? There's a bug with it that if any folder on the path between where sapi.exe is located and where your audioclip is has a space in the name, it will refuse to work. This basically means any folder between the LipSync folder (under Rogo Digital) and your audioclip.

Let me know if you're still having problems.

 

My folder is named "Audio", but there are several folders with spaces in between. I renamed my folder so that it sits directly on top of Rogo Digital folder, and it didn't work. I ran autosync on the boneanimation clip and it works. I dragged one audioclip into your Lincoln example folder (which contains the boneanimation clip), well it didn't work.

Is there's some very specific settings to convert the audio to?

 

It has to be an uncompressed PCM .wav file. These are the settings I used to export both the Gettysburg and bone animation files from Audacity, if it helps:

 

Thanks, I'll try it tomorrow!

 

Well I've figured out that the audioclips themselves cannot have spaces in their names. I just replaced every spaces with underscores and now it works.

 

Ah, yes. Sorry, I should've made that clearer! The problem with spaces will be fixed in the next update anyway.

 

Ok cool
I'm using bone instead of blendshapes, and my characters share identical armatures. Any ways for me to copy over the phenomes over? This could be solved if the position and rotation didn't reset each time I change the bone transform.

 

Yes - again, this is something I'm improving in the next update, but you can add the code I posed above in reply to Krebsimitat to allow presets to store bone information as well.
I'd forgotten about the positions resetting when the transform is changed though! @Krebsimitat, you'll want to do this too, so you can set the bones correctly after applying the preset.

Find these two lines in LipSyncEditor.cs:

Code (CSharp):

1. phoneme.bones[b].endRotation = phoneme.bones[b].bone.localEulerAngles;
2. phoneme.bones[b].endPosition = phoneme.bones[b].bone.localPosition;

and comment them out. It's not ideal, as it will make setting them from scratch more difficult, but it will make the presets work for now.

 

Thanks for your help. You were right, it's not perfect.

BUT I think thats my fault. I made a second instance of my character and copied over the complete Lipsync component. The bones were there but not with the same motion. The same with your modifications. Added the bone to the restored preset on my clone, but with the same results.

The model is identical, but I think I made some mistakes with the positioning and scaling. Starting from scratch now and try again. But hey, everytime it goes a little bit faster Shaping the phonemes now for the 5th time or so.

One other question (maybe better for the documentation but while I'm here ...):
Is there a way to set a default emotion or change the emotion without running the actual lipsyncing? When my character is idle I want to keep the emotion on the face.

And thanks again for your support! Never got much luck in the past with that, not to mention a working code snippet in a couple of hours!

 

Hi, so i just bought this asset, it looks really promising and example scenes looks nice, but i cant get it to work with my own audio files. I tried converting the audio to WAV, signed 16Bit PCM with Audacity, but the process bar just flashes and nothing happens. I also have both UK and US language packs installed, and still no luck. Im using Windows 10 64bit.
I also tried with the Gettysburg Audio File, but same happens, progress bar just flashes and nothing else happens.

Biggest problem here is that we have to get lipsync to our game, as thats the only thing it needs before publish.

Edit: And Console shows no errors.

 

Sorry for the late reply! You can call SetEmotion() on the LipSync character to blend into an emotion outside of an animation.

Well, it sounds like you did everything right! Sometimes the sapi_lipsync.exe just doesn't work on some computers and I haven't come across any real fix (or even a reason for it). LipSync 1.0 will be out later this month though, and it will have the new, cross-platform version of AutoSync included, which solves the compatibility problems SAPI had. Sorry for the inconvenience!

 

On the subject of updates, LipSync 0.611 has gone live on the Asset Store. It's a very small update with 2 bug fixes, one of which prevented Gestures from replaying after an animation has completed (so it's quite important if you're using Gestures!)

I'd also like to properly announce a few changes coming soon. The first is that from the 1.0 update onwards, there will be a free, cut-down demo version of LipSync available, called LipSync Lite. This will contain the same basic features and be free to use in projects, only without Emotion Markers, Gesture Markers or AutoSync.

We'll also be releasing our new tool for real-time lipsyncing, LipSync Live, in May. This will be a completely separate product that combines LipSync's pose editor, blend systems and presets with an entirely real-time lipsyncing system, for animating either pre-recorded dialogue or audio live from a microphone, with no per-clip setup required in advance. More info on this later.

The final thing is LipSync itself will become LipSync Pro, in order to differentiate it better from the other two LipSync products, and the price will be going up for the last time, to $35. Along with the change in names, the branding is getting an update:



Other than that though it's still the same product, with many new improvements to the editor and the component in 1.0, and the update will still be completely free for people who have already purchased it to download.

I'll be posting some more info on LipSync Live, the new features in LipSync Pro 1.0, and the new tutorial series I've been working on in the next week or so.
LipSync turns one year old in two weeks' time, and I've already seen loads of amazing projects people are doing with it. I can't wait to see what else everyone comes up with over the next one!

 

I'm considering to by LipSync but I need to know which platforms it supports?
We need support for PC, Mac, Android, iOS and WebGL.

 

Hi Henkjan,

LipSync supports building for all of Unity's platforms. I can't guarantee good performance for multiple characters on WebGL however.

 

Just a quick shoutout to Paradox Notion's new sequencer, SLATE: Cutscene Director, which has great built-in support for LipSync characters, including a real-time preview of your LipSync animation when scrubbing through the whole cutscene! If you're interested, it's just launched on the Asset Store, you can get it here.

 

@Rtyper
Greetings,

I posted e-mail to You, please look at it.

Best regards,
Vit

 

Hi Vit. Sorry, I wasn't around yesterday - I've replied to your email now.

 

Thanks for the mention!
It's been a pleasure integrating and working with LipSync! LipSync is very easy to use and with great results

 

There's a few new UI changes coming in 1.0 - I've set out to try and make everything as simple as possible for this final release. One part of that was reworking how emotion markers work, so that they're less fiddly to work with. They now work more like clips in a video editor, so they can be moved from one side of an emotion to the other (instead of being locked to a certain area of the timeline) and it's now possible to make small adjustments to the blend in/out times. Here's the old and new markers side by side to compare:




I'll be posting some of the other new features over the next few days!

No problem! Best of luck with Slate, I've had a chance to try it out in a project now and I really think it's the best cutscene tool out there.

 

Hi there,

first of all, thank you so much for the plug-in, it's been working beautifully so far.

I've just had a few issues with assigning emotions that hasn't come up before. When I add an emotion to the end of a audioclip for example (but it happens when placed anywhere) about half way through the clip the emotion will play slightly where it isn't supposed to for a second or two, like a mini-preview of what's to come. Also when it get's to the actual clip it seems to fade into an extreme version of the emotion that has been assigned, it looks like the fade in/fade out sliders have been messing with the extremes or curves of the emotion some how.

I hope this makes some sort of sense, I do have a video of this and tried sending it to you but I believe your inbox was full. Is there a different one I could send it two? Unless of course it isn't required.

Thanks you in advance and again for the software itself,

Joe

 

Hi Joey, sorry about that - we've been having email problems recently, still trying to get it sorted! Could you upload the video to Dropbox or Google Drive (or something similar) and email the link?

 

Thanks for the speedy response, I have emailed you the link and it hasn't come back so hopefully that's fine. Thanks again.

 

Hi, I am having some serious issues with getting auto target to work. I changed targetDistance from Mathf.Infinity to autoTargetDistance, but it doesn't seem to do it during runtime. If I set the distance to shorter than the distance to the player at start it will never target it, but if I set it to longer than the distance to the player, then it will always target it. It seems to never change the look at amount after start, maybe. Hope that makes sense. Any help would be great, thanks.

 

You're right, that's a bug - I'm not sure why it won't target anything that starts outside the range, but once it locks on to a target it won't lose it unless another target is closer. LipSync 1.0 will come with an update to Eye Controller that (amongst other things) adds directional checking to the auto targeting as well as distance checking (to prevent locking on to a target behind the character that's slightly closer than one in-front).

If you add this line:

Code (CSharp):

1. target = null;

just after

Code (CSharp):

1. float targetDistance = autoTargetDistance;

that should fix the problem with targets going out of range.

 

As promised, here's some more of the improvements to LipSync in 1.0: much better support for bone transforms!

Bones are now saved in presets along with blendables, and there's a brand new preset format that allows for more flexible matching up of the stored bones/blendables and their equivalents in your character. I've also added proper scene view handles for manipulating bones, instead of directly changing the position/rotation values, and a new wizard for creating poses from animation clips. Check out the video above for more detail on how this works!

 

Hi
Watched this thread for a longer time and the improvements are just great! Wanted to get it but whats about the Mac version?
Just a suggestion from me: Worked for a few years with video-editors like FinalCut and there you first played your sound footage and set markers by hitting a key on keyboard(M). Then you can align your visuals and transitions to the created markers. You also have the ability to name them. This really is fundamental in video editing.
I think that would be really helpful in LipSync too! My opinion.

Cheers
Michael

 

Hi Michael, sorry for the late reply!

It's already available for Mac, with the exception of the AutoSync feature, which is coming in 1.0 in a couple of days' time.

I'm not sure how that would relate to LipSync, really. In video editors it makes sense to have a rough set of markers as you play through it, but in LipSync the markers are basically the final result. I could see some set of keyboard shortcuts being useful though. I'll definitely look into adding them in future.

 

Do you have a more exact release date for LipSync 1.0. ? It's now end of March, and the problem is I stopped making Lipsyncs because awaiting the new version.

 

Sorry about that! I know I'd said the 28th, but there was a bit of a set-back with the new AutoSync system, which delayed it a bit. It is very close to being finished now (there's about 4 things left on the to-do list, 2 of which are very minor), so I expect to have it ready by the start of next week at the latest. To save waiting for the Asset Store approval, I'll send the update out to people on the day it's finished if you send your invoice number to contact@rogodigital.com.

Thanks for your patience!

 

Hi everyone! Very close now to being finished with this next update - I'm expecting it to be ready on Monday (the 4th) now, and it'll hit the store sometime after that (along with the final price rise to $35). As I said before, I'll still email the update out to those people who have sent me invoice numbers, but to make it easier I've set up a form on the site to download the latest update.
Just go to lipsync.rogodigital.com, click the downloads button in the top-left corner and enter your invoice number. At the moment, this will give you 0.611, but I'll update it to 1.0 once that's ready.

Cheers!

 

Any news about 1.0?

 

Hopefully today! It's very nearly finished, just dealing with one last issue then I can package it up for download.

 

Hi there,

any updates? Mainly just making sure I haven't missed anything.

Cheers

 

Don't worry, you haven't. It's basically finished - there's just one pretty important bug with audio conversion stopping me from releasing it at the moment. I don't want to delay for ages (especially as I'm already on Valve time!) but getting this feature working is pretty essential to the new version of AutoSync. If the worst comes to the worst and I can't fix within the next week, I'll release it anyway and patch this at a later date.

 

Cool, thanks for that. I was wondering, we have already used Autosync in the current version to get all our lip syncing matched to our audio files, I was waiting for the update because I was having issues with the emotion assignment. If I was to get the new version without the Autosync stuff fixed would that be fine for my needs? Basically will there be any issues with compatibility if we have used Autosync from the last version and emotion assignment from the new version?

Thanks again.