[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

Hi,

What does my character need to have in order to have lipsync? Do I need bones or blendshapes? Is there a way to automatically generate the blendshapes in another software?

 

Hi,

I have used LipSync Pro with no issues before but I cannot get any kind of lip sync to work with the new update. Using the SoX setup I get the following warning:

C:\Program Files (x86)\sox-14-4-2\sox.exe FAIL formats: can't open input file `unsigned': No such file or directory

I have tried deleting and installing, including previous versions of SoX. I have tried different Unity versions but still the same issue.

So then I tried to download the new ASMontreal module, which I first tried to download through the AutoSync Setup Wizard where I watch the package download, then install but the continue button on the wizard stays greyed out. So I try downloading it myself through the Extensions panel, same thing - it downloads and installs. Then when I go to AutoSync a clip (Gettysburg just to be sure) I get the following error:

Instance of ASMontrealPhonemeDetectionModule couldn't be created. The the script class needs to derive from ScriptableObject.

What's going on?

 

Rtyper, I don't know if you knew this or if it breaks things.... But, the "Default.asset" has a json field in it called Module Settings: Element 0. In this field is the following:

{"useAudioConversion":true,"lexiconPath":"G:\\Unity Projects\\LipSync\\LipSync Pro 1.X\\LipSync Pro 1.5X\\LipSync Pro 1.5 - Universal\\Assets\\Rogo Digital\\LipSync Pro\\AutoSync\\Editor\\Modules\\Montreal Forced Aligner\\montreal-forced-aligner\\pretrained_models\\librispeech-lexicon.txt","minLengthForSustain":100.0}

That is hardcoded to a path on your computer. Also, even if it was relative, the path to the pretrained models is way off of where it actually is:

Assets\Rogo Digital\LipSync Pro\AutoSync\Editor\Modules\Montreal Forced Aligner\Language Models\English\librispeech-lexicon.txt

Could this be the source of everyone's woes?

I tried to set this to the proper absolute location on my drive and still couldn't get autosync to work with my wav file however. I even tried the SoX trick mentioned by someone else earlier where the audio is manually converted in command line and then unticked the convert option.

The auto sync setup wizard says the modules are all installed for montreal. And, I checked under the verify button in settings in the clip window as well and it verifies.

Edit: I stepped through the C# code and determined that when the mfa_align tool is launched it is not being passed the -q option which causes an interactive prompt to appear "There were words not found in the dictionary. Would you like to abort to fix them? (Y/N)" And, because the C# code waits 20 seconds and doesn't see the program exit, it forces it to close and signals to the user that they must have failed to convert or encode their input audio properly. Which isn't the case.

Running the command from the command line for mfa works properly. For testing purposes I copied the mfa and sox folder from my project to another folder and created a subdir called corpus.

I followed Yannou's advice and ran sox as follows:
sox YOUR_FILE_INPUT.wav -r 16000 -b 16 -c 1 YOUR_FILE_OUTPUT.wav

With the sox converted mono .wav file and the txt file with the "transcript" renamed to the .lab extension inside the corpus folder. And, an empty output folder. I tested with some audio and typed:

mfa_align .\corpus ".\Language Models\English\librispeech-lexicon.txt" ".\Language Models\English\english.zip" .\output -c -v -q

Results:

Setting up corpus information...

Number of speakers in corpus: 1, average number of utterances per speaker: 1.0

Creating dictionary information...

Setting up training data...

Calculating MFCCs...

Calculating CMVN...

Number of speakers in corpus: 1, average number of utterances per speaker: 1.0

Done with setup.

 50%|██████████████████████████████████████████                                          | 1/2 [00:04<00:04,  4.35s/it]could not align ['onering']

100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:08<00:00,  4.05s/it]

Done! Everything took 47.13091826438904 seconds

This produced in the output folder:

07/26/2019 03:19 AM 28 oovs_found.txt
07/26/2019 03:19 AM 36 utterance_oovs.txt

So, I rewrote the .lab file to fix the unknown words and reran:

Setting up corpus information...

Number of speakers in corpus: 1, average number of utterances per speaker: 1.0

Creating dictionary information...

Setting up training data...

Calculating MFCCs...

Calculating CMVN...

Number of speakers in corpus: 1, average number of utterances per speaker: 1.0

Done with setup.

100%|████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:09<00:00,  4.58s/it]

Done! Everything took 41.62572765350342 seconds

This produced a proper .textgrid file.
Hope this helps someone else out there!

Edit 2:
Autosync completed successfully via the user interface after changing the code:

https://i.gyazo.com/5881d4578d39b68710ea2ff9ad117423.png

In the file:
\Assets\Rogo Digital\LipSync Pro\AutoSync\Editor\Modules\Montreal Forced Aligner\ASMontrealPhonemeDetectionModule.cs

Line 139 (added -q to the options):
process.StartInfo.Arguments = "\"" + corpusPath + "\" \"" + lexiconPath + "\" \"" + basePath + model.acousticModelPath + "\" \"" + outputPath + "\" -c -q";

Line 146 (Extended the timeout to 60 seconds):
process.WaitForExit(60000);

Though, there has to be a better way than picking an arbitrary wait time on exit. Maybe they have an API that would be better to use directly instead. The errors should be more informative too. Treating an arbitrary failure in the tool as if the user must have failed to prepare their inputs is misleading and creates a support headache where there does not need to be one.

 

Hello! I have a lot of voiceover for my project and I was hoping to do a batch processing on it. I set everything up in the AutoSync settings tab, I add all my clips to the batch process window, and then hit "Start Batch Process."

The result I'm hoping for is a bunch of *.asset files in my default save folder (which I can't seem to set in the settings tab). But instead the button generates an XML file. What do I do with this file? Can I use the XML file to generate the proper lip sync data files?

Thanks in advance! Hopefully my question makes sense.

-Dave

 

Thanks for pointing this out - I had made the same discovery about waiting for input and the --q argument just before I made my last post, and this is fixed in the latest update for the MFA module. All your other points are totally valid though - I had originally had a less specific error message for that final generic case but after some testing in a previous update I'd (mistakenly) come to the conclusion that the audio format was the only cause for it.

That hardcoded path to the lexicon shouldn't be there, you're right - fortunately though it isn't causing a problem, the module uses the chosen language model to get the path to the lexicon, so that must be a hold-over from a pre-release version when I created the preset. I'll re-save it now to avoid any confusion in future.

I'm currently both uploading the new MFA update and submitting a new LipSync update to the store, which improves on the error messaging and allows the MFA module to process even where there are out-of-vocab words found. I'll also be updating it later to allow extra words to be inserted into the lexicon per-project.

Either bones or blendshapes will work - you can mix and match them too. If you're using a custom-made model, there may be some tools out there that can generate facial blendshapes, but I wouldn't expect them to do too good a job of it. Some character creators like Adobe Fuse or (I believe) Reallusion's Character Creator can also export characters with blendshapes pre-made.

Your first error with the Legacy presets can be fixed by getting the latest version of the PocketSphinx module from the Extensions window - I always include the latest version when I do a full update to LipSync Pro, but if there are improvements between updates you can find them there.

Your second issue sounds like the scripts aren't compiling though. Check the console and see if there are any compilation errors, after the MFA module downloads and installs, it forces a script recompile which should unlock the setup wizard once it completes. I can only assume some error is preventing it from completing.

LipSync can use either .asset files or .xml files, so there's an checkbox in the batch processor for doing an xml export:

Make sure that's not checked, and if it's still doing it let me know! If you'd like to use the XML files instead, you can do - there's a form of LipSync.Play that takes a TextAsset and an AudioClip instead, though obviously that means you'd need to keep track of the audio clips yourself instead of the LipSyncData object doing that for you.

 

That tickbox is definitely unchecked. What does the XML file actually do? Can I convert that xml file into traditional *.asset files?

Also, I notice you wrote XML "files" instead of "file." When I hit the button, only one file is generated.

 

Hmm, that's strange - the xml file is simply a representation of the LipSyncData in xml format, so it's only one of the clips. The fact that it's only creating one is a known issue - check your email, I've sent you a patch for that problem. I'll have to look into it a bit more to figure out why it's creating an xml file instead of a .asset file though.

 

Rtyper, eh it's just bugs it happens. Thank you for the great product.

Also, when you put out the new version raise that 20 seconds timeout before killing the montreal process to something like 60 or more seconds. For example my short paragraph of text required 47 seconds to process.

 

Code (CSharp):

1. Assets\Rogo Digital\LipSync Pro\AutoSync\Editor\Modules\Montreal Forced Aligner\ASMontrealPhonemeDetectionModule.cs(121,31): error CS1501: No overload for method 'StartConversion' takes 6 arguments

I'm getting this error from the montreal forced aligner module. Using Unity 2018.4.4f1 on windows.

This is using the latest version that seems to have been updated yesterday..

EDIT: Oh i see, the issue is i needed to update PocketSphinix

 

I'm having this strange issue with the expression going weird at the end of a voice line.

Happens with :Keep expression when finished" is on or off with slightly different results. The pose itself doesn't have an expression like that, it looks like it's transitioning to the "Surprise" expression i set up.

 

Here is another error when making Batch process of files:

 

My Lip Sync component looks like this. It doesn't happen in an empty project, only in my game. Can it be because it conflicts with another file in the project?

Unity 2018.4.2f1, LipSync Pro 1.42

 

Hello, is there someone who could tell me what I am doing wrong. I have a list of audiosources and a list of lipsyncdata on a gameobject and I want to load these onto the Lipsync I have on the main character. I can see that the AudioSource gets placed in the AudioSource on the Lipsync component. But the lipsync never starts. Only the audio. If I choose play on awake on the lipsync component I can see the lipsyncData clip gets loaded onto their too, but my character never starts moving the mouth.

This is what I have on the audio Controller gameobjects start void:

Code (CSharp):

1.         girl.GetComponent<LipSync>().audioSource = audioF[1];
2.         aud = girl.GetComponent<LipSync>().audioSource;
3.         girl.GetComponent<LipSync>().defaultClip = lipsyncClipF[1];
4.         audClip = girl.GetComponent<LipSync>().defaultClip;
5.         aud.Play();
6.         girl.GetComponent<LipSync>().Play(audClip);

 

It seems I had to put another audio in the audiosource to make the first audio play. So problem solved

 

Hello any fix for bathc problem error? Batch creating of phenomenes is stops after first file and then there is an error:

 

Hi there, this plugin is great! I have been used it for a while and done well untill recently, I find that
for some audios, the Phonemes does not generate completely. Could you give me some suggestions?
I use LipSync Pro 1.5, Unity2018.1.0f

And this is my audio file.
https://drive.google.com/open?id=13qvTvaNqli7dqtUilPl0YKj0i0-MRHhS

I will really appreciate if you could help me

 

That's odd.. Does it happen if you move the end of the final emotion marker further in to the clip a bit? You could also try changing the character's emotion curve generation mode to "Tight" if it isn't already, or right click the last emotion in the editor and enable "Continuous variation", even if you set all the variation amount settings to 0, both of those things can help get rid of odd problems like this.

Are you able to update to version 1.51 from the asset store? This bug was fixed several versions ago

I'm not completely sure what the problem was, but you're massively overcomplicating playing a clip! The LipSync.Play method handles playing the audio and everything, so all you need to do is have a reference to the LipSyncData clip (presumably lipsyncClipF[1]), then pass that into the Play method:

girl.GetComponent<LipSync>().Play(lipsyncClipF[1]);

Though you probably should be caching a reference to the LipSync component in Start() - GetComponent<> is a very expensive method to be calling regularly.

I've replied to your PM If anyone else is having this issue as well, just update to the latest version from the Asset Store.

Sure, can you tell me what AutoSync preset you're using? If it's one of the Legacy ones, that system is quite old and isn't being updated any more. You can get much better results out of the Default (English) preset if you're able to get a transcript of your audio. Just make sure you've got the MFA module installed - if you haven't already, run the AutoSync Setup Wizard from Window > Rogo Digital > LipSync Pro to get everything set up and installed!

 

The new LipSync Pro 1.51 update is now live as well!

**If you're having problems with the batch processor, or compile errors when importing either the PocketSphinx or Montreal Forced Aligner AutoSync modules, make sure you download this update as both those issues are fixed!**

The full change list is below:


Features

• Added "Data Preprocessing" support.
  • New editor window that allows you to select a LipSync component and any number of LipSyncData clips, and process the animation for them ahead of time.
  • LipSync component will now check if LipSyncData clips contain pre-processed data, and bypass the animation generation step if they do, significantly improving runtime performance.
• Rect selection, Select All and Invert Selection options in the Clip Editor now only act on markers that are visible with the current filter setting.
• AutoSync modules now show a more informative error when the module compatibility check fails.
• The Clip Editor can now also display a message when AutoSync runs successfully, for more info (such as ignored words).

Fixes

• Fixed ClipSettings window not scrolling with a long transcript.
• Fixed bug that prevented batch processing mode from working correctly.
• Fixed SetEmotion method producing incorrect animation curves for some bone poses.

Changes

• Simplified some old code to do with opening LipSyncData files in the Clip Editor.
• Included fix to AutoSyncConversionUtility that has been distributed with previous PocketSphinx + MFA modules.
• [PocketSphinx Module] Updated included version of PocketSphinx module to latest.
• Updated minimum Unity version to 5.6.1

NOTE: This will be the last version of LipSync Pro to support Unity 5, in line with Unity's new asset guidelines.

 

The bug is fixed with 1.51, thank you! I thought I was using the latest version.

 

Just a heads up, I'm not able to access any @rogodigital.com email addresses at the moment. If you've sent an email to me and you need a reply, please either post here, or send me a PM. Cheers!
EDIT: Email is back up now!

Great, no problem!

 

Thanks for your reply, besides the issues I post below, I also find that when I process audio longer than 40 seconds using legacy presets, the unity editor will crash. So I use MFA module, but it failed to pass the check. Maybe I should upgrade to 1.5.1?

 

I'd definitely advise being on the latest version, but if you're seeing a "module failed compatibility check" error, that means your clip is missing some data that the module requires.
I'd guess in the case of the MFA module, you don't have a transcript of your dialogue?

Take a look at the documentation for more info.

 

I'm on 1.51 now, though I got the same error under 1.501. Most of the time, when processing an audio file, I get the following using the Legacy converter:

I've restarted Unity, and my computer, and I don't have these audio files open in any other programs. Any ideas?

 

Could you paste the actual path here? It may be some character(s) in the file name

 

It looks like it must be some weird file permission issue/corruption/problem in general.

I'm testing this on various audio files, all of which are in this folder:

C:\Users\Dan\Documents\GitHub\Gravia\Assets\Scenes\Production\Earth\MilitaryWing\FinalAssessment\FinalAssessmentI\Audio\Speech

One file which LipSyncPro handles fine is named "Final Assessment I - Flynn - Basic Awareness.wav", while one which gives the error is named "Final Assessment I - Flynn - Blindly.wav"

However, in responding to this post, I found that for the files that LipSyncPro errors out on, I'm unable to copy the filename for those files within Unity. For example, I highlighted the file name until it changed into a Rename prompt, and tried to ctrl-c the text, and it just copies a blank string. So I'm going with the assumption that these files are just corrupted/broken (even though they play fine as audio files) and that this isn't LipSyncPro's fault.

In short, nevermind, probably just a problem with the files, as Unity itself is showing odd behavior with them (despite them appearing identical to other files.) Sorry for the false alarm.

 

@dgoyette Hmm, ok. Do let me know if anything shows up that might link the problem to LipSync Pro though!

 

Nope, probably just some weird behavior when I converted MP3s to Wav outside of Unity using FFMPEG. I deleted all the audio files, and started over, and LipSyncPro works fine on them. Definitely some weirdness with the files themselves.

 

Hi, Rtyper! I've sent you an email but apparently you had a problem accessing it so i'll post it in here. I’m having difficulties with Rogo Eye controller. Whenever the character has the Look At target in sight, it’s eyes rotate to the side and ignore targets position completely. It doesn’t matter if Look At Target is set to Auto or to the specific target, as the results are the same.

I’m using both Rogo lipsync and eye controller in Unity 2019.1.10f1. If you could give me any tips on where to start looking for the problem I would be very grateful!

Thank you!

 

Hello There is still problem with batch processing in 1.51 version, sometimes it work,s but more often not work. I sent You PM with more info