[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

Hi,
I'm currently using the "bones only" blend system and it seems that the phoneme curve generation always makes linear curves.
The "Tight" and "Loose" options doesn't seem to change anything.
After a look in lipSync.cs it seems (but I'm really not sure) that the loose and tight parameters are only used when using blendshape, is this true? If yes, is it possible to add these curve options to the bones-only system?

 

Good point - I'd noticed the the glitching when doing the new tutorials recently. I'm looking into it now, for a fix in 1.52, but in the meantime you can reduce it by using the "Tight" emotion curve generation mode. Renaming emotions is a little difficult as they're just referenced by name, but I'll see what I can do!

That's right - it's not strictly a linear curve, but the overshooting that Loose mode can cause is made much worse when dealing with rotating/moving bones so it's not used! Curve generation is a little rigid at the moment though, that's something I want to look at improving.

 

Thanks man, I'm looking forward to these updates. Can I support you anywhere else? I'm willing to pay for these fixes. (Dunno, since it's been so long, I can't expect you to fix these kind of things for free entirely.)

 

Hello again
Having an issue with FPS while the lip sync is playing. Drops from a smooth 60 to about 30fps until the clip finishes. Any suggestions? Using a Morph3D model with the blendshape set that was provided. Really want to use lipsync in gameplay and not just cutscenes, but having trouble at the moment. Other than that love the asset! Any help would be appreciated. Thank you

 

Can you share your animation exporter integration? I really need something of the sort right now.

 

Hey whenever I try to setup Phonemes in LipSyncPro component I get an error and the editor UI bugs out:

Code (CSharp):

1. ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
2. Parameter name: index
3. System.ThrowHelper.ThrowArgumentOutOfRangeException (System.ExceptionArgument argument, System.ExceptionResource resource) (at <7d97106330684add86d080ecf65bfe69>:0)
4. System.ThrowHelper.ThrowArgumentOutOfRangeException () (at <7d97106330684add86d080ecf65bfe69>:0)
5. System.Collections.Generic.List`1[T].get_Item (System.Int32 index) (at <7d97106330684add86d080ecf65bfe69>:0)
6. RogoDigital.Lipsync.BlendSystem.SetInternalValue (System.Int32 blendable, System.Single value) (at Assets/DMAI/3rd Party/Rogo Digital/LipSync Pro/Classes/BlendSystem.cs:230)
7. RogoDigital.Lipsync.BlendshapeBlendSystem.SetBlendableValue (System.Int32 blendable, System.Single value) (at Assets/DMAI/3rd Party/Rogo Digital/LipSync Pro/BlendSystems/BlendshapeBlendSystem.cs:65)
8. LipSyncEditor.OnInspectorGUI () (at Assets/DMAI/3rd Party/Rogo Digital/LipSync Pro/Editor/LipSyncEditor.cs:791)
9. UnityEditor.UIElements.InspectorElement+<CreateIMGUIInspectorFromEditor>c__AnonStorey1.<>m__0 () (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorElement.cs:462)
10. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

In BlendSystem.cs:230, _blendables has 12 elements and blendable = 18.

 

Not a whole lot, I'm afraid. I've looked into this before, and the frame drops came from MCS' internal morph system. (Enable deep profiling and follow that tree down as far as it goes, almost all of LipSync.LateUpdate()'s frame time there is coming via the MCS BlendSystem from some MCS internal class.) I'd hoped they would eventually fix this, then the entire product became deprecated. The best options you have are using as few morphs per-pose as possible, and limiting the number of clips you have playing at once. I'm sorry I don't have any better advice here!

Is this on a new character? There shouldn't be any way for those to get out of sync like that. What you can do is put the Inspector into debug mode, find the "Phonemes" list and remove anything with too high a number.
If you can tell me what happened to cause this though, that'd be a big help in figuring out how to stop it from happening!

 

Yes, my workaround is to only assign blendshapes with lower numbers for now. it's not really a good solution, but at least it doesn't break things for the time being.

I added a debug log for characterMesh in BlendshapeBlendSystem in GetBlendables (line 82). When I expand the phoneme tabs of the character, I get a print out of all the characters(skinned meshes) in my scene so it seems that _Blendables in BlendSystem is getting overwritten buy other characterMesh which has a lower number of blendshapes. I'm not sure how this "leak" is happening as I can't find any script that assign characterMesh (maybe by reflection?).

To give you a full picture of what I'm doing:

-I have 5 characters in the scene. They have an average of 12 blendshapes. The character in question has 24.

- I have reorganized LipSyncPro in 2 assembly definitions, run time assembly and editor assembly. In order to do so, I had to put everything under editor folder in one editor folder (I tried keeping the file structure as is but there are inter dependencies). This doesn't affect anything.



- I tested the plugin as is in a new project. still getting the same behavior.
- If I remove all characters from scene except that character, debug log only prints out that character skinned mesh, but still get the same error.

 

Sent an e-mail for support but have not gotten a response.

Installed asset, even on a blank project to make sure no conflicts, the example scenes aren't working. The eyes move but not the lips.

Get this warning when hitting 'Start Talking':
Playing pre-processed clip on a different character. The animation may look incorrect or not play at all. You can remove the pre-processed data by selecting the clip in the Project window.
UnityEngine.Debug:LogWarning(Object)
RogoDigital.Lipsync.LipSynclayPP(LipSyncData, Single, Single) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:636)
RogoDigital.Lipsync.LipSynclay(LipSyncData, Single) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:707)
RogoDigital.Lipsync.LipSynclay(LipSyncData) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:765)
UnityEngine.EventSystems.EventSystem:Update()

If I recompile get two of these warnings:
Assets\Rogo Digital\LipSync Pro\Editor\LipSyncClipSetup.cs(608,3): warning CS0618: 'SceneView.onSceneGUIDelegate' is obsolete: 'onSceneGUIDelegate has been deprecated. Use duringSceneGui instead.'

 

Slightly off-topic but I'm wondering about this.. I'm not sure if this is true either.
But wouldn't being 'in editor' and having an object selected or having the gameobject's expanded in the hierarchy also affect profiling performance? I'm pretty new to that profiling thing though, so I have no clue.

 

Small feature request: It's typical for me to slightly tweak audio clips over time, requiring me to re-process my LipSync Pro files to account for the changes. I'd like to be able to do this with the Batch Processing dialog. However, it seems like the Batch Processing can only be used to create new LSP assets, not update existing assets. Is that correct? If so, it would be nice if the batch process could accept LipSync Pro assets and reprocess them, in addition to its current functionality.

 

Wanted to say thanks to @DGordon for sharing his Timeline playable from back in 2017.

@Rtyper Is there still no official playable for LipsyncPro?

 

Hi there. I recently purchased this and I'm just in the early stages of testing it out, but it's not playing any of my audio files in the clip editor. The original audio file was an MP3, and that didn't work, so I tried installing SoX as per the documentation and that didn't work either (the green light shows up when I go to see if it's verified). I also tried converting it to a 16 bit wav but that also didn't work.

Any ideas?

*Edit to add: It plays fine in unity when I push play to preview it while it's selected, so I don't think it's a problem with the clip, it just doesn't want to play in the clip editor (and yes, the waveforms do show up)

 

My turn to give something back to the community...

I've found a workflow that allows you to combine multiple skinned meshes on one or more characters, using MeshBaker, while still being able to target each individual character's blendshapes in LipSyncPro:

I've added a few lines of code to the BlendshapesManager.cs script that allows you to filter the blendshapes on a mesh as they're parsed into the dictionary that LipSyncPro uses. @Rtyper would you be happy for me to share this altered script, or perhaps I could let you have a copy for you to include as part of your integrations modules?

 

Hi everyone.

Is there a guide somewhere about doing a clean install of Lip Sync Pro? I can't seem to find one on the website. I ended up getting it working in a clean project with nothing else in it, but as soon as I try to get it set up in our main project so we can integrate it with our game, it doesn't seem to want to install the montreal forced aligner module. I can see the files and the folder sitting there, but it's like lip sync pro doesn't know they're there when I run the autosync setup wizard.

I also get a constant stream of compiler errors when I open the clip editor to check the settings there, and none of the demo scenes work that come with the plugin (they'll play but the lip syncing doesn't activate with the audio).

 

can you add a null check for boneCurves on line 611 of LipSync.cs? I'm getting NullReferenceExceptions after upgrading to 1.51, thanks!

 

Has this asset been deprecated?

 

Not yet. The dev says he's working on 2 updates: 1.6 for the upcoming 2019.3 release and 1.52 for folks who are stuck with pre-2019.3 Unity. I get the feeling it's an individual developer, not a studio, and they probably have a full-time job outside of this. Which means that their time is limited, especially in holiday season, and they're probably focused on coding instead of forum support right now.

But I wouldn't be surprised if the dev lets it go in the near future. It seems like an awful lot of work and stress for something that isn't a primary income. I made a recent purchase, too - and I hesitated for just this reason. In the end I threw the dice here instead of going with SALSA because at this stage, I significantly prefer the LSP output over SALSA's.

But about your problem: I saw some references to MFA issues on the last couple pages of this forum. Maybe you'll find an answer there.

 

Sounds like there's potentially some good updates coming, was wondering if these errors are related to the situation that Lip Sync Pro is in right now, if so, then it sounds like I just need to wait.

I installed fresh on Unity 2019.1.2f1 and doing anything related to Lip Sync Pro (viewing the clip editor, adding the component to a game object etc.) will throw any combination of these errors:



Sorry if I'm beating a dead horse but I wasn't sure if these errors were the ones related to the state of things (Lip Sync Pro having impending compatibility patches) or if this is aberrant behavior.

Thanks in advance, hoping it's just something dumb that I did, or that it really is just a situation of standing by for the Lip Sync Pro updates that were talked about.

 

Apologies for the wait guys - I've been working on the new LipSync updates, I've got a couple of other projects on the go and I'm getting ready to move house in just under 2 weeks, so things will probably continue being a bit sporadic until everything's settled down!

The main thing is LipSync 1.52 was submitted to the Asset Store this morning. It ended up having a few more changes in than I was originally planning which delayed it a bit. Full changelist at the end as per usual.

@jeromeWork That's awesome! Feel free to share it, just keep the core LipSync API scripts (LipSync.cs, BlendSystem.cs etc) out and there's no problem.

@DeadSeraph The Getting Started guide on the website covers basic installation, and use of the LipSync component and the Clip Editor are covered on the YouTube channel, though I want to do text versions of these at some point as well. If the wizard isn't detecting the right path automatically you can browse for the mfa_align application yourself and it should then work.
Could you post the errors you get when opening the clip editor? I haven't run into anything like this before, so I'd help to see what exactly is happening.

@nocanwin Good call about the null check there, sorry I missed this before submitting the last update, but I'll get on it for the next one.

@Safemilk I can't say I've seen this happening before, though that first error (ReflectionTypeLoadException) looks interesting. Is there a stack track or line number in the full version of that error at the bottom of the console? Both AutoSync and the editor for the LipSync component use reflection in order to find modules and blend systems in the project, but I can't think of a reason it would break the clip editor.

 

Quick update, 1.52 is now available. Here's the changelist that I forgot to include in the previous post

1.52
Features

• Added support for saving and loading LipSync presets outside a folder named "Presets".
  • Presets can now be saved anywhere, and a new "Display Name" property is used for sorting into categories.
  • Added a new custom inspector for presets, showing an overview of the stored data, and allowing the display name to be edited.
• Improved AutoSync Preset UI in the AutoSync window to allow presets to be added to, overwritten or saved as copies.
• Added AudioClip merging support to AutoSyncConversionUtility.
• Added GetAnimationCurves() method to TransformAnimationCurve to support the animation clip export package.

Fixes

• Fixed bug where deleting pre-processed data from a clip didn't get serialized.
• Fixed loading a preset on a prefab in prefab mode not forcing the prefab to save.
• Fixed how the "Yes" and "No" options in the "Save File" dialog show up.

Changes

• Made some data in LipSyncClipSetup public so extensions can access them.
• Removed some legacy code supporting Unity 5.3.5 and lower.
• Removed old "BlendshapePreset" class that has been deprecated since 1.0.

 

Hey, Rtyper! Great news on the new version, I'll do a clean install of Lip Sync Pro and see if the issue persists!

Before the update though I'll send the error with the line locations of the error just for posterity, then I'll see how I fare with 1.52! Thanks again!

Here's the full error.

 

Dang, no luck, I removed Lip Sync Pro and installed the new version fresh, when I opened the clip editor these 3 errors popped up:



So then I made a fresh project and brought 1.52 into it to see if there was something in my project interfering with it. A fresh project with 1.52 seems to work perfectly, until I restarted! Then the errors started coming back.

Very strange. We are on 2019.1.2f1 Pro still.

 

No problem, and thanks for the line numbers! I've looked into the exception and it means there's some class in your project that isn't loading correctly. I'd guess it's possibly a DLL as you don't have any compiler errors. What other assets do you have in the project?

Either way, I can add a check for this to that class that should make it usable, or at least help us figure out what's breaking it. I'll PM you a patch when I get it sorted.

 

@Rtyper hey, did you get a chance to look into the bug I mentioned?

 

Thanks for the reply. Wasn't sure if the asset had been deprecated or not. I will grab the latest update and see if that solves the issue I was having previously. If not, I'll post again with screenshots.

 

So, I installed the new update and it still doesn't want to recognize the montreal forced aligner extension. I run the wizard, wait for it to install, it installs, compiles, then the wizard doesn't recognize it as installed, and the console displays the following (bottom most error, the others are unrelated):


*Edit to add*: The settings for autosync in the clip editor do not show any way for me to point the plugin to the extension manually (only shows 1 for sox)

 

Also, when I try to download the forced aligner extension manually, it says it's already installed.

 

I think those other errors are actually the problem. It looks like there are errors in some script code (or possibly a DLL, I'm not 100% sure) that are preventing Unity from compiling all the scripts in your project. This would explain why the files for the MFA module could be in the project, but LipSync can't detect it because it hasn't compiled.

 

Hey thanks a ton for the info, that definitely gives me a lead to look into, I'll see what I can turn up, we definitely have a ton of assemblies in our project so this could actually be code smell to a much larger issue!

 

Restarting unity did the trick. This didn't work with the previous version, so I'm not sure what changed with the build you released today, but it's functioning now

 

Hello,

I can see from the roadmap on your website that French and German langage models for MFA should be available since August, but I can't find this extension in the plugin. How can I get this ?

 

Hi there. I am running into this problem today. I even tried an audio file that worked previously (with the same transcript that worked previously) and now it doesn't work after installing lip sync pro with a pre-existing project. The audio files are mp3's. Any suggestions? I also tried them with legacy autosync and it still won't work.

 

Does the latest release of LipSync pro let you export animations as .Anim files? My team and I would really like to use this product, but we need our animations as .Anim files for our game.

 

Hey! Just start discovering the LSP and so far it looks great! I have a question:
What's the status of Timeline support? Right now i'm on a research phase for a new project so I'm choosing the workflow and instruments. Should I count on a native Timeline support in a moderately nearest future or should I consider some other ways?

 

When I set the phoneme This way and click the component. This error message and GUI error happens. What did I do. I downloaded latest version 1.52 and unity 2019.2.10f1


ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index System.ThrowHelper.ThrowArgumentOutOfRangeException (System.ExceptionArgument argument, System.ExceptionResource resource) (at <599589bf4ce248909b8a14cbe4a2034e>:0) System.ThrowHelper.ThrowArgumentOutOfRangeException () (at <599589bf4ce248909b8a14cbe4a2034e>:0) System.Collections.Generic.List`1[T].get_Item (System.Int32 index) (at <599589bf4ce248909b8a14cbe4a2

 

Hi all, I'm trying to auto-lipsync an audio file but keeps on getting this error. I have several audio files, half of them are spoken by a female, and half of them are spoken by a male. The male audio files were the ones which got the error.

This is the error message that showed up:

NullReferenceException: Object reference not set to an instance of an object

RogoDigital.Lipsync.AutoSync+<ProcessAudio>c__AnonStorey1.<>m__0 () (at Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/AutoSync.cs:237)

RogoDigital.ContinuationManager.Update () (at Assets/Rogo Digital/Shared/Editor/ContinuationManager.cs:31)

This error stops the process of auto-lipsync I think, because the green bar on the "Generating Data" popup window doesn't progress anymore after this error showed up. It keeps on showing "Please wait, generating LipSync data". Btw just out of curiosity I tried to handle the statements that give the error around the AutoSync.cs:237 by putting them in a try catch, but the process is still not progressing after this change.

/// UPDATE ///
I edited the audio files again and this time the error doesn't show up. So I guess my problem can be solved. But the error will still be there! Would be nice if it can be handled maybe.

 

I'm getting the same errors as @Safemilk . For me it starts throwing them if the Lipsync window is opened in an editor layout. If I open it using Window->Rogo->Lipsync it doesn't throw the errors. I'm on 2019.2.11f1

 

Hello . Great product but I have one problem. everything work correctly when I have eye controller with lipsync pro component but when I order to speak the eye blink only one side. I don't have any emotion setting in clip data just phoneme setting. Thank in advance

 

I get error "Mfa application error. Check your audio encoding or enable conversion. Error: mfa codecs.py "utf-8" codec can't decode byte 0x3 at position. invalid start byte.

 

is there a way I could set LipSyncData from the script? like I just was to change clips of LipSyncData or make LipSyncData in the script so I can add to LipSync character

 

Hello, I'm getting a weird error when I try to add body animations to my existing gestures. Not sure why it's doing this but for some reason, once it starts I can only get rid of it by removing all the additions and rebuilding the old graph.


[LipSync - Character_Female_Torso_Head] Animator does not contain a trigger called ''. This Gesture will be ignored.
UnityEngine.Debug:LogWarning(Object)
RogoDigital.Lipsync.LipSync:Awake() (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:327)

 

Does there happen to exist a list of appropriate blend shape values for Mixamo models for each phoneme? For example, here are the blend shapes for my model:



When adding blend shapes for each phoneme, I know there's a little image to help me make the mouth do the right thing, but I'm not highly confident I'm setting all the right blend shape values as accurately as possible. I was wondering if anyone else with a mixamo model already has the individual blend shapes and values for each phoneme?

 

@dgoyette There are two Mixamo presets that you can use. One normal, the other more exaggerated. They load in the required set of blendshapes and values for each phoneme

 

That's great. Completely missed that, and more than I would have expected. Thank you.

 

Hello I started configuring this and got this warning please help me.

 

Unity has a pre-fab system that it uses for creating "templates" for game objects, allowing for those game objects to be re-used within a scene, with individual instance changes made after the fact. It looks like you may have linked to the pre-fab, as opposed to the game object in the scene. You will want to point it to the instance of the pre-fab in the hierarchy. I recommend selecting the object in the scene, and then giving it a unique name in the hierarchy. Then select the unique name in your script.

 

Thank you very much for your quick response.

 

Hello, I tried to set up blinking for my 3d model. But I do not see among my settings at least approximately what was in the tutorial

manual

my settings

I have only 4 elements there
​

 

The blend shapes it shows are simply those blend shapes that are available on your model. Blend shapes aren't automatic, so if the author of your model didn't put them in, they won't be available for LSP to use. Where did you get the model?

Another thing to consider is whether your model has multiple meshes, and the blink blend shapes are on another mesh. If that's the case, I think both the Blend Shapes and Advanced Blend Shapes approaches allow you to specify multiple meshes.

Anyway, does your model actually have blend shapes for blinking?