[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

I've run into an issue with my sprites and I'm not sure what's going on. I've created my own mouthchart with sprites for both 'happy' and 'sad' versions. When I do the lipsync though, some sprites are displayed correctly while others are way off. Below you'll see that the lipsync calls for the 'm-s-mbp' sprite (which is the closed mouth 'm' sprite), but it seems the 'M-S-L' sprite (L mouth shape) is being displayed instead. Any idea what could cause this mismatch of sprites?

 

just hangs on please wait recognising phonemes. please help.

 

I totally understand - I'll obviously do my best to answer any questions, so if you need pointing in the right direction, or explaining how some feature works/where its code is, I'll be happy to help. Like I said originally, I can create the basic version of that blend system if you'd like, but major modifications to the LipSync component are just a bit too much for my workload at the moment!

If I understand the idea right, I think it would probably be more difficult to be honest! I think the simplest way will be to create a manager component for the blend system (like the sprite and advanced blendshape blend systems have) that will let you add either changes of texture or uv offsets and then use them in your phoneme/emotion poses. I'd be happy to make this as part of putting the blend system together too.

Baking animations is a planned feature for 2.0 (which will be a $10 paid upgrade), but it's not possible just yet. As a bit of advice, it's likely to only be possible when using bone-based rigs (and/or possibly blend shapes), at least at first. That'll mean it won't be compatible with character systems like Morph 3D or UMA.

There is a preset for MCS! Have you downloaded the Morph3D blend system? If you have, you need to import the presets package that downloaded with it before the preset will show up.

You're trying to play both clips at the same time, so only the second one plays (as it interrupts the first). To play them one after the other you'll need to add a listener to the onFinishedPlaying event on your LipSync component and play the second one when that fires.

 

I can deal with that. Any idea of an ETA though? I'd need to know how this would work before I get too deep into making the character models only to realize I did something to jeopardize my workflow more.

I would totally appreciate that. With that framework, it might not be too difficult to tweak/modify the code (I hope) to allow it to support more layered emotions/etc.




Do you think adding layered emotional data to a given frame and/or blend system could be in the pipeline so that one could plug it in from their own custom "emotion" format perhaps? I was thinking this might be independent of how they might want to display that data each frame. For example, it'd be sort of a pluggable script that lets you take custom emotional data supplied (even if we had to write our own editors for it) and write your own script that would let that custom editor data drive your script (which does whatever it needs to e.g. display the custom data for that keyframe, which would let your system not care whether it's a model, a sprite, or just a model with texture or uv animation because they would all be driven the exact same way). Is this something achievable right now with your code?


One other (final) request I have is to allow the user to enter text into a textbox (instead of just from an audio file) to generate the phonemes for an animation. This would be highly useful if I wanted to, say, grab my dialogue from an Excel file (en-masse) and process it quickly through LipSync Pro, then later tweak the animation to look better on a final pass. This feature alone would make LipSync Pro a true power-house tool for real game development. Right now, I can see it only useful for small games without much dialogue (due to the amount of work a single dev would have to exert for a full game's dialogue to be created manually -- if it could be generated first, then tweaked later, this process could be totally a one-man show, assuming dialogue could be plugged in through text, without needing an audio file to generate it).



Do either of these other two things seem like they'd be something you'd be interested in implementing in some form? -- I know, from my end at least, they would be the sort of features that could make or break the ability (time-wise) to accomplish a dev's dream game in a way that truly matches their vision.

Just something to consider if you would.

 

Not sure if you missed it, but if you could help me with my issue I would really appreciate it! I'm hoping to do a lot of lipsync in the coming days.

https://forum.unity.com/threads/rel...l-animation-tools.309324/page-14#post-3322684

 

Does LipSync Pro / Morph3D work in WebGL, as mine is working in the editor but not in WebGL? No lips are moving.

 

Hi there! Any response would be appreciated!

 

I see, sound good! any estimate date for the version 2? next year I guess?

 

I'll have some extra time tomorrow so I can put it together then. I'll send you a PM with it by the end of the day.

If I understand right, then it could be feasible. The current sprite blend system allows you to add both Sprites and 'Layers' (just separate sprite renderers) to the manager component, then you're given a choice of sprite and layer when adding them to poses in LipSync. This lets you have multiple sprites displayed at once, for separating phonemes and emotions. Does this sound like it could help achieve what you're after?

The idea of LipSync not caring whether it's acting on a model or sprite etc is basically why blend systems exist in the first place. Having them just receive state information like you're describing, instead of being told to just set the value of X to Y is a good idea though, and part of what I'm doing in 2.0 with moving a lot of control out of LipSync and into blend systems. The problem is mainly just dealing with backwards compatibility!

This one is actually on the way! Again, it's a 2.0 feature (sorry!) but it's the one that's probably furthest in development at the moment and I'll be posting more info about it here soon. Basically, there will be an option to run AutoSync on just audio (generating a transcript in the process) or on a manually entered transcript, and a new tool for quickly aligning words for when they don't match.

It is supported, but you'll need to use the "custom timer" timing mode on your characters, as WebGL doesn't work with 'audio playback' mode.

Sorry about that Joel! It looks like it could be a blending problem - can you see what the 'phoneme curve generation mode' is set to on your character? If it isn't already, set it to 'tight' and see if that makes a difference! If that doesn't help, would you be able to send the scene with your character and the LipSyncData clip to me so I can have a look myself? Either a PM here or an email to contact@rogodigital.com, thanks!

Hopefully by early-mid February! I had been hoping to get it done before the end of the year, but progress has been a bit slower than I expected on a couple of features. I'll keep you posted here though!

 

I am having the two "ArgumentOutOfRangeException"s as well. The Character comes from Blender.
One in LipSync.cs:339 and the other in EyeController.cs:270

Code (csharp):

1.  
2. ArgumentOutOfRangeException: Argument is out of range.
3. Parameter name: index
4. System.Collections.Generic.List`1[RogoDigital.Lipsync.Blendable].get_Item (Int32 index) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633)
5. RogoDigital.Lipsync.BlendSystem.SetInternalValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync Pro/Classes/BlendSystem.cs:193)
6. RogoDigital.Lipsync.BlendshapeBlendSystem.SetBlendableValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync Pro/BlendSystems/BlendshapeBlendSystem.cs:60)
7. RogoDigital.Lipsync.LipSync.LateUpdate () (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:339)
8.  

and

Code (csharp):

1.  
2. ArgumentOutOfRangeException: Argument is out of range.
3. Parameter name: index
4. System.Collections.Generic.List`1[RogoDigital.Lipsync.Blendable].get_Item (Int32 index) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633)
5. RogoDigital.Lipsync.BlendSystem.SetInternalValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync Pro/Classes/BlendSystem.cs:193)
6. RogoDigital.Lipsync.BlendshapeBlendSystem.SetBlendableValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync Pro/BlendSystems/BlendshapeBlendSystem.cs:60)
7. RogoDigital.EyeController.LateUpdate () (at Assets/Rogo Digital/LipSync Pro/Components/EyeController.cs:270)
8.  

Happy to mail you the project, if that helps.

 

I've finally got some good news for you @Andy_liu , @Diegoale2000 , @leochou2ha , @Casanuda , @JoMaHo , @kukuhbasuki and anyone else I couldn't find in the thread! AutoSync support for macOS Sierra and higher has been fixed. I'm putting the finishing touches on a new patch today that includes this fix and I'll be submitting it to the Asset Store as soon as possible. My apologies that this has taken as long as it has!

If you could, that would help a lot! The address is contact@rogodigital.com. It looks like the internal list of blendables (blend shapes) has got out of sync somehow, but I'm not sure how that could have happened. One thing you can try is having a script somewhere that calls ClearBlendables() on the blend system in Start() or Awake() and see if that fixes it.

 

@Rtyper a few pages back there was mention of a memory leak in the Late Update when using a Morph 3d Character which totally kills any performance, I just wondered if you'd had time to address it?

 

I'm having trouble setting the emotion with LipSyncPro, can someone help me out? Thanks

 

I don't know if it was specifically a memory leak, but there definitely was a performance issue with MCS characters. Unfortunately, the problem was somewhere in Morph3D's code, not mine, so I wasn't able to fix it. I may be able to improve the extension slightly by not calling into the MCS code when there's no change in blendshape value, but I can't do anything about the performance hit of those calls that do have to happen.

Hi, I was just looking into this after seeing your email but I might as well post here instead! The error means that there's something causing invalid numbers (NaN meaning Not a Number) to be used as the bone's position. These numbers come straight from Unity's animation curves though so I'm not sure why they would be invalid. Would you be able to send your project folder (or a cut-down version where the error still occurs) to the same email address you sent your previous message to? I should be able to track down the cause of the problem then!

 

Thankfully I was able to solve the problem. I used the clip editor to create an emotion and save it in a folder. From there I stored it in an array and used the following code which seems to work.

if (!GetComponent<LipSync>().IsPlaying)
{
GetComponent<LipSync>().Play(emotion_sync[0]);
}

Can lip sync set a default emotion for when no lip syncs are playing or does my code need to manage that?

 

No probs thanks for getting back, i had a feeling you were gonna say that cos its coming from the MORPH3D namespace.

I don't think ill use Morph in that case one character is smashing my dev PC like 11ms per frame

 

Whenever I select "AutoSync > Start (Default Settings)" it gets stuck on "Please wait, recognising phonemes" with no error messages in the console. Is this the MacOS Sierra bug?

edit: I just tried it on our artists PC, it does appear to be the OS, is there an ETA on the release for the fix?

 

Hi Rtyper, is there any progress on this error?:

ArgumentOutOfRangeException: Argument is out of range.
Parameter name: index
System.Collections.Generic.List`1[RogoDigital.ContinuationManager+Job].get_Item (Int32 index) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633)
RogoDigital.ContinuationManager.Update () (at Assets/Rogo Digital/Shared/Editor/ContinuationManager.cs:28)
UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at C:/buildslave/unity/build/Editor/Mono/EditorApplication.cs:127

It is (I guess) first mentioned here on Aug. 2017. I just updated from Lipsync 1.31 to 1.41, and now got this error too. It's starting as soon as Unity starts and counts up continuously. Unity version 2017 . 3 0f3

 

Hey Rtyper, thanks for all the great work.

- Do you have ETA on 2.0?

I am interested in baking the lipsync asset (ideally into the model's FBX for export). If that's possible.

Thx

 

I'm glad you got it sorted, but that still shouldn't be happening! If you still want me to look into it, I'll be happy to. There's no default emotion at the moment, though the normal pose of your model is used as the "default", so it may be possible for you to just modify that instead?

Just came out yesterday! You're right, it was the OS - there's been issues with compatibility since macOS Sierra was first launched and I've just got it fixed now. Just updaing to 1.41 from the store should fix the problem.

Not really, as I was never able to recreate the issue myself, and no-one sent a project for me to investigate. All I can really suggest is to make sure you followed the import instructions (including opening a new scene and deleting the old version first) when you updated.

No problem! I'm still aiming at (probably late now) February or just early March for 2.0, at least for a beta version. It'll definitely be possible to bake LipSync assets into standard Unity animations, though all it currently supports exporting to is a Unity-only .asset file. I know Unity has recently added some interesting functionality for exporting models back into 3DS Max, so I'll have to look into how this works a bit more. It may be possible to get .fbx export in there too at some point!

 

[QUOTE="
↑
Hi Rtyper, is there any progress on this error?:

ArgumentOutOfRangeException: Argument is out of range.
Parameter name: index
System.Collections.Generic.List`1[RogoDigital.ContinuationManager+Job].get_Item (Int32 index) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633)
RogoDigital.ContinuationManager.Update () (at Assets/Rogo Digital/Shared/Editor/ContinuationManager.cs:28)
UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at C:/buildslave/unity/build/Editor/Mono/EditorApplication.cs:127

It is (I guess) first mentioned here on Aug. 2017. I just updated from Lipsync 1.31 to 1.41, and now got this error too. It's starting as soon as Unity starts and counts up continuously. Unity version 2017 . 3 0f3
Click to expand...
Not really, as I was never able to recreate the issue myself, and no-one sent a project for me to investigate. All I can really suggest is to make sure you followed the import instructions (including opening a new scene and deleting the old version first) when you updated.
[/QUOTE]

@Rtyper As someone reported before, the error disapeared after about a feeled count of a million, after just workinng on. But it made the editor slow. Sending you the project is a little difficult, 65 GB in size .

 

Now had this error 2 times more and other errors. Quite often. It might has to do with adding new emotions while in Clip Editor.

 

Hi,

Beforehead: i'm a hobbyist. Even as the following list of problems may look somewhat dramatic given it's length and the number of problems, it's not time critical for me myself. Most of the bugs are simply annoying, but nothing stopping progress.

Something general: It looks like the mail server of the contact-mail does not like Microsoft's @Live-addresses. It looks like my mails from mid-december (and my mail from about a week asking if it got through) got eaten somewhere in between. For a long time, i thought the devs were simply on holidays, so did not bother much about it.

As the new version suggested only my initial -shallow- report via rating got through and the devs solved some of the problems described there, i searched for an alternative and found the forum-link. So i'm reposting the portions that still apply to version 1.4.1 in this place:

Under some circumstances, the synchronization from Blendshape-Manager to Lip Sync fails
a) I rename a blendshape, but do not confirm it with enter. If i just leave focus after editing, lip sync thinks of it as <Name> minus 1 char from the end.
b) Should i change a blendshape name by pasting it from somewhere else, there won't be an update at all. It still refers to it with the old name [so a special case of a with Strg-v as the ominous one character]
c) If i delete a blendshape without doing anything further, lip sync thinks of it as still existing. That's kind of bad because it still thinks of the old ids. So, any blendshape inserted into the emotion that had a higher id will slide down by one (or more if more deletions) to blendshapes that were not selected before.
There seem to be more edge cases, however i failed to reproduce them by force. So they seem to depend on non-trivial constellations of previous inputs:
d) At least once a desynced state let me select blendshapes of a SkinnedMeshRenderer no longer present in the Blendshape-Manager

At least once a desynced state resulted in a reduction of blendshapes inside an emotion as described in the initial post (#n going down to #n minus something and breaking the ui). This seems to be an edge case as it defied all tries of clean reproduction. Perhaps it's a rather random sideffect of the numerous other exceptions and problems.

A very reproducible problem that somewhat widens the problem above: If by any chance, a lip sync emotion does refer to a blendshape whose ID is bigger than the maximum ID currently inside the blendshape manager (or the state lip sync thinks it does), a Out of Range Exception occurs
This leads to 2 problems (or better: problem areas)
1. When it happens, it crashes the display of blendshapes inside a emotion. On selecting the emotion, nothing follows. The next Component is drawn straight afterwards.
Unity refers to that with the following Warnings:
Unexpected top level layout group! Missing GUILayout.EndScrollView/EndVertical/EndHorizontal?
UnityEngine.GUIUtilityrocessEvent(Int32, IntPtr)

Unexpected top level layout group! Missing GUILayout.EndScrollView/EndVertical/EndHorizontal?
UnityEditor.ContainerWindow:InternalCloseWindow()
Surrunded with one or more of these exceptions:
IndexOutOfRangeException: Array index is out of range.
RogoDigital.Lipsync.AdvancedBlendshapeBlendSystem.SetBlendableValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync Pro/BlendSystems/AdvancedBlendshapeBlendSystem.cs:56)
LipSyncEditor.OnInspectorGUI () (at Assets/Rogo Digital/LipSync Pro/Editor/LipSyncEditor.cs:661)
UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor[] editors, Int32 editorIndex, Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1240)
UnityEngine.GUIUtilityrocessEvent(Int32, IntPtr)
Variant: Last line is "UnityEditor.ContainerWindow:InternalCloseWindow()" instead.

IndexOutOfRangeException: Array index is out of range.
RogoDigital.LipSyncEditorExtensions.DrawShapeEditor (UnityEditor.Editor target, RogoDigital.Lipsync.BlendSystem blendSystem, System.String[] blendables, Boolean useBones, Boolean allowCreationFromAnimClip, RogoDigital.Lipsync.Shape shape, System.String name, Int32 id, System.String invalidError) (at Assets/Rogo Digital/LipSync Pro/Editor/LipSyncEditorExtensions.cs:135)
LipSyncEditor.OnInspectorGUI () (at Assets/Rogo Digital/LipSync Pro/Editor/LipSyncEditor.cs:461)
UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor[] editors, Int32 editorIndex, Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1240)
UnityEngine.GUIUtilityrocessEvent(Int32, IntPtr)
2. Lip Sync previews emotions on my character. That's fine and very valuable.
But it only unloads them if i close the emotion line by clicking it again or if i switch tabs (e.g. from emotions to gestures).
If focus of Lip Sync is lost before (e.g. by selecting another node in the hierarchy), the preview stays applied to the character.
It then is only unloaded on clicking the relevant emotion twice (one time opening it, one time closing it properly).
This behaviour opens door to a range of possible disfunctions leading to permanent emotionalization.
a) Should the ids get mapped to different blendshapes after restoring focus, it can no longer be unapplied because the definition of what to unapply is different from before. That can happen by selecting a emotion (so, previewing it), importing different blendshapes into the blendshape manager and then returning to lip sync. As soon lip sync registers the changes inside the blendshape manager, it reallocates the ids to their new blendshapes on that position. The emotion is still displayed because it was previously selected, but as the content is now different, it can no longer be unapplied.
b) Should an out of range exception happen, the list can no longer load and therefore, it can no longer unapply. The character stays "emotionalized" until re-inserted or it's blendshapes get adjusted by hand.


Then my new findings about version 1.4.1:
Compile-Error inside PhonemeSet.cs / struct Phoneme
Fails to compile: this can only be used once all fields are initialized
git between 1.4 and 1.4.1 shows the following diff:
- public Phoneme (string name, int number, int flag) : this() {
+ public Phoneme (string name, int number, int flag) {
... undoing that change resolves the problem.

Emotions stays in place when selecting an emotion, then leaving the inspector: Still present

New additive mode for emotions:
First of all: Thank you for implementing such a thing as that opens many new possibilities in my case.
It's slightly bugged on reload:
1) Put a mixed emotion and configure it as additive
2) Save and Close LipSync-Window
3) Open LipSync-Data again
4) The mode is now again "normal", but the values remain as configured. Should any value be adjusted, the others will be adjusted too.
Workaround: Apply the "additive" Mode every time before editing the mixed emotion.

New search system
The new design resolves the previous problem of similar names. Thank you.

Running it on Unity 2017.3 with Player configuration .NET 3.5 Equivalent / Mono / .NET 2.0 suggests some kind of memory leak.
Most of my characters involve a big number of blendshapes. Often 800, sometimes 1100 or more.
Initial typing in the search field is fast.
However, the longer one uses the search function, the more unresponsive it gets. At first it doesn't particularly stress my Skylake-i7. Therefore, typing is not slowed down.
At some point i found that the longer i typed (the length of searched text is irrelevant, it also occurs if the field is regularly emptied), the more RAM Unity needed and the more CPU was needed when typing. After some time, it hits about 1 char per second.
It does not seem like Unity is able to regain memory lost this way, as it only goes down when restarting Unity.
It seems rather random how fast and heavy this effect kicks in (probably some garbage collection magic), but in general, memory and cpu usage go straight up over time. In one case that got me to "1 char per second" really fast, in the following reproduction of the problem it occurred much later.

Also, as some sort of side-effect the new search mechanismn has a different approach of loading presets that do not exactly match the present blendshapes.
In the previous version, loading a preset of a similar character using similar named blendshapes (e.g. same blendshape on different but similar named clothing, so different prefix) sometimes auto-magically found rather good matches. The current version seems to look for an exact match, or falls back to using the pure index. Resulting in much less intelligent results. That's no showstopper for me as the old results also needed adjusting, but something that should be considered.

Something more relevant: When falling back to using a pure index, it's possible to provoke an overflow if the blendshape resided on an index that's no longer inside bounds (because the list itself got smaller).
"Array index is out of range. RogoDigital.Lipsync.AdvancedBlendshapeBlendSystem.SetBlendableValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync Pro/BlendSystems/AdvancedBlendshapeBlendSystem.cs:56)
LipSyncEditor.LoadPreset (System.Object data) (at Assets/Rogo Digital/LipSync Pro/Editor/LipSyncEditor.cs:951)
UnityEditor.GenericMenu.CatchMenu (System.Object userData, System.String[] options, Int32 selected) (at C:/buildslave/unity/build/artifacts/generated/common/editor/GenericMenuBindings.gen.cs:119)"

For future problems i will just post to this forum instead of relying on email.

 

I'm having this issue with Lipsync Pro in Unity 2017.3.0p2:

ArgumentOutOfRangeException: Argument is out of range.
Parameter name: index
System.Collections.Generic.List`1[RogoDigital.ContinuationManager+Job].get_Item (Int32 index) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633)
RogoDigital.ContinuationManager.Update () (at Assets/Rogo Digital/Shared/Editor/ContinuationManager.cs:28)
UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at C:/buildslave/unity/build/Editor/Mono/EditorApplication.cs:127)

It just keeps doing the error infinitely.

it's very easy to reproduce:
1. Install Lipsync Pro into an empty project.
2. Load Example_01_Lincoln scene.
3. Go to Window->Rogo Digital->LipSync Pro->Open Clip Editor. Open the clip editor in a floating window.
4. Play the scene.
5. Stop the scene.
6. Errors start streaming.

It will also happen if you exit a project with the Clip Editor open and reload, so that the clip editor is already open.

 

Hey Rtyper.

I'm very interested in Lip Sync Pro. I'm developing a 2d game using sprite meshes, using mesh deformation to manipulate and animate the shapes of my characters. By doing this, I can trigger facial animations in 2d with different animation layers, so blinking, lip movement, and emotions can all be handled with different weights in the animator. I'm trying to create 2d lip sync that looks animated, not frame by frame with sprites, and I'd love to be able to control each phoneme with an animation layer using the animator instead of frame by frame. This way, there would be smooth transitions between phoneme shapes, like you'd normally see with blend shapes. I've seen a few people ask about using the animator in place of blend shapes, and I'm wondering if that has made it's way into release, or if it's just a fun idea people have played around with. Thanks for your time.

 

This is really confusing me now, it's clearly a problem for several people but I just can't get it to happen at all. Even following your steps @ceebeee fails to cause any errors on my machine. I've made a slight change to the ContinuationManager class that I'm hoping will prevent the errors, but I still can't figure out the actual root cause...
The script is attached to this post, can you please try replacing the existing version in Rogo Digital/Shared/Editor with this one and let me know if you still get the errors!

I think I have found this other error though. Adding a new emotion with the clip editor open then adding that and an existing emotion to the clip with preview mode on, all without selecting the character again will produce ArgumentOutOfRangeExceptions because the character hasn't had that emotion copied to it yet. I'll see if I can make this fail better in a patch.

It's an interesting idea! That should be possible using a custom Blend System - there isn't one built in to LipSync, but it definitely sounds possible. There's some info on creating them here, but for this specific one you'd need to have the blend system keep a reference to an Animator, and an array of the layers (or possibly Animator Parameters if you wanted to do it through blend trees). The blend system would then report the layer/parameter names to LipSync as blendables, then set the values on the Animator when called to (through SetBlendableValue). If you go ahead with it and need any help, just let me know.

@JX272 I'll have to have a look at my email settings because you're right, I didn't see any messages from you. If you give me a few minutes I'll try and look into the issues you've reported there and then go through it all in a separate post. Thanks![/QUOTE]

 

Thanks for your response, I have another question, will LipSync be able to map the phonemes in songs for use in singing?

 

Hi, I tried the posted script. It does seem to resolve the issue. I can't get it to produce the errors anymore.

 

@JX272 Thanks for your patience, sorry this took longer than expected! I found your emails in my spam folder too. My bad for not checking it - I've added your email to my whitelist so hopefully it won't happen again.

Thanks for pointing this out - the advanced blend shape system is pretty new so I'm not too surprised there are edge cases like this that break it! I'll try and get these fixed asap.

Yes, this is related to the previous issue. I'll make sure to add a check in the blend system to make sure the indices are in range.

This one is interesting - the second half makes sense and ties into the same problem above with the advanced blendshape blend system not handling changing blend shape lists well, but I don't know why the preview isn't being restored when you deselect the object with the LipSync component on. That's something I've had in the asset from the very beginning, and just checking it now (2017.1, same blend system) it works as expected for me. Does this just happen all the time for you, or are there some circumstances where deselecting the game object does unapply the pose? What about on one of the example scenes?

This compiles just fine for me, and though I'd have to double check, I don't actually recall changing this class in 1.41! It may seem obvious, but did you follow the import instructions and delete the previous LipSync Pro version when updating?

Fixing this right now! I'll send you the altered file(s) when it's done and get it into a patch soon.

I have just tried this myself, and though I didn't find any actual slow down occurring, I did notice an increase in memory used while typing in the box that was not regained (at least not in full) afterwards. I'm really not sure of the cause just yet though. My editor code doesn't do anything with the string until the search button is pressed, it's not even checking for changes each frame any more so that will definitely take a bit more time to investigate!

The preset loading code never actually used the old search system, and hasn't been changed in the last version either, so I'm not sure how this could have changed at all. When loading presets, it simply removes the index in brackets from the name (if it's there) and then tries to match it exactly. If it can't match by name it just falls back on loading whatever is in the same index.

This will be partly solved by just adding the extra checks into the blend system that I mentioned above, but I think I'll also stop blendables at invalid indices from being added from presets anyway to make this cleaner.

Thanks for all the feedback anyway! I'm still working on 2.0 at the moment, but I'll try and get a 1.x patch out this week to fix those issues.

 

Hiya

I can also confirm that the Continuation Manager script fixes the ArgumentOutOfRangeException error.

A problem we are experiencing though is that the auto lip sync produces terrible results generating very few phonemes for the first 30 seconds of audio then after this the problem corrects itself. I've tested this with an audio clip which repeats itself after 30 seconds and it reliably only produces sensible results in the latter half of the clip.

Any thoughts?

 

@Rtyper: As i got some time, i checked your response. My observations are following.
(I do not expect you to respond during the weekend. A reply next week is perfectly fine for me)

@Problem "Lip Sync previews emotions on my character. That's fine and very valuable. But it only unloads them if i close the emotion line by clicking it again or if i switch tabs (e.g. from emotions to gestures).":
I'm using Unity 2017.3 right now, but at that time i also observed it in Unity 2017.2. Can't say about .1. I started using Lip Sync with .2.
I'll try to be as concise as possible.
1. I loaded Example 04. No other scenes loaded.
2. Select "Lincoln_Char"
3. In Inspector
3.1 Select Tab "Emotions"
3.2 Select "Eyebrows Up Emotion". This expands it and shows a preview on the character.
4. Select "Lincoln_Char/Beards".
The emotion stays visually in place. Skinned Mesh Renderer also shows the corresponding values as non-zero.
5. Repeat step 2 to 3.2
6. Unfold "Eyebrows Up Emotion" by clicking it again
Now the pose is correctly unapplied.

An important observation: This seems to be dependent on what emotion i select. Selecting the first (Happy Emotion) or second (Sad Emotion) does indeed reset the pose as soon as "Lincoln_Char" looses focus. Doing it with the third or fourth one however, behaves like explained above. In my project, it always behaves like the third or fourth one. As i defined custom emotions inside the project settings and do not use the first four emotion-slots, it could simply be an index-based problem.
Or a problem with the origin of a pose. In my case, the blendshape controller resides on top of my character-prefab. The nodes with the Skinned Mesh Renderers holding the blendshapes however are two levels below it.

Update: It seems to be related to the blendshapes used. Given enough zoom it becomes obvious that even the first emotions do not *fully* reset on losing focus. The very obvious mouth-movement does, but the much less apparent brow-blendshapes do not reset on losing focus.

------------------------------------------------------------------------------------------------------------------------------------------------------------
@Problem "Compile-Error inside PhonemeSet.cs / struct Phoneme":
Yes, i deleted the folder before installing 1.41.
It's weird indeed. I double checked against my archived version of 1.4 and yes, the file is 100% identical.
As i think about it: I give it a chance i changed it myself in 1.4 to be able to run. I remember something like that.
The problem is very real for me. Compiling the original gives me the following errors:
Assets\Rogo Digital\LipSync Pro\Classes\PhonemeSet.cs(51,5,51,9): error CS0188: The 'this' object cannot be used before all of its fields are assigned to

Assets\Rogo Digital\LipSync Pro\Classes\PhonemeSet.cs(50,11,50,18): error CS0843: Backing field for automatically implemented property 'RogoDigital.Lipsync.PhonemeSet.Phoneme.name' must be fully assigned before control is returned to the caller. Consider calling the default constructor from a constructor initializer.

Assets\Rogo Digital\LipSync Pro\Classes\PhonemeSet.cs(50,11,50,18): error CS0843: Backing field for automatically implemented property 'RogoDigital.Lipsync.PhonemeSet.Phoneme.number' must be fully assigned before control is returned to the caller. Consider calling the default constructor from a constructor initializer.

Assets\Rogo Digital\LipSync Pro\Classes\PhonemeSet.cs(50,11,50,18): error CS0843: Backing field for automatically implemented property 'RogoDigital.Lipsync.PhonemeSet.Phoneme.flag' must be fully assigned before control is returned to the caller. Consider calling the default constructor from a constructor initializer.

As you do not get this error, some more details about my environment: As i'm just a hobbyist and my project does not justify bloating my computer with Visual Studio, i'm using the lightweight MonoDevelop (5.9.6) provided by Unity. Unfortunately i wasn't able to see what compiler this Unity-customized version of MD is using. But it seems like it's compiler isn't as smart as the one VS uses.

An alternative (probably, also a more sane) approach to resolve this error would be to make the "struct Phoneme" a "class Phoneme". That's perfectly understood by MonoDevelop.
------------------------------------------------------------------------------------------------------------------------------------------------------------
@Problem "memory leak": As you described there should be nothing, I did some more testing to try to isolate that from pure Unity weirdness. My observations so far:
1. While keeping any key pressed inside the field one of my CPU-cores spikes to 60% and Memory goes up continuosly. On Unity 2017.3 only about 50% of it is ever regained. BUT this seems to even happen on a very fresh project with zero assets in it. So it's rather a leak inside Unity itself.
2. Then i tried something different:
2.1 I opened the Blendshape Manager and expanded it's list of blendshapes. In my case, that's a very, veeeery long list with 800 to 1200 entries.
2.2 I typed inside the search field. Yes, indeed it slowed to a crawl.
2.3 I folded the list of blendshapes.
2.4 Typed in the search field again. Normal speed again.

So, it's some kind of interference with these many other fields.

I never saw something like this with the previous version 1.4 on Unity 2017.3.
So i can only guess that some change inside the blendshape manager made it's fields to listen to other things as soon as they are visible.

But having found that, i can workaround it now, should there be no apparent reason inside Lip Sync. Unity 2017.2 and .3 are a mess themselves. Unity .2 and above keep using one of my 8 cores at 100% for literally looking at a super-fresh zero-asset project explorer not even showing the Scene or Game-Tab. Pff.....
------------------------------------------------------------------------------------------------------------------------------------------------------------
@loading preset-behaviour: Hmmmmmmmmmmmmmmmmmmmmm ...
I had some serious thinking about your description how it always worked.
That lead me to the following: It could have been mathematical coincidence. Combined with luck beforehand and ... less luck afterwards.
It could be because of the properties of my characters: The characters always have the same number of blendshapes because i define what to export into a blendshape and what not. Worn clothing gets the same count of blendshapes (because it's complementary). So (in cases where i do not include clothing-specific blendshapes), it's indeed the case that for a length L of blendshapes on the character, the blendshapes corresponding to each other all have the same distance L. As these are named also named similary, that lead to thinking of it as an intelligent matching by parts of the name. As the previous search system also worked rather fuzzily, i got that confused with each other.
A rational explation for the emergence of this problem could be that i increasedly used additional clothing-blendshapes and therefore distorted this fragile balance. The distance between matching blendshapes was no longer L. That could have coinceeded more or less with the release of 1.41.

So, i drop this point and know what to do against it. Thanks for clarification.

--------------------------
Given that state, it should suffice if these points are kept in mind for version 2. I can work around with version 1 up to then.

 

Hi @Rtyper . I noticed that there's a lot of 'Left Eye Dummy' and 'Right Eye Dummy' objects appearing in the root of the hierarchy when I use the eye controller. Can't say I've noticed this before, so I assume this might be a bug that's been introduced recently?

 

Further to this I tried running the autosync on a different computer and got different results. These seemed to be slightly better but still had the noticeable change after 30 seconds. I then delinked SoX for each machine and re exported the audio so that it no longer required converting and I now get consistent results between the two.

Now when using the default settings for autosync it finds all the phonemes but the mouth motion is very sharp and discrete. If I use the high quality settings the motion is much smoother and believable in some sections and entirely missed in others.

Enabling the allphone_ci setting drastically reduces the amount of time required for setting the phonemes (from a few minutes to maybe 3 seconds). I've tried playing with the other values but I couldn't see any documentation for them.

Any help on this would be appreciated. Thanks

 

hey i get all time a console error and he never stops i need to restart unity all time

 

Have you used the updated Continuation Manager?

https://forum.unity.com/threads/rel...l-animation-tools.309324/page-14#post-3364783

 

Actually I can confirm I'm also getting this but it's intermittent. It either spams or it doesn't not sure what sets this off yet.

 

Could you try the replacement file attached to this post? It just needs to replace the existing version of ContinuationManager.cs in Rogo Digital/Shared. Edit: @SticklandPWorm got there first!

Are you on the latest version of LipSync Pro? There were some issues with clips over 30 seconds in an older version due to how AutoSync splits clips into 30 second chunks and recombines the data afterwards. The other parameters that are exposed are directly from the CMU Sphinx library that is used for phoneme recognition (so weren't created by me) - beam and pbeam are related to the "search width" and are for balancing accuracy against performance. Playing around with them and seeing how the results are affected has been the approach I've used so far, as they're not well documented at all by the library's authors.
There's a new version of AutoSync in 2.0 which doesn't use this library, I'll be posting more info about this very soon.

Since the last update, Eye Controller re-parents your chosen left and right eye transforms at runtime to these new dummy objects it creates in their place in the hierarchy. It's to allow for the forward axis choice that was added.
I haven't found an issue with them though, are they staying after you leave play mode? There should only ever be 2 dummy objects in the scene, and they should disappear when you go back to edit mode.

 

Even with the modified ContinuationManager script?

 

Yes after I added the new ContinuationManager.

Initially I was only using phonemes and the new ContinuationManager seemed to fix the error. Then once I added Emotions I started running into problems again. I tried deleting the default emotions and replacing them with my own, I was able to remove them in the Edit Emotions dialog but then was not able to remove them from the main dialog with the Delete Pose button.

When I'm using the clip editor (which was working with the new ContinuationManager and phonemes before) it will generally work for the first emotion placement but then after that will no longer allow me to preview the animation or add more markers. I tried to set the character back up in case it was the initial removal of the default emotions causing the problem but it has persisted.

 

@Rtyper It looks like I get one pair of eyes for each character/prefab. They do disappear when I leave play mode, so it's not a huge problem. If there were just two dummy objects I wouldn't mind, but it clutters the hierarchy with ~10 eyes, and I will add more characters eventually.

 

Thanks for all this detail - I can confirm this happens for me as well, so I can look into it now. I've got several fixes ready to be published now, so I'll hold off for a few more days and get as many of these into it as well.

Just so that I'm clear, are you saying that the version with ": this()" in it was the older version or the newer one? I'll have to go back and check an older version, but I don't believe I added that at any point. The addition of that to the line would allow it to compile even if it thinks not all the variables have been assigned (even though they are...).

As far as I'm aware, the IDE you use doesn't have any effect. The Unity editor handles compilation, not the IDE so that shouldn't be a problem. Got to say, this one's still confusing me!

Something within Unity was my thought as well, though it is possible there's some code in the wider LipSync editor script that could be causing it. I'll need to check if the same thing happens within Eye Controller.

OK. That is interesting though - some more intelligent matching system might be feasible as well. It definitely won't make the 1.42 patch, but it's a possibility for 2.0.

I've got some free time tomorrow, so I'll try to get back to you on the first couple of points before the end of the week. Thanks!

 

Yes I am on the most recent version of lipsync pro. Thank you for the information on beam and pBeam etc.

 

Problem "Compile-Error inside PhonemeSet.cs / struct Phoneme":
What i wanted to say: No, probably the "this()" was not there in any version. By now i'm rather sure i've added it myself when i began using LipSync (Unity 2017.2 as of then) because it did not compile. As it was a rather minor fix, i quickly forgot about it.
If the compilation is done by Unity itself, that would point to a change in Unity itself.
Guess i'll fire that one into a fresh VM and try it with an old Unity 2017.1 to see if that changes things.

Update: Unity 2017.1 with MonoDevelop, using standard project settings in an empty project results in the same errors. More precisely: Player: Scripting Runtime Version: Stable (.NET 3.5); Scripting Backend: Mono; API Compatibility Level: .NET 2.0 Subset.
My current project does not use .NET 2.0 Subset but .NET 2.0, so Subset should be irrelevant.

I never used Unity with Visual Studio, but perhaps using VS would not result in the Mono-Backend? Mono vs. .NET could explain such a difference.

 

Hey there @Rtyper ,
Thanks for such a great asset. Purchased Pro (v1.41) last week (working in Unity 2017.3of3) and have been configuring my character with it the last few days. I'll preface by saying I'm new to the asset so I haven't exhaustively tried to debug my issues, but the current @Problem are making it impossibly slow to do all the dialog in my game. I can second the emotion editor bugs @JX272 is experiencing, but the main question I have relates to AutoSync. I've installed SoX, verified in the the AutoSync settings, made sure to troubleshoot with a 16khz mono .wav file (< 30sec), but still when I click Process, it hangs on "Please Wait, Recognising Phonemes." Closing/Re-opening LipSync doesn't halt the process; I must Force Quit Unity, re-open my project, and reload the .asset to continue working. Any help would be much much appreciated as I don't believe we can move forward with this volume of dialogue without a rough AutoSync pass getting us in the ball park. Thanks so much! And if it helps, we're getting hundreds of Array Index errors pushed to console:

Array index (2) is out of bounds (size=2)
UnityEngine.SkinnedMeshRenderer:SetBlendShapeWeight(Int32, Single)
RogoDigital.Lipsync.AdvancedBlendshapeBlendSystem:SetBlendableValue(Int32, Single) (at Assets/Rogo Digital/LipSync Pro/BlendSystems/AdvancedBlendshapeBlendSystem.cs:59)
RogoDigital.Lipsync.LipSyncreviewAtTime(Single) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:681)
LipSyncClipSetup:UpdatePreview(Single) (at Assets/Rogo Digital/LipSync Pro/Editor/LipSyncClipSetup.cs:3841)
LipSyncClipSetup:Update() (at Assets/Rogo Digital/LipSync Pro/Editor/LipSyncClipSetup.cs:545)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions()

 

I'll have to test it out with MonoDevelop then. Mono is still instead of .net used with VS (as .net is only supported on Windows and Unity is cross-platform), but I can't rule out some kind of mono weirdness! I'm including the addition of :this() to that constructor in the patch though, so hopefully this won't affect anyone else.

No problem! I've attached another replacement ContinuationManager.cs to this post that should fix the problem completely. Another user pointed out a very basic flaw that I'd overlooked in the class, so I've included their fix in it now. This should be some use to you as well @Salja .

 

When are you getting those errors? They're related to the LipSync component (and blend system), not AutoSync. It's likely they'll be solved by the changes I'm making based on what JX272's said.

Your AutoSync issue is strange though. Are you on Windows or Mac? If you're on Windows, can you try installing the Visual C++ runtime redistributable that's in the extensions window (Window/Rogo Digital/Get Extensions), restart your PC and let me know if the same problem is occurring. If it is, are there any errors in the console that appear when you click the AutoSync button?

 

Thanks for the update. I've used the new script but still have problems. The lipsyncing behaves but when I add an emotion, the model no longer plays anything in the preview. If I click on the real-time preview button, switch it to No Preview and then back to select my model both options disappear and the entire play bar from the bottom of the window disappears.

My First console error is:
IndexOutOfRangeException: Array index is out of range.
LipSyncController.OnEnable () (at Assets/Scripts/LipSyncController.cs:27)
UnityEngine.Timeline.ActivationMixerPlayable:OnPlayableDestroy(Playable)

 

That error looks like it's in one of your own scripts. LipSyncController.cs isn't part of LipSync, so I can't really debug it.

 

Opps that's embarrassing sorry, I meant to paste this:

ArgumentOutOfRangeException: Argument is out of range.
Parameter name: index
System.Collections.Generic.List`1[RogoDigital.Lipsync.Blendable].get_Item (Int32 index) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633)
RogoDigital.Lipsync.BlendSystem.SetInternalValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync Pro/Classes/BlendSystem.cs:191)
RogoDigital.Lipsync.BlendshapeBlendSystem.SetBlendableValue (Int32 blendable, Single value) (at Assets/Rogo Digital/LipSync Pro/BlendSystems/BlendshapeBlendSystem.cs:60)
LipSyncEditor.OnInspectorGUI () (at Assets/Rogo Digital/LipSync Pro/Editor/LipSyncEditor.cs:687)
UnityEditor.InspectorWindow.DrawEditor (UnityEditor.Editor[] editors, Int32 editorIndex, Boolean rebuildOptimizedGUIBlock, System.Boolean& showImportedObjectBarNext, UnityEngine.Rect& importedObjectBarRect) (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorWindow.cs:1242)
UnityEditor.PopupCallbackInfo:SetEnumValueDelegate(Object, String[], Int32)

 

Thanks so much for getting back to me. I'm on a Mac running the most current "Sierra" OS (Not High Sierra). Although if you'd suggest working LipSync on PC instead I can easily install it there. I'll install the redistributable and let you know if I get errors printed to console.

As far as the other bug, would you be able to give me an idea of when those fixes will be available? Just want to make some 11th hr exec decisions on our game. Thanks a lot


EDIT: @Rtyper , the errors are getting thrown anytime I select or deselect a Phoneme, when I click the Emotions tab, and when I switch between the Phonemes and Emotions tabs.=