[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

I have an additional issue, it seems that emotions do not blend smoothly when changing clips, For example on the Demo scene if you press the Talk button it will start speaking, if you press it again, the emotion will abruptly change. Emotions should blend smoothly between different lipsync clips. Is there a configuration that can help me with this_

 

Further more, I keep observing some problems, on the video you can see that when playing different clips with emotions, if one clip is interrupted and you start playing another one (time 0:10 of the video), the emotion will not go back. This is even if you have the Keep Emotion When Finished flag unchecked.

Additionally if you try to manually reset the emotion with the ResetEmotion(float time) it wont reset it (time 0:27 of the video). The only option is to use ResetDisplayedEmotions but this will abruptly reset it and doesnt look good.

I hope you can take a look at this and the previous issues mentioned before and help me with this.

I appreciate it, Thanks

 

You are Copying Lipsync Component, its not allowed right now. You should add a new fresh LipSync component.

 

Seeing the following error when trying to install autosync on MAC with Unity 2019.4

"Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/Modules/Montreal Forced Aligner/ASMontrealPhonemeDetectionModule.cs(44,24): error CS0115: 'ASMontrealPhonemeDetectionModule.ProcessWithTemplates(LipSyncData, AutoSync.ASProcessDelegate, PhonemeMarker, EmotionMarker)': no suitable method found to override"

Any guidance on how to fix this? Thanks

 

Woah I really like the hair used, did you do this?

 

AudioClip ModelClip= ModelLip.audioSource.clip;
float dialougelengthFloat = ModelClip.length;
List<PhonemeMarker> markers = new List<PhonemeMarker>();
List<EmotionMarker> Emotion = new List<EmotionMarker>();
List<GestureMarker> GestureAnim = new List<GestureMarker>();
int k = 0;
Phoneme phoneme = 0;
int phonemeNum=0;
for (float i = 0.01f; i <= dialougelengthFloat; i += .008f)
{
int j =RandomInt.Random(1,9);
PhonemeMarker phonemeMarker = new PhonemeMarker(tempPhonemeState, i, 1f,false);
markers.Add(phonemeMarker);
// ModelLip.GetComponents<AutoSyncDataReady>();
// AutoSyncDataReady(ModelLip.audioSource.clip, phonemeMarker);
int z = RandomInt.Random(0, 9);

if (z == 0)
{
tempPhonemeState = Phoneme.AI;
phonemeNum = 0;
}
else if (z == 1)
{
tempPhonemeState = Phoneme.CDGKNRSThYZ;
phonemeNum = 1;
}
else if (z == 2)
{
tempPhonemeState = Phoneme.E;
phonemeNum = 2;
}
else if (z == 3)
{
tempPhonemeState = Phoneme.FV;
phonemeNum = 3;
}
else if (z == 4)
{
tempPhonemeState = Phoneme.L;
phonemeNum = 4;
}
else if (z == 5)
{
tempPhonemeState = Phoneme.MBP;
phonemeNum = 5;
}
else if (z == 6)
{
tempPhonemeState = Phoneme.O;
phonemeNum = 6;
}
else if (z == 8)
{
tempPhonemeState = Phoneme.U;
phonemeNum = 8;
}
else if (z == 9)
{
tempPhonemeState = Phoneme.WQ;
phonemeNum = 9;
}
if (phonemeNum ==7)
{
phoneme = 0;
phonemeNum = 0;
}
phoneme++;
phonemeNum++;
k++;
}
lipSyncData.name = ModelLip.audioSource.clip.name;
lipSyncData.clip = ModelLip.audioSource.clip;
lipSyncData.length = ModelLip.audioSource.clip.length;
lipSyncData.phonemeData = markers.ToArray();
float EndTime= 0.01f;
float StartTime;
for (float i = 0.03f; i <= dialougelengthFloat; i += .06f)
{
EyesState tempEyesState = EyesState.CloseEyes;
int l = RandomInt.Random(0, 2);

if (l == 0)
{
tempEyesState = EyesState.CloseEyes;
}
else if (l == 1)
{
tempEyesState = EyesState.CloseEyes;
}
StartTime = EndTime+i+.08f;
EndTime = StartTime+0.001f;
EmotionMarker emotionMarker = new EmotionMarker(tempEyesState.ToString(), StartTime, EndTime, 0.025f,0f,false,false,false,false);
Emotion.Add(emotionMarker);
}
EndTime = 0.01f;
for (float i = 0.03f; i <= dialougelengthFloat; i += .06f)
{
EyesState tempEyesState = EyesState.LookingAround;
int l = RandomInt.Random(0, 3);

if (l == 0)
{
tempEyesState = EyesState.LookingAround;
}
if (l == 2)
{
tempEyesState = EyesState.EyebrowsUp;
}
else if (l == 1)
{
tempEyesState = EyesState.LookingAroundRight;
}
StartTime = EndTime + i + .10f;
EndTime = StartTime + 0.01f;
EmotionMarker emotionMarker = new EmotionMarker(tempEyesState.ToString(), StartTime, EndTime, 1f);
Emotion.Add(emotionMarker);
}
for (float i = .01f; i <= dialougelengthFloat; i += .4f)
{

AnimState tempState = AnimState.Talking;
int j = RandomInt.Random(0, 2);

if (j == 0) {
tempState = AnimState.TalkingTwo;
}
else if (j == 1)
{
tempState = AnimState.Talking;
}
// GestureMarker emotionMarker = new GestureMarker(tempState.ToString(), i);
// GestureAnim.Add(emotionMarker);
}
lipSyncData.emotionData = Emotion.ToArray();
lipSyncData.gestureData = GestureAnim.ToArray();
lipSyncData.transcript = dialouge.text;
ModelLip.Play(lipSyncData, 0f);

}

 

NullReferenceException: Object reference not set to an instance of an object
RogoDigital.Lipsync.LipSync.SetEmotion (System.String emotion, System.Single blendTime) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:611)
EmotionSetToModel.GetlipSync (System.String Emotionstring) (at Assets/Game/Script/EmotionSetToModel.cs:69)
EmotionSetToModel.TaskOnClick () (at Assets/Game/Script/EmotionSetToModel.cs:40)
UnityEngine.Events.InvokableCall.Invoke () (at C:/buildslave/unity/build/Runtime/Export/UnityEvent.cs:166)
UnityEngine.Events.UnityEvent.Invoke () (at C:/buildslave/unity/build/Runtime/Export/UnityEvent_0.cs:58)
UnityEngine.UI.Button.Press () (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/Button.cs:66)
UnityEngine.UI.Button.OnPointerClick (UnityEngine.EventSystems.PointerEventData eventData) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/Button.cs:108)
UnityEngine.EventSystems.ExecuteEvents.Execute (UnityEngine.EventSystems.IPointerClickHandler handler, UnityEngine.EventSystems.BaseEventData eventData) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/ExecuteEvents.cs:50)
UnityEngine.EventSystems.ExecuteEvents.Execute[T] (UnityEngine.GameObject target, UnityEngine.EventSystems.BaseEventData eventData, UnityEngine.EventSystems.ExecuteEvents+EventFunction`1[T1] functor) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/ExecuteEvents.cs:261)
UnityEngine.EventSystems.EventSystem:Update() (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/EventSystem.cs:377)

 

Hey guys. I recently upgraded to 2020 from 2019 and I'm getting an error on the LipSync Pro component: "Phoneme does not exist in the chosen Phoneme set. You can change the Phoneme Sets from the Project Settings". Where abouts in the project settings is this setting?

 

Could you both make sure you're on the latest version from the Asset Store?

@faizan143143 Can you give me any info about what you're doing to get that error? (Just basing off the post you've quoted there) If you're creating a LipSyncData object from a script make sure you set the "version" field to match the version of LipSync Pro you're using (probably 1.531) - LipSync uses this number to determine if the data needs updating from an old format.

 

Hi all, thought id share a short video animation using lipsync pro with a standard Uma.

It's my first one, like ever.

comments and critic welcome

 

Hi, FYI --I'm running into a null ref error when playing back emotions. On Unity 2019.3, MacOS 10.12, LSP 1.532.

The null ref occurs at line 611 of LipSync.cs where it loops thru boneCurves. Adding a null check fixes the problem.
Before:

Code (CSharp):

1. // Fix Quaternion rotations (Credit: Chris Lewis)
2.                 foreach (TransformAnimationCurve curve in boneCurves)
3.                 {
4.                     curve.FixQuaternionContinuity();
5.                 }

After:

Code (CSharp):

1.  
2. // Fix Quaternion rotations (Credit: Chris Lewis)
3. //Hack fix for null ref
4. if(boneCurves != null)
5. {
6.                 foreach (TransformAnimationCurve curve in boneCurves)
7.                 {
8.                     curve.FixQuaternionContinuity();
9.                 }
10. }
11.  

 

Hello everyone, I am making an automatic playback of a LipSync clip with audio recorded with a microphone. Everything is fine with the recording, no problems. But then I need to arrange phonemes using the AutoSync Window, and I can't find the functions I need and is it possible to do this at all. I would be very grateful if someone can provide some algorithm of actions or methods so that AutoSync can be done in code.

Thank you very much!

 

Hello everyone - I've got a bit of news to announce. I recently got offered a new job that will take up much more of my time, and so I've been considering my options regarding what will happen with LipSync Pro and Cinetools going forward.

Ideally, I'd like to be able to keep selling LipSync Pro, and publish the new Cinetools (v2) assets as planned - realistically though it looks like I just won't have the time to provide the required support in future, and I'll have a lot less free time to spend working on the Cinetools assets. Unfortunately, for this reason I've decided to stop selling the current version of LipSync Pro from the start of January 2021, when the New Year's sale ends. I will continue to provide support through email and discord when I can, until I've finished updating the documentation. I also plan to publish further patches as and when, to keep the final version stable and working.
My long-term plan here is to release LipSync Pro under an open-source license (details TBD) so others can continue improving it if they wish, but I want to leave some time between deprecating the asset on the store and releasing it for free in fairness to anyone who has bought it recently.

As for Cinetools (including LipSync Pro 2), I'm not cancelling it outright - I've put a lot of work into it so far - but it needs a fair bit more work to be usable. For now, I'm just going to put the project on ice and revisit it in future - there may be bits I can make available in some other way, we'll see.

Regardless of what happens, I'd like to thank everyone who's bought and used LipSync over the last 5 and a half years. It's been amazing seeing all the varied projects you guys have made!

Thanks for the heads up - I'll put it in a patch with a few other small fixes.

I'm afraid AutoSync isn't available at runtime, as stated in the Asset Store description. You can access its functions from editor code though if that's any use - you can find the required methods in the RogoDigital.Lipsync.AutoSync class.

 

Oh, i understand.
Thanks a lot!

 

Hey
Somehow lipsyncpro component throw this error and nothing work,as you can see theres nothing in the inspector under the lipsync pro component
Unity version 2019.4.10f1
Lipsync pro latest version

 

please add Mandarin (Chinese) recognition support, we are looking forward to it!

 

That's weird - can you tell me what happened before that occurred? I see you've got the Sprite Manager component attached, so I assume it worked at first until some point after choosing the blend system?

I'm afraid there's no plans to add Mandarin support at this time - as explained above, I'm actually stopping active development on LipSync Pro for now.

 

And on that note - today's the day the LipSync Pro package is being deprecated from the Asset Store (the New Year's sale lasted longer than I expected it to). If you've just come to this thread after seeing it marked as deprecated, don't worry, this isn't the end of support yet!

The full post is a little further up this page if you want more info, but the short version is I no longer have time to properly support it and continue development on v2 going forward due to a new job. I've taken it off sale as it doesn't seem fair to continue selling it without the expectation of long-term support, but I do plan to eventually release it under some open source license in the future. For now, I'll continue responding to questions when I can (most likely on weekends, so please be patient!). If you've only just bought it and aren't happy with the circumstances, I understand - my aim in not making it open source immediately was to give some value to brand-new customers, but if you haven't yet downloaded it and would prefer a refund, get in touch with your order number and I'll get that sorted with Unity.

 

Hey,

So when I imported the newest package from package manager window in a new unity 2019.4.17f1 project. I then installed without problems until I had to download the montreal forced aligner package. After downloading it I got the same error as the other guys:

Code (CSharp):

1. "Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/Modules/Montreal Forced Aligner/ASMontrealPhonemeDetectionModule.cs(44,24): error CS0115: 'ASMontrealPhonemeDetectionModule.ProcessWithTemplates(LipSyncData, AutoSync.ASProcessDelegate, PhonemeMarker, EmotionMarker)': no suitable method found to override"

When I checked the code, the error seemed accurate; the
ProcessWithTemplates virtual method is not declared in the AutosyncModule.cs.
However, if I remove the ProcessWithTemplates implementation in ASMontrealDetectionModule.cs (it wasn't being used)
it also gave me some errors, when I looked into it, it seems variables were using membernames that were already refactored in their corresponding classes. I changed them to the correct version: phonemeList.Count became phonemes.Length.

Makes me think if the montreal forced aligner package maybe contains an outdated/wrong version of ASMontrealDetectionModule.cs?

Either way, after changing those things in ASMontrealDetectionModule.cs everything works great!

Thank you for your hard work and good luck with your new job!

 

Hey, I'll take a look at this asap, but it looks more to me like Package Manager may be serving you an older version of LipSync Pro - can you confirm the version number you see in the top-right corner of the Clip Editor? I have seen other publishers talking about similar issues with the Package Manager & Asset Store packages, so it may be related.

 

Ohhh wow you're right! Package manager is making me use 1.52 despite it saying it's 1.532.0. That's a super lame bug, considering their choice to force the use of package manager in unity 2020 and beyond

I've downloaded it again in a new unity 2019 project through the asset store window and now I don't have any errors at all.

So to avoid this unity bug for now, we just have to use unity 2019 or below because those still have the asset store window?

 

Delete the old package from your computer's Asset Store cache folder:

Windows: [user]\AppData\Roaming\Unity\Asset Store-5.x
Mac: ~/Library/Unity/Asset Store-5.x

Then you can download the latest version using the Package Manager window.

 

/sigh .. the pain of buying several assets at a big sale .. and JUST getting around to seeing what I want to run with them .. picked up Slate and was looking at this lip sync to go with it ..
understand shifting prioroties .. especially these days ..
will follow the progress here .. hope your new job is successful !!

 

I just started a project and was looking for lipsync that I could try for to learn from.
I saw that there was a "Lite" version to use for free, but I'm a bit late. Good luck on your new ventures.

 

Dead Asset

 

OK what's going on? Rogo gave up on the asset again?...


Edit: OK, I've just read the post from 9 Jan. RIP.
To anyone who tried the Slate+LipsyncPro extension: does it work well with the final Lipsync version, or is it time to go for Salsa?

 

Ugh so I can't even buy Lip Sync Pro even if I wanted to. The next best thing is Salsa, is it any good? I think it's only realtime?

 

As a long time user I'm very sad to hear of the demise of LipSync Pro. I've since moved on to Reallusion's CC3 and the AcuLips functionality in iClone but I've still been recommending it as the best lipsyncing asset on the Unity store.

@thatnzguy yeah, Salsa is intended for realtime only (meaning no preprocessing or tweaking of phoneme positions) it's got better and better over the years, adding an emotion layer on top of basic lipsyncing, as well as rudimentary eye movements (not as good as RealisticEyeMovements). It's not in the same league as LipSync Pro but support is great if/when you need it. The real benefit is that it can handle languages other than just English and German. I used it the other day because I needed a quick and dirty Italian version of a work project. It worked ok-ish:

 

I've been using SALSA in conjunction with Realistic Eye Movements, they are not in conflict. Obviously you need to switch off SALSA's eye features beforehand. Good results.

 

Yep that's exactly what i'm doing too

 

There is no quality LipSync in Unity. That sucks big time as it is key feature for the so called "AAA" titles. Rogo tried but since there are not many high quality projects that require Lip Sync they ultimately gave up.

 

There's always FaceFX if you need AAA-quality lipsync. But it's AAA-priced ($899 USD).

 

Nvidia's Audio2face looks promising for automatic lip sync.

 

I'm sorry to hear that LipSyncPro has fallen off the Asset Store. It was one of the major lip-sync packages for Unity.

Audio2Face looks like it might be pretty kick-ass. Nvidia may have a winner on their hands. I'm interested to see how the animation data is exported, and how easy it might be to integrate into other packages.

In my free time, I've been learning how to utilize PocketSphinx using C++. I'm seriously considering resurrecting Cheshire. (a small Unity plug-in for lip-sync from back in the day) Fair warning, I'm planning on cooking up an internal tool first, so I can use it to produce my own videos before I start working on an updated Unity plug-in. So don't expect it to be showing up anytime soon.

 

I can't seem to find much information on FaceFX since about 2015, has anyone used it recently? And if so is it still worth the price tag if you want AAA quality animation?

 

This is with an older version of FaceFX. The first clip better represents how FaceFX looks. The second is a quick example in Unity using an old model from 2012 that I threw into FaceFX without any tweaking:

 

Hello.
I have a question, Can I somehow pre-install LipsyncData, and not in Play (LipSyncData lpd). I work with AR, and with large files it freeze.

Thanks!

 

Has anyone else run into an issue where the phoneme markers are not visible in the editor. The data is there because the character animates and the tooltip shows up when I hover...but I'm unable to effectively edit the marker positions.

 

Okay. I solved my own issue. I guess the "Editor Default Resources" needs to stay in the root Assets folder.

 

https://docs.unity3d.com/Manual/SpecialFolders.html please read this page

 

Hi I have posted a couple of issues on the GitHub page but wondering if I could get any help here.

I have started with an empty project and have followed the setup process but when I try to convert any audio I get errors. I am on a Mac so wonder if it is part of the issue.

I get the following error when I add an audio file to the AutoSync window. I also have a transcription text file with the same name next to the audio file.

Thanks for any help on this.

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. RogoDigital.Lipsync.AutoSync.ASMontrealPhonemeDetectionModule+<>c__DisplayClass10_0.<ProcessWithTemplates>b__0 (RogoDigital.Lipsync.LipSyncData outputClip, RogoDigital.Lipsync.AutoSync.AutoSync+ASProcessDelegateData data) (at Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/Modules/Montreal Forced Aligner/ASMontrealPhonemeDetectionModule.cs:55)
3. RogoDigital.Lipsync.AutoSync.ASMontrealPhonemeDetectionModule.Process (RogoDigital.Lipsync.LipSyncData inputClip, RogoDigital.Lipsync.AutoSync.AutoSync+ASProcessDelegate callback) (at Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/Modules/Montreal Forced Aligner/ASMontrealPhonemeDetectionModule.cs:86)
4. RogoDigital.Lipsync.AutoSync.ASMontrealPhonemeDetectionModule.ProcessWithTemplates (RogoDigital.Lipsync.LipSyncData inputClip, RogoDigital.Lipsync.AutoSync.AutoSync+ASProcessDelegate callback, RogoDigital.Lipsync.PhonemeMarker phonemeTemplate, RogoDigital.Lipsync.EmotionMarker emotionTemplate) (at Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/Modules/Montreal Forced Aligner/ASMontrealPhonemeDetectionModule.cs:52)
5. RogoDigital.Lipsync.AutoSync.AutoSync.RunModuleSafely (RogoDigital.Lipsync.AutoSync.AutoSyncModule module, RogoDigital.Lipsync.LipSyncData data, RogoDigital.Lipsync.AutoSync.AutoSync+ASProcessDelegate callback, RogoDigital.Lipsync.PhonemeMarker phonemeTemplate, RogoDigital.Lipsync.EmotionMarker emotionTemplate, System.Boolean silent) (at Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/AutoSync.cs:74)
6. RogoDigital.Lipsync.AutoSync.AutoSync.RunSequence (RogoDigital.Lipsync.AutoSync.AutoSyncModule[] moduleSequence, RogoDigital.Lipsync.AutoSync.AutoSync+ASProcessDelegate onFinishedCallback, RogoDigital.Lipsync.LipSyncData inputData, RogoDigital.Lipsync.PhonemeMarker phonemeTemplate, RogoDigital.Lipsync.EmotionMarker emotionTemplate, System.Boolean silent) (at Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/AutoSync.cs:40)
7. AutoSyncWindow.OnGUI () (at Assets/Rogo Digital/LipSync Pro/Editor/Modals/AutoSyncWindow.cs:609)
8. UnityEditor.HostView.OldOnGUI () (at /Users/bokken/buildslave/unity/build/Editor/Mono/HostView.cs:240)
9. UnityEngine.UIElements.IMGUIContainer.DoOnGUI (UnityEngine.Event evt, UnityEngine.Matrix4x4 parentTransform, UnityEngine.Rect clippingRect, System.Boolean isComputingLayout, UnityEngine.Rect layoutSize, System.Action onGUIHandler, System.Boolean canAffectFocus) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/IMGUIContainer.cs:366)
10. UnityEngine.UIElements.IMGUIContainer.HandleIMGUIEvent (UnityEngine.Event e, UnityEngine.Matrix4x4 worldTransform, UnityEngine.Rect clippingRect, System.Action onGUIHandler, System.Boolean canAffectFocus) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/IMGUIContainer.cs:676)
11. UnityEngine.UIElements.IMGUIContainer.HandleIMGUIEvent (UnityEngine.Event e, System.Action onGUIHandler, System.Boolean canAffectFocus) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/IMGUIContainer.cs:648)
12. UnityEngine.UIElements.IMGUIContainer.HandleIMGUIEvent (UnityEngine.Event e, System.Boolean canAffectFocus) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/IMGUIContainer.cs:641)
13. UnityEngine.UIElements.IMGUIContainer.SendEventToIMGUIRaw (UnityEngine.UIElements.EventBase evt, System.Boolean canAffectFocus, System.Boolean verifyBounds) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/IMGUIContainer.cs:607)
14. UnityEngine.UIElements.IMGUIContainer.SendEventToIMGUI (UnityEngine.UIElements.EventBase evt, System.Boolean canAffectFocus, System.Boolean verifyBounds) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/IMGUIContainer.cs:596)
15. UnityEngine.UIElements.IMGUIContainer.HandleEvent (UnityEngine.UIElements.EventBase evt) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/IMGUIContainer.cs:529)
16. UnityEngine.UIElements.CallbackEventHandler.HandleEventAtTargetPhase (UnityEngine.UIElements.EventBase evt) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/Events/EventHandler.cs:125)
17. UnityEngine.UIElements.MouseCaptureDispatchingStrategy.DispatchEvent (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/Events/MouseCaptureDispatchingStrategy.cs:83)
18. UnityEngine.UIElements.EventDispatcher.ApplyDispatchingStrategies (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel, System.Boolean imguiEventIsInitiallyUsed) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/EventDispatcher.cs:373)
19. UnityEngine.UIElements.EventDispatcher.ProcessEvent (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/EventDispatcher.cs:336)
20. UnityEngine.UIElements.EventDispatcher.ProcessEventQueue () (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/EventDispatcher.cs:299)
21. UnityEngine.UIElements.EventDispatcher.OpenGate () (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/EventDispatcher.cs:264)
22. UnityEngine.UIElements.EventDispatcherGate.Dispose () (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/EventDispatcher.cs:75)
23. UnityEngine.UIElements.EventDispatcher.ProcessEvent (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/EventDispatcher.cs:364)
24. UnityEngine.UIElements.EventDispatcher.Dispatch (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel, UnityEngine.UIElements.DispatchMode dispatchMode) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/EventDispatcher.cs:216)
25. UnityEngine.UIElements.BaseVisualElementPanel.SendEvent (UnityEngine.UIElements.EventBase e, UnityEngine.UIElements.DispatchMode dispatchMode) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/Panel.cs:399)
26. UnityEngine.UIElements.UIElementsUtility.DoDispatch (UnityEngine.UIElements.BaseVisualElementPanel panel) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/UIElementsUtility.cs:466)
27. UnityEngine.UIElements.UIElementsUtility.UnityEngine.UIElements.IUIElementsUtility.ProcessEvent (System.Int32 instanceID, System.IntPtr nativeEventPtr, System.Boolean& eventHandled) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/UIElementsUtility.cs:209)
28. UnityEngine.UIElements.UIEventRegistration.ProcessEvent (System.Int32 instanceID, System.IntPtr nativeEventPtr) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/UIElementsUtility.cs:74)
29. UnityEngine.UIElements.UIEventRegistration+<>c.<.cctor>b__1_2 (System.Int32 i, System.IntPtr ptr) (at /Users/bokken/buildslave/unity/build/External/MirroredPackageSources/com.unity.ui/Core/UIElementsUtility.cs:28)
30. UnityEngine.GUIUtility.ProcessEvent (System.Int32 instanceID, System.IntPtr nativeEventPtr, System.Boolean& result) (at /Users/bokken/buildslave/unity/build/Modules/IMGUI/GUIUtility.cs:189)
31.  

 

@Rtyper : Congrats on the job, sorry to hear about this being deprecated. This is still the best lip synching tool I've found!

Everyone:

The Montreal Forced Aligner (MFA) is the bit which has the most difficulty in my project. This is what worked for me to get more reliable execution:

First, I used SoX to make a low-quality version of my audio files in 16000 wav. SoX comes with a batch file where you can drag/drop .wav files on it, and they will convert quickly/easily. Just be sure to edit the batch file and set the bitrate.

Then, I renamed the files so that the filenames where short. My filenames were long to distinguish them from each other, but this caused errors somewhere in the system.

I created a mapping from my original filename to the lower quality/short name version, so I could keep things straight. I do recognition on the lower quality file, and then delete once I have my lip synch data, which I then associate with the long filename / high quality version of the file.

 

Yup. you can script a whole pile of that too on the fly with sox fwiw. it's got a pretty comprehensive command line support and obviously the renaming and MKLNK -J for the folder (or actual symbolic links for the files I dunno your platform/solution) can be in the one liner. Done that before, works fine. Fwiw. you can even add effects in line with sox and get Pixie voices and stuff for free.

 

@Rtyper Hi, I bought an asset (Npc Chat) which seem to rely on Lipsync Pro. However Lipsync Pro is apparently not being sold anymore. Is there a replacement product? What is the recommended suggested workaround?

 

If you're looking for a replacement for NPC Chat that doesn't require LipSync Pro, there are other options, including the Dialogue System for Unity and Naninovel.

If you prefer to use NPC Chat and you're looking for a lipsync replacement, look into SALSA Lipsync Suite. It does lipsync approximation based on whatever audio is playing on an audio source, so you probably won't have to do any special integration with NPC Chat. It also has some nice one-click setup wizards for popular character systems such as UMA.

 

@TonyLi new releases like NPC Chat do a fraction of what DS + QM do better, sometimes for the same cost. Unity themselves should advertise DS + QM as the gold standard so that people don't get sidetracked by lesser assets.

 

Hi, I am guessing DS stands for Dialog System, what does QM stand for? Sorry I am not fully emerged into the vernacular.

 

Quest Machine. I own both Dialogue System & Quest Machine highly recommended them.

 

Quest Machine. (Thanks for supporting both of them!)

 

I installed the assets from the GitHub download and keep running into this error:

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. LipSyncClipSetup.OnModalGUI () (at Assets/Rogo Digital/LipSync Pro/Editor/LipSyncClipSetup.cs:1345)
3. RogoDigital.ModalParent.OnGUI () (at Assets/Rogo Digital/Shared/Editor/ModalParent.cs:24)
4. UnityEditor.HostView.InvokeOnGUI (UnityEngine.Rect onGUIPosition, UnityEngine.Rect viewRect) (at <55729f52d042492e9efc384182ae2feb>:0)
5. UnityEditor.DockArea.DrawView (UnityEngine.Rect viewRect, UnityEngine.Rect dockAreaRect) (at <55729f52d042492e9efc384182ae2feb>:0)
6. UnityEditor.DockArea.OldOnGUI () (at <55729f52d042492e9efc384182ae2feb>:0)
7. UnityEngine.UIElements.IMGUIContainer.DoOnGUI (UnityEngine.Event evt, UnityEngine.Matrix4x4 parentTransform, UnityEngine.Rect clippingRect, System.Boolean isComputingLayout, UnityEngine.Rect layoutSize, System.Action onGUIHandler, System.Boolean canAffectFocus) (at <e88faa3aac574192a3ac9d4dae8147a1>:0)
8. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr, Boolean&)

A fresh install works just fine, but when reloading the project, this error loops as long as the Clip Editor panel is open. Causes the File menu to stop working.

Weirdly, this issue goes away when you close and reopen the panel in the same session. I guess it's not too much of an inconvenience, but I thought I'd still mention it here because it was driving me up the wall.