[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

I purchased Lip Sync Pro yesterday, I tried some Turkish voice and it worked nice. I've got some questions.

1. How do we create blendshapes? The models in our game wasn't created with something like Morph3D. I am totally lost here, I couldn't find a tutorial about it.

2. Is it an important necessity to have language models for non-English voices? The system already converts my Turkish voice to English phonemes, then Lincoln uses these English phonemes for correct lip syncing. I wonder if I am missing something.

 

@Gord10 I have hardly any skills in 3d modelling, and I use a plugin for blender to create characters with blend shapes included that will be tweaked by someone that does know stuff later on.

You create blend shapes in your 3d modelling software of choice. It's called shape keys in blender, morph targets in 3dsmax and blend shapes in Maya. You should be able to find tutorials on that.

 

Hey, I'm on Mac Sierra and while Autosyncing I was getting stuck on "recognising phonemes" again. (I posted in this thread a few weeks ago). It worked fine in 1.41, but since upgrading to 1.42 it started getting stuck again. I reverted my project files, but you may want to check out the bug for future updates.

And thanks for being so fast to fix it in 1.41, very happy with this plugin and your support.

 

Hi,

i've downloaded version 1.42 and did some checks.
-Losing focus of now no longer leaves my character "emotionalized"
-Additive Mixing-Mode is now properly saved/loaded.
-No more compilation problems using Mono

Many thanks for fixing those

Further observations (probably those just did not make it into this release)
-"Array index is out of range." on loading a preset with higher indices than currently present in the advanced blendshape manager still occurs.
-After that, the inspector ui still breaks, is cut at the point the OoR occurs
... but for me it would be fine if this is considered for version 2. It's not that critical to me.


Also: A rather minor and very easy to fix problem (already existed in previous versions)

EyeController.cs: FixDummyHierarchy() should do more null-checks.
The first line already does check if the lookAtBones are null.
Given it's inner workings, it should however also handle the case the dummies are null, for example:
if (leftEyeDummy == null || rightEyeDummy == null) return;

Otherwise, it can be a problem in cases where eyeController is instanciated automatically.
I make use of that as i've got plenty of characters that are all constructed the same way regarding their eyes. So, i create EyeController on the fly on game load. Instanciating EyeController like this inside a start()-Function does not run eyeControllers start() right away. Instead, it's run some time in the future after my instanciating script has done its work.
As the dummies are created inside eyeController's start(), they are empty as i do my instanciation and configuration thing.
Because of that, as soon as i set the look at bones, a NPE occurs as EyeController wants to FixDummyHierarchy() that is not there.

Inserting the additional null-check
if (leftEyeDummy == null || rightEyeDummy == null) return;
fixed that for me.

 

I've noticed 'Look at target' -> 'Look at amount' resets to zero each time I press 'Play' if 'use auto target' is enabled.

 

@jerryyeh and @JX272 I've attached a slightly improved version of the AdvancedBlendshapeBlendSystem to this post. This will prevent the errors when a pose uses a blendshape with an index that's no-longer in the list, but there's very little I can do to prevent the issues with indices being re-assigned to different blendshapes without changing how they're stored (and likely introducing some extra processing cost at runtime). I'd just recommend you make sure you don't have a pose currently open or the clip editor in preview mode when you change the blendshape list and you should be fine. I'll also add some extra editor code to either warn about this or totally prevent it in the next update.

It's a bit odd seeing this error coming from the standard blendshape blend system, so I'm not sure exactly what's caused it. It looks like the internal blendable list has got out of sync somehow though, so adding this:

Code (CSharp):

1. if(blendable >= _blendables.Count)
2.     GetBlendables();

to BlendSystem.cs (in Rogo Digital/LipSync Pro/Classes) at line 191 (inside SetInternalValue() and immediately before "_blendables[blendable].currentWeight = value;") should fix the problem.

Corvwyn's right about your first question, as for your second - no, not necessarily! If you're getting good results out of it, then that's fine. Different language models for other languages would probably be slightly more accurate, but the US English one is the only complete one and it ended up looking a bit too expensive to create others for the current system.

Thanks for reporting this, I've actually had another report of the same issue as well since. Did reverting the upgrade actually fix the problem? My first thought was that I somehow replaced the fixed .dylib from 1.41 with the older one when I uploaded 1.42, but that doesn't look to be the case in my local copy of the project... I'll need to do some more testing on this one!

 

No problem!

This is a very quick fix, I'll add this in now.

I hadn't considered Eye Controller being added at runtime to be honest! As both it and LipSync are centred around poses that are setup in the editor it seemed like it would always have been created at edit time. I'll put null checks in there though, not a problem!

This is done to revert to random looking if there's no target - you're right though that shouldn't be happening when auto target is enabled as you might just be out of range. I'll get this fixed asap.

 

I've resolved the morphing, thank you all. I've got two more problems;

1. It seems like lip sync doesn't work perfectly with Unity's recorder. The character doesn't move his lips at the first phonemes, he does it later. Am I doing something wrong? It happens even in 480p resolution.

2. Is there an official way of using Lip Sync on Timeline at certain points? I have been using my own "Command Objects", which executes certain commands when its object gets active. It seems like I will call Lip Sync commands with that system, but I wondered if there is a better way.

 

Been doing a lot of lip syncs, thanks for making such a great plugin! One minor annoyance, if zoomed, the viewport will often scroll past the end of the file.

Behavior


Expected

 

Any plans to update this to support Unity 2017? I just purchased but there are a few dozen namespace errors when opening the example scenes. My version is 2017.3.0f3

 

I use LipSync Pro with Unity 2017.3.0f3 without problems. What error messages do you exactly get?

 

Sorry for the delay everyone! I've been really busy with other commitments recently, so work on LipSync had to take a bit of back seat.

Can't say I've got any experience with Unity's recorder (are you referring to this one?), so I don't know for sure. I'll download it today and see if I can figure out what's going wrong. I have written a timeline extension for LipSync, but it's definitely in the "experimental" stage to borrow Unity's naming! I've attached the .unitypackage to this post so you can download it, but just be aware that there may be bugs (especially relating to scrubbing on the Timeline editor or jumping around the Timeline at runtime). A stable version will be included with 2.0, as part of my re-writing efforts have been specifically to enable proper Timeline support!

I'm glad you like it! Can you give me any more info on how you got it to do that? I've not seen it do that myself!

As @Gord10 said, 2017 is already supported (as should 2018.1 be, though not officially). What are the errors?

 

hi your asset look very good, will you implement some blending texture for each shapemorph? that would be awesome to have these wrinkles during the lipsync, like these guy does

 

@Rtyper : Yes, I am talking about it. I discovered that this problem happens when "Play on Awake" is on. I made the system so that she talks when F2 is pressed, it works fine now.

I couldn't find your attachment in your post.

 

Hi there,

Any way to play lipsync pro audio from the unity timeline? Still in process?

Thanks!

 

Unfortunately, probably not. I agree it's a great effect but implementing it requires a custom shader, and if I created (for example) a variation of the Standard shader with normal map blending then it would limit people's options for other effects such as subsurface scattering or shader frameworks like Alloy. The only real solution is a much more in-depth skin shader with all these features, but that's a bit beyond the scope of this package!

Oops, sorry about that! I've added it to this one

There is, and it's attached to this post, but it's definitely not finished! See my reply to Gord10 above for more details.

 

Thanks a lot.

 

Hi,

We've been using Lipsync Pro and so far it has been great! There is one issue we are having with the Morph3D extension that we are having trouble solving. When playing an animation on the timeline and then triggering Lipsync to play (using your timeline plugin or a workaround we made for the timeline) it causes the character to 'pop' a few inches. This only happens when Lipsync is playing. Have you encountered this issue before? We've been in contact with the Morph3D guys but so far nothing has come from that. Here is a example of the popping:

Thanks!

 

Are you using any gestures in your LipSync animation? It looks like maybe there's a gesture animation being triggered there that doesn't align with the idle animation (or has some strange issue with additive vs override blending maybe?)

 

Hi Rtyper, I'm using LipSync Pro with MCS and realized that the shape blending is really slow, but that is the MCS scripts fault.
I found a workaround to directly blend the SkinnedMeshes.

I changed this function, but the mapping is not correct anymore, maybe you can help me with that:

Code (CSharp):

1.  
2.         SkinnedMeshRenderer[] skins;
3.         public override void SetBlendableValue (int blendable, float value) {
4.             if (!isReady) return;
5.             SetInternalValue(blendable, value);
6.             //m3DCharacter.SetBlendshapeValue(blendshapes[blendable], value);
7.             //m3DCharacter.SetBlendshapeValueAsync(blendshapes[blendable], value);
8.             if (skins == null) {
9.                 skins = m3DCharacter.GetComponentInChildren<CIbody>().GetComponentsInChildren<SkinnedMeshRenderer>(true);
10.                 skins = skins.OrderBy(x => x.name).ToArray();
11.             }
12.             if (skins[0] != null)
13.             {
14.                 //TODO test if LOD works...
15.                 skins[(int)((m3DCharacter.currentLODLevel - 1.0f) * 4)].SetBlendShapeWeight(blendable, value);
16.             }
17.         }
18.  

Edit:
I made a simple mapping dict. but I think that can be done easier... anyhow here is my final workaround:

Code (CSharp):

1.         SkinnedMeshRenderer[] skins;
2.         Dictionary<int, int> skinBlendMapping = new Dictionary<int, int>();
3.         public override void SetBlendableValue(int blendable, float value)
4.         {
5.             if (!isReady) return;
6.             SetInternalValue(blendable, value);
7.             //m3DCharacter.SetBlendshapeValue(blendshapes[blendable], value);
8.             //m3DCharacter.SetBlendshapeValueAsync(blendshapes[blendable], value);
9.             if (skins == null)
10.             {
11.                 skins = m3DCharacter.GetComponentInChildren<CIbody>().GetComponentsInChildren<SkinnedMeshRenderer>(true);
12.                 skins = skins.OrderBy(x => x.name).ToArray();
13.  
14.                 int length = blendshapes.Count;
15.                 for (int i = 0; i < length; i++)
16.                 {
17.                     Mesh m = skins[0].sharedMesh;
18.                     for (int n = 0; n < m.blendShapeCount; n++)
19.                     {
20.                         string sName = m.GetBlendShapeName(n);
21.                         if (blendshapes[i].Contains(sName))
22.                         {
23.                             skinBlendMapping.Add(i, n);
24.                             break;
25.                         }
26.                     }
27.                 }
28.             }
29.             if (skins[0] != null)
30.             {
31.                 //TODO test if LOD works...
32.                 if(skinBlendMapping.ContainsKey(blendable))
33.                     skins[(int)((m3DCharacter.currentLODLevel - 1.0f) * 4)].SetBlendShapeWeight(skinBlendMapping[blendable], value);
34.             }
35.         }

 

How to revert to previous version(to 1.41)?

I'm facing the same problem. I don't have previous version backup. I'm currently on macOS High Sierra.

 

@Rtyper great plug in, works well. Just one thing - I'm trying to use it in conjunction with VR Panorama asset to render out a 360 film. This project was built around a year ago so doesn't use timeline (stuck on 5.6p4 atm). During play mode everything works as expected, but when Vr Panorama is on, the rendered out images show that the lips aren't moving. I understand Vr panorama adjusts the time scale to render out frames, so I've left LipSync Pro on scale audio speed so it should match, but it doesn't. I do see there is another option - set FPS, how does that work exactly? Vr panorama is rendering out at 25 FPS but I'm not sure if changing LipSync Pro to FPS/25 will work.

Any tips appreciated.

Edit: after making some quick test renders turns out just setting LSP to fixed frame rate and the same frame rate as vr panorama and everything works perfectly!

 

Same here! I have seen this bug more often than I would like.

 

I'm on the same boat, I'm on the latest version, but autosync is not working on my macOS High Sierra. Any fix or a way to get a previous version?

 

I'm getting a little weird result with eyecontroller and Morph3d. Maybe there are some suggestions?




Upd: nvm, fixed it.

 

Nevermind I reinstalled and it works #forreasons !

 

Hi,

I've been using this for my Makehuman character with Blendshapes. I dragged the eyelashes, teeth and tongue into the additional mesh fields, but somehow none of them are moving with the face. How can I fix that??

 

How did you? I'm having the same problem.

 

Hey all
Testing out Lip Sync Lite, and interested in purchasing Pro. I have two questions before I buy though.
[OSX 10.13, Unity 2018.1 b13]
(I know my issues may be related to High Sierra and/or using Beta unity)

1.) In 2D Sprite-swapping projects, I'm a little confused on how to set the weight values in the phoneme setup section of the Lip Sync component. I'm having trouble getting consistent results.

The sprites trigger far too early, linger either too long, or revert to the rest position regardless of how I've set the rest and pre-rest hold times. I think maybe I just don't understand the weight system? I had trouble finding relevant info online, so I came here. Could someone point me in the right direction to learn how the weight values relate to the 2D sprite swapping?

2.) I've set up my phoneme markers and saved my files. When I launch Unity's maximized play mode, the markers become unaligned with the audio's waveform in the clip editor. I must reload the lipsync data file to get the markers back in place. Is this a known issue, or maybe something that only happens in Lip Sync Lite?

Much thanks for your help! It's a great tool and I look forward to getting past this and using the pro version

 

I rewrote it a little without dummies. I don't know can I share it here or not because it's a whole file

 

You can share the bits you've edited, no need to post the whole script.
thanks

 

Hi everyone, first off I'd like to apologise for not replying to this thread recently - I've got some news about the future direction of LipSync that I wanted to announce here, and I was planning to answer the last few questions at the same time. Typically though things took longer than I expected to figure out, so I'm sorry for leaving you in the dark for so long!

It's a long post, so I'll get the questions out of the way here, and then the next post will be all the new info about the version 2 update etc.

You're right about the cause of the slow-down. You can make it a bit faster by making sure you use as few morphs as possible per-pose, but your solution should work well enough too. It may be possible to just use the advanced blend shape blend system too, but you'd need to make sure you have all the morphs checked to be included on the mesh otherwise some may be stripped out and your poses won't work as expected.

I've so far been unable to track this issue down - I don't believe reverting to 1.41 will actually help the issue. I've double checked the files and there was no change to the AutoSync system between 1.41 and 1.42. Weirder still, AutoSync works normally for me when I run it on my MacBook Air (with the latest macOS High Sierra version). Can any of you confirm if there are any errors in the console when you try running AutoSync or does the process just hang indefinitely with no other output? Any info at all you can give me about your setup or what happens when using AutoSync would be helpful!

As before, if you have access to a Windows PC or Windows running under Bootcamp then you can run AutoSync there and copy the resulting LipSyncData files over in the meantime. I know it's not ideal, but I'm trying to figure out the cause of the problem as quickly as possible.

It looks like the eye bones/models are being offset from the dummies for some reason - usually the bones are re-parented and positioned exactly where they were before under their new parent. Are you doing anything to the eye positions from a script, maybe? Is there an animation clip playing? I know you worked around the problem, but I'd like to be able to fix this properly for a patch!

P.S. I'd prefer it if you didn't share the complete Eye Controller script on here, but if you want to PM it to anomas, that's fine.

It may be that they don't have the same blendshapes as the main mesh. Try using the 'advanced blend shape blend system' instead, as that allows for the same blendable in LipSync to control different blend shapes on difference meshes.

1) Weight values are a bit unclear for sprite swapping, you're right. It's really just a side-effect of the fact that the sprite swapping system has to fight against LipSync's normal behaviour a little bit by ignoring most of the blending LipSync is trying to do and just choosing a single sprite to display at any one time per layer. Generally, I'd say just leave all the weights at 1, so long as you only have one sprite per layer per pose. Additionally, it's best to leave the 'Rest' phoneme empty when using the sprite blend system, and instead rely on the default sprite fields of the blend system - otherwise you get the rest sprite having a higher priority most of the time and hiding all your actual animation. You may also want to try changing the curve generation modes on your character and see if you get better results from one than the other.

2) I haven't come across this issue before, I'll have to look into it. I usually don't have the clip editor open when not using it, so it may just be something I haven't noticed!

As a warning, several users have reported that AutoSync doesn't work for them on macOS High Sierra. I know the problem isn't universal, but I don't know for sure how widespread it is (users don't generally come to post on the forum when everything's working fine!). It still works fine on Windows, older OS X versions and (at least) some installations of High Sierra.

 

It was a default MCS Female character with some changed blend shapes, lEye and rEye are bones. The problem persists with and without animation (for me it was animation from Mixamo).
For me, fixing was removing code for dummies and rotating these bones directly. By the way, I think it might be helpful in some situations.

 

I bought Lip sync pro last week, and manually keyed an audio file, and tied it into my character, and was happy with the results. I've tried using autosync multiple times on multiple files, and always end up with the same result, all the phoneme's are smashed together at the end:



Any pointers on what i'm doing wrong? as far as I can tell its detecting the phoneme's correctly, it just isn't putting them in the right places. I'm using Sox 14.4

as a side note, if I convert the file to 16k sampling, and save it as a wav and bypass sox the autotagging looks much better. How do I troubleshoot what is going wrong with sox?

 

Really enjoying using Lip-sync pro, but I can't seem to get the emotions to work. I have several posed set up, but no matter how long I stretch the poses in the mixer it only plays for a frame or two and goes back to normal. I am using bones only for the emotions and phonemes. Is there something I am not doing correctly that would cause this?

 

Hey everyone just a quick question. I have everything setup and the lip sync works, but when I press play the characters jaw is wide open. The mouth still moves.. it's just open as well. Does not do this when I preview in the clip editor thanks. Also im using Morph 3d Characters and presets

 

This is a known problem with MCS characters. If you go to the MCS script on the character, down on 'advanced', there's a checkbox to "force jaw shut'. They say this happens when you use base animations that do not close the jaw by default.

 

Thank you very much!

 

Hi, I want to use Chinese or Japanese language model. How can I build them? Can u tell me the method detail @Rtyper

 

Do u get the other language models phonemes now especially Mandarin and Japanese?

 

well I need your help in a diff way If I may ask please reply <3

 

@Rtyper I need to use autosync in a code enter a conversation I started I will explain to you everything

 

Hey!

The plugin is working as it should. Thank you very much!

I love the preview feature in the LipSync Panel, but is there any chance to have a preview of the gestures? Right now "just" Phonemes and Emotions are being played. I could dive into the source code, if you could tell me where to look. Or is it a issue because we aren't in play mode?

Thank you!

 

Hey @Rtyper,

Found a couple bugs in PlayFromTime (LipSyncData dataFile, float delay, float time) method. I added in a ! when checking loadFailed and added a call to ProcessData() right after the load failed check.

Cheers,
Colton

P.S Not sure if these are fixed in a newer version

 

Yes, PLEASE, it is urgent! Can I get a copy of your Timeline support extension that you wrote for another customer ?!?!?

 

I need some help with a weird bug in Clip editor / emotions:
I made a new project, imported the package(LipSync Pro) and followed your instructions for installing everything necessary. Then I used autoSync on one of my speech clips and added some emotions. Everything works fine.
Now when I close Unity and reopen the project I can't select any emotions anymore. When I mouse over an emotion, all previously saved emotions get selected, but I cannot select any or resize or move them.
No errors in console and everything else seems to work fine. Attaching screenshot.

 

Same problem as 11Tomi12 above. Emotion editor is pretty much unusable for me.

Both Mac (High Sierra) and Windows 10.

Also, the AutoSync bug described above appears on my High Sierra as well.

 

runtime autosync? my mp3 files are generated i runtime !so how to sync those audio files

 

Hello everyone. I'm just starting with the LipSync Pro and I'm also just starting in Unity. I've been following the tutorial and I'm glad everything works now. However, I want my LipSync Data to be played when I press a button on my keyboard.

I think I should write something like this on my script file:

Code (CSharp):

1. using RogoDigital.Lipsync;
2. void Update () {
3.         if(Input.GetKeyDown(KeyCode.T))
4.         {
5.             LipSync.Play("Greeting-LipSynced.asset");
6.         }

But obviously this code by itself doesn't work. It seems like I need to have a "LipSyncData" file to be put inside the "LipSync.Play" but how to get this? Any help would be so appreciated.. Thanks!

- - - - - - - - - - - - - - - - - -
// EDITED //
- - - - - - - - - - - - - - - - - -

So I already found the solution in this forum:

Then I just drag my character game object and my lipsync clip on the attached script

 

On a clean project (no Morph3D yet), I download the latest Lipsync Pro version (1.42) and then download the Morph3D 1.63+ blend system extension and I get two errors:

Assets/Morph3D Blend System/Morph3dBlendSystem.cs(60,24): error CS0115: `RogoDigital.Lipsync.Extensions.Morph3dBlendSystem.OnBlendableRemovedFromPose(int)' is marked as an override but no suitable method found to override

Assets/Morph3D Blend System/Morph3dBlendSystem.cs(55,24): error CS0115: `RogoDigital.Lipsync.Extensions.Morph3dBlendSystem.OnBlendableAddedToPose(int)' is marked as an override but no suitable method found to override

Unity 2018.1.0f2. What's wrong?