[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

Cool man, Thanks

 

Hey @Rtyper,

I was wondering if you have any plans on supporting the upcoming Timeline feature. Right now I've managed to set up a custom track that can call specific functions from LipSync, so far it seems to work fairly well except for scrubbing in editor. Is there a proper approach to that? I'm calling PreviewAtTime, but I'm sure I'm missing something else for it to work, if it is the right function to call at all.

Thanks.

 

Yep, full Timeline support will be available when it's in a beta Unity release. PreviewAtTime should be enough for scrubbing to work, it's the same function that the clip editor uses for that. Make sure you've loaded the data and ProcessData before that point, so that the animation curves exist. The time parameter that PreviewAtTime takes is also normalised, so you may need to divide the true playback time of the clip by the clip length first.

 

Hi,

I am currently using LipSync Pro on Unity v5.5.0f3, and whilst i can see a lot of potential in this asset there are a lot of unusual errors that have been constantly looming. I can run auto sync, i can generate Lip sync data files for all my audio files (they are all wav) and i can play them all using the play on awake setting and dragging them in, so i know they work. Also certain characters in my scenes work perfectly with blend shapes, audio and loaded in code.

The most common and frustrating error i am having currently, is where by using the in-build Unity Resource.load function i load a given file such as (example):

LipSyncData CustomerLip;
CustomerLip = Resources.Load (CDB4.ConversationDatabase.CustomerTalk) as LipSyncData;

if (CustomerLip != null){
Debug.Log("Customer audio loaded");
}
else {
Debug.Log("Customer audio FAILED" );
}

Some files load and others don't. The ones that fail result in the following error:

NullReferenceException: Object reference not set to an instance of an object
SceneFourControl+<CustomerSpeaks>c__Iterator1.MoveNext () (at Assets/Scripts/Scene 4/SceneFourControl.cs:207)
UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress) (at C:/buildslave/unity/build/Runtime/Export/Coroutines.cs:17)
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)

Referring to when i say try and get CustomerLip.length. Now i know it is because the files aren't been loaded because every file that fails, results in the log message "Customer audio FAILED" as mentioned above.

The reason i am confused is because all my audio files are wav, they are all in the same folder, and their lipsync datafiles were all created EXACTLY the same way! Yet some work and some don't, yet they are all on the same character? Why are some working and some not?

I need to solve this error as the module i am currently creating has over 60 audio files, and for a given character with audio files 0-10 for example, everyone will work expect files 3 and 6, with no apparent reason why. P.s. I'm sure it's not the original auido.wav files since we have even tried re-editing both the original and lipsync data, which apparently still remain "Unloadable" files.

 

I have a set of phonemes switched on and off with bone scale. How would I drive these using your system?

 

Any news on this working with MacOS Sierra?

 

Did you have any luck solving this? I'm not sure what I can really suggest as those errors are coming from your code.
LipSyncData files are standard serialised ScriptableObject (.Asset) files, so there shouldn't be any problem loading them through Resources.Load so long as the paths are correct.

At the moment, there isn't a built in way to do this. It wouldn't take many changes to track scale in bone transforms though. Send an email with your asset store order number (letters 'OR' then 9 digits, you'll have been emailed it when you bought LSP) to contact@rogodigital.com and I'll sort out a patch to enable it for you.

Sorry this has taken so long! The short answer is no. The problem is with the PocketSphinx DLLs that are used, and they were modified by the same C++ developer who created the whole Sphinx integration that AutoSync uses. He's no longer available to work on the project though, so I'm looking into having another developer take a look at recompiling them to work on Sierra.

We're also working on a big 2.0 update for the not-too-distant future, which will have a brand-new version of AutoSync in it that doesn't use any plugins and, hopefully, will be fully functional at runtime as well - so even if the current version can't be fixed, the next one will work just fine.

 

I am working on a Hololens Project and getting the following build errors after adding Lipsync Pro 1.32 which I want to use on a character in the project.

Code (CSharp):

1. Assets\Rogo Digital\LipSync Pro\Components\LipSync.cs(1753,49): error CS1061: 'XmlDocument' does not contain a definition for 'SelectSingleNode' and no extension method 'SelectSingleNode' accepting a first argument of type 'XmlDocument' could be found (are you missing a using directive or an assembly reference?)
2.  
3. Assets\Rogo Digital\LipSync Pro\Components\LipSync.cs(1772,49): error CS1061: 'XmlDocument' does not contain a definition for 'SelectSingleNode' and no extension method 'SelectSingleNode' accepting a first argument of type 'XmlDocument' could be found (are you missing a using directive or an assembly reference?)
4.  
5. Assets\Rogo Digital\LipSync Pro\Components\LipSync.cs(1797,49): error CS1061: 'XmlDocument' does not contain a definition for 'SelectSingleNode' and no extension method 'SelectSingleNode' accepting a first argument of type 'XmlDocument' could be found (are you missing a using directive or an assembly reference?)
6.  
7. Assets\Rogo Digital\LipSync Pro\Components\LipSync.cs(1936,32): error CS1061: 'XmlDocument' does not contain a definition for 'SelectSingleNode' and no extension method 'SelectSingleNode' accepting a first argument of type 'XmlDocument' could be found (are you missing a using directive or an assembly reference?)

Attached is a screenshot of my build settings.

Any ideas?

 

how can i make this asset work with morph3d 1.6.2
i download the extension but i get errors .

 

It seems like the UWP platform is missing some parts of the XML reading/writing classes. If you're not using the XML functionality you can simply comment out those lines (and a couple around them that will bring up errors afterwards) without any problem.

I'll also have a look at changing that code in the next update to fix the issue.

That'll be coming back soon! 1.6.2 is missing a couple of features that the LipSync integration relies on, but I've been told they'll be back in 1.6.3. When that's out, the extension will be updated to work with it.

 

Ok, I'll give that a shot. Thanks

 

@Drowning-Monkeys & @hawam - There's a new version of the Morph3D extension available! Both the old and new versions are still there, so make sure you get the right one and you've updated to MCS 1.63 before downloading it.

Cheers!

 

thanks my man

 

LipSync Pro is now driving the mouth sprites of my character. When I add expressions on a second layer the first layer of mouth sprites stops functioning. Is there an example case showing the proper setup for driving mouth and eye sprites at the same time?I might be misunderstanding the layer attribute.

 

I can't get autosync to work.

We bought LipSync Pro, but it keeps saying "No language models found. You can download language models ... " when I'm trying to use AutoSync. FYI I have SoX installed and it is verified. The files are all there in Assets/Rogo Digital/LipSync Pro/AutoSync/Language Models/EN-US/en-us en I tried reimporting all the things into a clean project several times. Unity 5.5.

Any help would be very much appreciated!

 

I am currently using LipSync Pro 1.321 on Unity v5.5.1f1, I want to control the clip when it is needed to be played by script, but the "RogoDigital.Lipsync.Lipsync.Play" function cannot be found, why?




Thanks!

 

Sorry for missing your posts guys, I haven't been getting notifications from the forum!

No prob, I'll put together an example scene for you today.

I'm really not sure about this. Can you try deleting that language model (the EN-US folder) and re-downloading it from the extensions window? If the en-us.asset file is there, LipSync should detect it.

The Play function isn't static, so you need to call it on a specific instance of the LipSync component.
Either put a public LipSync variable at the top of that script and drag the correct character into it in the editor, or use GetComponent<LipSync>() to find it.

 

Hey guys, So I imported an audio clip into lip syncs editor but no wave form is showing up. Any ideas? the preview also doesn't seem to be working

 

You're on a very outdated version there! Update to 1.321 from the asset store - there have been quite a few bug fixes since 1.1, so there's a good chance your problem's already been fixed

 

thank you !

 

Hi, I am currently using LipSync Pro 1.321 on Unity v5.5.1f1, and I am working with MCS character 1.6.3 on Hololens project, the phonemes animation(mouth movement) worked well on my desktop, but when I export it to Hololens hardware, the phonemes animation did not work anymore(i.e. there were no mouth movement anymore). Why?

 

This sounds like it could be an audio timing problem (which crops up on WebGL most often). Can you try changing the timing mode on your character to 'Custom Timer' instead of 'Audio Playback' and let me know if that fixes the problem?

 

LipSync Pro is on sale now for 30% off!
Get it for just $24.50 until April the 30th.
​

 

Hi Rtyper,

I am having trouble pausing my characters speaking (or even using Stop(true) for that matter). Any ideas? Neither Stop or Pause are working at all.

Im using Unity5.6.0f3
and the latest version of LipSync pro

 

How to add a new voice audio in LipSync Lite version, if i use different voice audio i am getting NullReferenceException: Object reference not set to an instance of an object. Then, if i use the sample audio (ie., Gettysburg) its work fine.

 

Hi Rtyper,

I am using unity 5.6.0f3 and lipsync pro 1.32.

Is there any idea about how to add two different clips to one character? Based on the current version, I can only add one default clip to the character.

Thanks

 

Hi

Can you please explain how you import bone setup from animation works.

1 choose animation
2 mask doesn't seem mandatory? What is it for?
3 choose transform - I have no idea what you are looking for...

If we use the example of let's say a blink...I choose left eyelid ass the transform?

Does this mean I have to run the import from animation for every bone associated with the blink?

I am just not sure what output this feature will give me.

Thanks
Cas

 

Could you post the whole error message here? I'm assuming you're talking about choosing an audio clip in the clip editor? If the example clips work fine, any others should too.

Just like an AudioSource, there's no way of queuing multiple clips from the editor. You need to either call the .Play command from a script and pass it the new LipSyncData object you want to play, or use the integration with some visual scripting tool (like Playmaker or NodeCanvas) if you use one.

Sure, the list of transforms are based on the bones in the AnimationClip you picked. It will try to match them up to the character model you're creating a pose for, but if it can't find matches then you'll have to pick transforms yourself.
So if your animation has a frame with both eyes closed that you want to use as a blink pose, then it will (probably) have 2 transforms in it (left and right eyelid), and you'd drag the left and right eyelid bones from your character in the scene into the appropriate fields in the setup wizard.

The mask can be used to quickly filter out certain bones from being used. So instead of manually checking/unchecking the checkboxes next to the bones in that list, you can drag in a mask to do the same job.

There's a video on our youtube channel here that shows the process of using that wizard.

 

Thanks for the info...is it broken in 5.6? Going back to 5.5 I could get it to work...

Tnx

 

A question regarding to batching auto lip syncing. I want to change the audio clips in several LipSync data files which all require to be lip synced again. I've developed a solution to swap the audio on all the lipsync data objects but essentially I want to run auto lip sync on them without doing them one by one in the editor window. Whilst I could import the new audio and create new lipsync data objects, that would then mean I would have to manually replace the references through the project.

 

Hello I recently bought Lipsync Pro, and I'm using Lipsync pro by controlling bone transformations. but, I can't know how to save my bone settings.

Can you tell me where to find E-mail address for technical support?

by the way, I love your Lipsync Pro very much for making my avatar speak and make facial emotions!

 

Not that I'm aware of... Testing it out on the default bone-based example model works fine in 5.6. Would you mind giving me more details on what's going wrong?

There's no super clean way to do this. The most flexible way would involve writing a custom editor script that runs AutoSync with it's own callback that replicates what the default one in AutoSyncWindow.cs does, but replaces the data in existing assets instead.

It might be simpler though to take advantage of the fact that the batch processor will overwrite existing files. If you import your new audio with the same names as the old audio files, in the same directories, then batch run AutoSync as before, the new .asset files will have the same GUIDs as the old ones and so won't break any references.

Thanks! You can save presets from the LipSync component editor so they can be applied to other characters. Just click the preset icon next to "Gestures", click "Save New Preset" and choose a name and directory for it.

The support email is contact@rogodigital.com, but you can also check our documentation first at lipsync.rogodigital.com/documentation/ and see if your question's answered there!

 

Hi there!

Loving the product, however am having a bit of trouble with the 'SetEmotion' part of the code.

Sometimes this feature works and sometimes it doesn't, often it doesn't work before the character has had any audio (lipsync data) play.

More documentation on this feature (and other features) would be greatly appreciated so I can problem solve and figure out how to make my character smile.

Thanks,
Simone

 

Glad you like it! More documentation is on the way, there's a lot of changes coming soon so some big updates to the docs will be coming with them.
I'm afraid I haven't been able to recreate your problem though - it blends into the new pose every time I call the function, whether or not a clip was played first. Double check that the name you're passing it matches an emotion on the character exactly (they're case sensitive) and that there's no clip playing at the time.

 

I am Chinese user. The built-in language model is en-us. Can you tell me how to build a mandarin language model?

 

Hi, No, still works on desktop but not working while export it to Hololens.

 

Update: When I applied Lipsync Pro onto Mixamo characters, it worked well both on desktop and Hololens. But to the MCS characters, the Lipsync only worked on desktop but not on Hololens. Could you figure out this issue so that we can use MCS on Hololens ASAP. Thanks!

 

I've replied to your email

That sounds to me like it's a problem with MCS rather than LipSync Pro then. I don't know much about the Hololens platform, but can you run a debug build on it to see if there's anything being output to the console? Otherwise, I'd say the best bet would be posting in their thread, they'll probably already know about compatibility between their product and Hololens.

 

I am Chinese user too. And I have bought it! Do you have mandarin language model?

 

Hi, I have issue with the autosync. It will always get stuck in the middle of the process for recognising phonemes. This makes me no choice but to force quit the unity. Is it because I'm using macOS Sierra? What should I do?

 

Hi there,

Trying to download the English language pack from the first tutorial but getting the following error 20% of the way through:


transfer closed with 24617585 bytes remaining to read
UnityEngine.Debug:LogError(Object)
RDExtensionWindow:<OnGUI>m__1() (at Assets/Rogo Digital/Shared/Editor/RDExtensionWindow.cs:421)
RogoDigital.ContinuationManager:Update() (at Assets/Rogo Digital/Shared/Editor/ContinuationManager.cs:30)
UnityEditor.EditorApplication:Internal_CallUpdateFunctions()


Has anyone else had the same error? I know the download mechanism is supposed to be convenient - but when something goes wrong it would be soooooo much easier if we could download directly from the website.

Thanks!

 

Hi,
This is the error:
NullReferenceException: Object reference not set to an instance of an object
RogoDigital.Lipsync.LipSync.ProcessData ()
RogoDigital.Lipsync.LipSync.Play (RogoDigital.Lipsync.LipSyncData dataFile, Single delay)
RogoDigital.Lipsync.LipSync.Play (RogoDigital.Lipsync.LipSyncData dataFile)
UnityEngine.Events.InvokableCall`1[RogoDigital.Lipsync.LipSyncData].Invoke (System.Object[] args) (at C:/buildslave/unity/build/Runtime/Export/UnityEvent.cs:189)
UnityEngine.Events.CachedInvokableCall`1[RogoDigital.Lipsync.LipSyncData].Invoke (System.Object[] args) (at C:/buildslave/unity/build/Runtime/Export/UnityEvent.cs:309)
UnityEngine.Events.InvokableCallList.Invoke (System.Object[] parameters) (at C:/buildslave/unity/build/Runtime/Export/UnityEvent.cs:637)
UnityEngine.Events.UnityEventBase.Invoke (System.Object[] parameters) (at C:/buildslave/unity/build/Runtime/Export/UnityEvent.cs:773)
UnityEngine.Events.UnityEvent.Invoke () (at C:/buildslave/unity/build/Runtime/Export/UnityEvent_0.cs:52)
UnityEngine.UI.Button.Press () (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/Button.cs:35)
UnityEngine.UI.Button.OnPointerClick (UnityEngine.EventSystems.PointerEventData eventData) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/Button.cs:44)
UnityEngine.EventSystems.ExecuteEvents.Execute (IPointerClickHandler handler, UnityEngine.EventSystems.BaseEventData eventData) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/ExecuteEvents.cs:50)
UnityEngine.EventSystems.ExecuteEvents.Execute[IPointerClickHandler] (UnityEngine.GameObject target, UnityEngine.EventSystems.BaseEventData eventData, UnityEngine.EventSystems.EventFunction`1 functor) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/ExecuteEvents.cs:261)
UnityEngine.EventSystems.EventSystem:Update()

Thanks.

 

Hi,
I found it, i didn't add phoneme in the audio clip.
Thanks.

 

Not at the moment, sorry! It's possible to create one for PocketSphinx (the tech that AutoSync currently uses), but as I don't speak any Mandarin I wouldn't be able to create it myself. The new version of AutoSync will be available in a few week's time anyway, which uses a new language model format, so I may look into having someone else create one after that's released.

You're right, AutoSync isn't compatible with Mac OS Sierra (as stated in the Asset Store description). There's an update on its way in a few weeks that will fix this though. Until then though, you should be able to run it on any older version of OS X, or Windows if you have access to either of them (on another machine, or using bootcamp etc). The saved files will work fine on Sierra.

Glad you got it sorted. That still sounds like it shouldn't be happening though! I'll have a look at it and get the error patched out soon.

 

Hey @Rtyper,
I think we have decided to go with LipSync Pro as our lip syncing solution for our upcoming VR game, just a few questions before hand.

1. Do you notice a difference in lip syncing quality between character models from Fuse and ones from Morph3D (we are currently in the very early stages of our game and can go with either solution)? Is there any advantages or disadvantages to either when it comes to LipSync pro?

2. Does LipSync pro have any support for the new timeline editor in Unity 2017 (I haven't experimented with it so I'm not familiar with how it works yet)

3. Does LipSync Pro work with the new ambisonic sounds in 2017?

4. Is the PS4 build platform supported?

5. Are there any known issues when using LipSync and Oculus and/or Vive?

Also I saw a few posts where you mentioned a new 2.0 build, what does that include and when will it be released?1

Cheers,
Colton

 

Hi Colton,

1. Not particularly - the Fuse system is slightly easier to work with as it uses standard blend shapes, but the quality of the lip sync animation between those two depends more on how your phoneme poses are set-up. Using the default presets Fuse might just come out on top, but I'm sure MCS can look just as good with a little more work.

2. Not in the current version, but this is coming soon. If you want to send me an email (contact@rogodigital.com) with your order number if/when you've bought it, I can send you a beta version of the timeline support in the next couple of weeks.

3. I haven't tested this (hadn't actually come across the feature until just now), but I don't see any reason it wouldn't. As long as they use standard AudioSources, then there shouldn't be a problem.

4. I don't have access to PS4 builds, so I haven't tested it. Again though, I don't see a reason it shouldn't - the runtime part of LipSync Pro doesn't use any unusual features. The only potential issue is the XML reading/writing classes, which are missing on some platforms (Windows Store for one), but this is very simple to disable if it does cause a problem.

5. Nope! LipSync & Eye Controller have both been used in several VR projects already.

2.0 is a few months away I'd say. There will be a 1.4 update coming in the next week or so that contains (amongst other things) a new version of AutoSync that fixes some compatibility problems on macOS. After that, 2.0 will be a fairly cheap (~$10) upgrade with new features like runtime AutoSync and better workflows for creating conversations & adding gestures and emotions.

There'll be more info on this soon though - I'll be putting a roadmap with details on all the new features up on the site once 1.4 is out.

 

Thanks for the reply!
Not really interested in runtime autosync or anything like that, but the conversation tool sounds interesting, can you give any more information on that? Are you looking to create something similar to the "Dialogue System For Unity"? Which conversation tool would you recommend using right now?
Cheers,
Colton

 

I can confirm that at least one dev is using LipSync Pro on PS4 (with the Dialogue System for Unity), but I don't know if they disabled the XML reading/writing classes as Rtyper mentioned. I got they feeling they were using it straight out of the box, though.

 

Fair enough! It won't be a full replacement for Dialogue System (Don't worry @TonyLi ), which is still the system I'd recommend as this won't have branching conversations or dialogue choices etc. The basic use-case will be setting up a sort of playlist of LipSync clips to play one after another, on one or more characters. Then to make the creation process simpler, it has tools for quickly switching between the various clips in the editor and setting up simple "shot reverse shot" camera sequences.

And thanks for this - I hadn't come across any projects on PS4 so far so that's good to know!

 

Thanks for the reply from both of you, I will pick this up and start implementing tomorrow!