Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

Discussion in 'Assets and Asset Store' started by Rtyper, Mar 11, 2015.

  1. ggendron

    ggendron

    Joined:
    Mar 19, 2019
    Posts:
    4
    Hi,
    I'm currently using the "bones only" blend system and it seems that the phoneme curve generation always makes linear curves.
    The "Tight" and "Loose" options doesn't seem to change anything.
    After a look in lipSync.cs it seems (but I'm really not sure) that the loose and tight parameters are only used when using blendshape, is this true? If yes, is it possible to add these curve options to the bones-only system?
     
  2. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Good point - I'd noticed the the glitching when doing the new tutorials recently. I'm looking into it now, for a fix in 1.52, but in the meantime you can reduce it by using the "Tight" emotion curve generation mode. Renaming emotions is a little difficult as they're just referenced by name, but I'll see what I can do!

    That's right - it's not strictly a linear curve, but the overshooting that Loose mode can cause is made much worse when dealing with rotating/moving bones so it's not used! Curve generation is a little rigid at the moment though, that's something I want to look at improving.
     
    Alvarezmd90 likes this.
  3. Alvarezmd90

    Alvarezmd90

    Joined:
    Jul 21, 2016
    Posts:
    79
    Thanks man, I'm looking forward to these updates. Can I support you anywhere else? I'm willing to pay for these fixes. (Dunno, since it's been so long, I can't expect you to fix these kind of things for free entirely.)
     
  4. StarsideStudios

    StarsideStudios

    Joined:
    Jul 15, 2013
    Posts:
    30
    Hello again :)
    Having an issue with FPS while the lip sync is playing. Drops from a smooth 60 to about 30fps until the clip finishes. Any suggestions? Using a Morph3D model with the blendshape set that was provided. Really want to use lipsync in gameplay and not just cutscenes, but having trouble at the moment. Other than that love the asset! Any help would be appreciated. Thank you

     
  5. liquidwes

    liquidwes

    Joined:
    Jun 5, 2019
    Posts:
    1
    Can you share your animation exporter integration? I really need something of the sort right now.
     
  6. NawarRajab

    NawarRajab

    Joined:
    Aug 23, 2017
    Posts:
    18
    Hey whenever I try to setup Phonemes in LipSyncPro component I get an error and the editor UI bugs out:
    Code (CSharp):
    1. ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
    2. Parameter name: index
    3. System.ThrowHelper.ThrowArgumentOutOfRangeException (System.ExceptionArgument argument, System.ExceptionResource resource) (at <7d97106330684add86d080ecf65bfe69>:0)
    4. System.ThrowHelper.ThrowArgumentOutOfRangeException () (at <7d97106330684add86d080ecf65bfe69>:0)
    5. System.Collections.Generic.List`1[T].get_Item (System.Int32 index) (at <7d97106330684add86d080ecf65bfe69>:0)
    6. RogoDigital.Lipsync.BlendSystem.SetInternalValue (System.Int32 blendable, System.Single value) (at Assets/DMAI/3rd Party/Rogo Digital/LipSync Pro/Classes/BlendSystem.cs:230)
    7. RogoDigital.Lipsync.BlendshapeBlendSystem.SetBlendableValue (System.Int32 blendable, System.Single value) (at Assets/DMAI/3rd Party/Rogo Digital/LipSync Pro/BlendSystems/BlendshapeBlendSystem.cs:65)
    8. LipSyncEditor.OnInspectorGUI () (at Assets/DMAI/3rd Party/Rogo Digital/LipSync Pro/Editor/LipSyncEditor.cs:791)
    9. UnityEditor.UIElements.InspectorElement+<CreateIMGUIInspectorFromEditor>c__AnonStorey1.<>m__0 () (at C:/buildslave/unity/build/Editor/Mono/Inspector/InspectorElement.cs:462)
    10. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
    In BlendSystem.cs:230, _blendables has 12 elements and blendable = 18.
     
    Last edited: Oct 23, 2019
  7. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Not a whole lot, I'm afraid. I've looked into this before, and the frame drops came from MCS' internal morph system. (Enable deep profiling and follow that tree down as far as it goes, almost all of LipSync.LateUpdate()'s frame time there is coming via the MCS BlendSystem from some MCS internal class.) I'd hoped they would eventually fix this, then the entire product became deprecated. The best options you have are using as few morphs per-pose as possible, and limiting the number of clips you have playing at once. I'm sorry I don't have any better advice here!

    Is this on a new character? There shouldn't be any way for those to get out of sync like that. What you can do is put the Inspector into debug mode, find the "Phonemes" list and remove anything with too high a number.
    If you can tell me what happened to cause this though, that'd be a big help in figuring out how to stop it from happening!
     
  8. NawarRajab

    NawarRajab

    Joined:
    Aug 23, 2017
    Posts:
    18
    Yes, my workaround is to only assign blendshapes with lower numbers for now. it's not really a good solution, but at least it doesn't break things for the time being.

    I added a debug log for characterMesh in BlendshapeBlendSystem in GetBlendables (line 82). When I expand the phoneme tabs of the character, I get a print out of all the characters(skinned meshes) in my scene so it seems that _Blendables in BlendSystem is getting overwritten buy other characterMesh which has a lower number of blendshapes. I'm not sure how this "leak" is happening as I can't find any script that assign characterMesh (maybe by reflection?).

    To give you a full picture of what I'm doing:

    -I have 5 characters in the scene. They have an average of 12 blendshapes. The character in question has 24.

    - I have reorganized LipSyncPro in 2 assembly definitions, run time assembly and editor assembly. In order to do so, I had to put everything under editor folder in one editor folder (I tried keeping the file structure as is but there are inter dependencies). This doesn't affect anything.

    upload_2019-10-25_9-35-38.png

    - I tested the plugin as is in a new project. still getting the same behavior.
    - If I remove all characters from scene except that character, debug log only prints out that character skinned mesh, but still get the same error.
     
  9. Arctel

    Arctel

    Joined:
    Apr 9, 2015
    Posts:
    16
    Sent an e-mail for support but have not gotten a response.

    Installed asset, even on a blank project to make sure no conflicts, the example scenes aren't working. The eyes move but not the lips.

    Get this warning when hitting 'Start Talking':
    Playing pre-processed clip on a different character. The animation may look incorrect or not play at all. You can remove the pre-processed data by selecting the clip in the Project window.
    UnityEngine.Debug:LogWarning(Object)
    RogoDigital.Lipsync.LipSync:playPP(LipSyncData, Single, Single) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:636)
    RogoDigital.Lipsync.LipSync:play(LipSyncData, Single) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:707)
    RogoDigital.Lipsync.LipSync:play(LipSyncData) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:765)
    UnityEngine.EventSystems.EventSystem:Update()

    If I recompile get two of these warnings:
    Assets\Rogo Digital\LipSync Pro\Editor\LipSyncClipSetup.cs(608,3): warning CS0618: 'SceneView.onSceneGUIDelegate' is obsolete: 'onSceneGUIDelegate has been deprecated. Use duringSceneGui instead.'
     
  10. Alvarezmd90

    Alvarezmd90

    Joined:
    Jul 21, 2016
    Posts:
    79
    Slightly off-topic but I'm wondering about this.. I'm not sure if this is true either.
    But wouldn't being 'in editor' and having an object selected or having the gameobject's expanded in the hierarchy also affect profiling performance? I'm pretty new to that profiling thing though, so I have no clue.
     
  11. dgoyette

    dgoyette

    Joined:
    Jul 1, 2016
    Posts:
    2,067
    Small feature request: It's typical for me to slightly tweak audio clips over time, requiring me to re-process my LipSync Pro files to account for the changes. I'd like to be able to do this with the Batch Processing dialog. However, it seems like the Batch Processing can only be used to create new LSP assets, not update existing assets. Is that correct? If so, it would be nice if the batch process could accept LipSync Pro assets and reprocess them, in addition to its current functionality.
     
  12. jeromeWork

    jeromeWork

    Joined:
    Sep 1, 2015
    Posts:
    305
    Wanted to say thanks to @DGordon for sharing his Timeline playable from back in 2017.

    @Rtyper Is there still no official playable for LipsyncPro?
     
    DGordon likes this.
  13. DeadSeraph

    DeadSeraph

    Joined:
    Sep 8, 2017
    Posts:
    84
    Hi there. I recently purchased this and I'm just in the early stages of testing it out, but it's not playing any of my audio files in the clip editor. The original audio file was an MP3, and that didn't work, so I tried installing SoX as per the documentation and that didn't work either (the green light shows up when I go to see if it's verified). I also tried converting it to a 16 bit wav but that also didn't work.

    Any ideas?

    *Edit to add: It plays fine in unity when I push play to preview it while it's selected, so I don't think it's a problem with the clip, it just doesn't want to play in the clip editor (and yes, the waveforms do show up)
     
    Last edited: Nov 8, 2019
  14. jeromeWork

    jeromeWork

    Joined:
    Sep 1, 2015
    Posts:
    305
    My turn to give something back to the community...

    I've found a workflow that allows you to combine multiple skinned meshes on one or more characters, using MeshBaker, while still being able to target each individual character's blendshapes in LipSyncPro:


    I've added a few lines of code to the BlendshapesManager.cs script that allows you to filter the blendshapes on a mesh as they're parsed into the dictionary that LipSyncPro uses. @Rtyper would you be happy for me to share this altered script, or perhaps I could let you have a copy for you to include as part of your integrations modules?
     
    DesertJay likes this.
  15. DeadSeraph

    DeadSeraph

    Joined:
    Sep 8, 2017
    Posts:
    84
    Hi everyone.

    Is there a guide somewhere about doing a clean install of Lip Sync Pro? I can't seem to find one on the website. I ended up getting it working in a clean project with nothing else in it, but as soon as I try to get it set up in our main project so we can integrate it with our game, it doesn't seem to want to install the montreal forced aligner module. I can see the files and the folder sitting there, but it's like lip sync pro doesn't know they're there when I run the autosync setup wizard.

    I also get a constant stream of compiler errors when I open the clip editor to check the settings there, and none of the demo scenes work that come with the plugin (they'll play but the lip syncing doesn't activate with the audio).
     
  16. nocanwin

    nocanwin

    Joined:
    May 7, 2009
    Posts:
    139
    can you add a null check for boneCurves on line 611 of LipSync.cs? I'm getting NullReferenceExceptions after upgrading to 1.51, thanks! :)
     
    Last edited: Nov 14, 2019
  17. DeadSeraph

    DeadSeraph

    Joined:
    Sep 8, 2017
    Posts:
    84
    Has this asset been deprecated? o_O
     
  18. DesertJay

    DesertJay

    Joined:
    May 8, 2019
    Posts:
    3
    Not yet. The dev says he's working on 2 updates: 1.6 for the upcoming 2019.3 release and 1.52 for folks who are stuck with pre-2019.3 Unity. I get the feeling it's an individual developer, not a studio, and they probably have a full-time job outside of this. Which means that their time is limited, especially in holiday season, and they're probably focused on coding instead of forum support right now.

    But I wouldn't be surprised if the dev lets it go in the near future. It seems like an awful lot of work and stress for something that isn't a primary income. I made a recent purchase, too - and I hesitated for just this reason. In the end I threw the dice here instead of going with SALSA because at this stage, I significantly prefer the LSP output over SALSA's.

    But about your problem: I saw some references to MFA issues on the last couple pages of this forum. Maybe you'll find an answer there.
     
  19. Safemilk

    Safemilk

    Joined:
    Dec 14, 2013
    Posts:
    16
    Sounds like there's potentially some good updates coming, was wondering if these errors are related to the situation that Lip Sync Pro is in right now, if so, then it sounds like I just need to wait.

    I installed fresh on Unity 2019.1.2f1 and doing anything related to Lip Sync Pro (viewing the clip editor, adding the component to a game object etc.) will throw any combination of these errors:

    upload_2019-11-14_16-32-17.png

    Sorry if I'm beating a dead horse but I wasn't sure if these errors were the ones related to the state of things (Lip Sync Pro having impending compatibility patches) or if this is aberrant behavior.

    Thanks in advance, hoping it's just something dumb that I did, or that it really is just a situation of standing by for the Lip Sync Pro updates that were talked about.
     
  20. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Apologies for the wait guys - I've been working on the new LipSync updates, I've got a couple of other projects on the go and I'm getting ready to move house in just under 2 weeks, so things will probably continue being a bit sporadic until everything's settled down!

    The main thing is LipSync 1.52 was submitted to the Asset Store this morning. It ended up having a few more changes in than I was originally planning which delayed it a bit. Full changelist at the end as per usual.

    @jeromeWork That's awesome! Feel free to share it, just keep the core LipSync API scripts (LipSync.cs, BlendSystem.cs etc) out and there's no problem.

    @DeadSeraph The Getting Started guide on the website covers basic installation, and use of the LipSync component and the Clip Editor are covered on the YouTube channel, though I want to do text versions of these at some point as well. If the wizard isn't detecting the right path automatically you can browse for the mfa_align application yourself and it should then work.
    Could you post the errors you get when opening the clip editor? I haven't run into anything like this before, so I'd help to see what exactly is happening.

    @nocanwin Good call about the null check there, sorry I missed this before submitting the last update, but I'll get on it for the next one.

    @Safemilk I can't say I've seen this happening before, though that first error (ReflectionTypeLoadException) looks interesting. Is there a stack track or line number in the full version of that error at the bottom of the console? Both AutoSync and the editor for the LipSync component use reflection in order to find modules and blend systems in the project, but I can't think of a reason it would break the clip editor.
     
    Safemilk and DesertJay like this.
  21. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Quick update, 1.52 is now available. Here's the changelist that I forgot to include in the previous post :D

    1.52
    Features

    • Added support for saving and loading LipSync presets outside a folder named "Presets".
      • Presets can now be saved anywhere, and a new "Display Name" property is used for sorting into categories.
      • Added a new custom inspector for presets, showing an overview of the stored data, and allowing the display name to be edited.
    • Improved AutoSync Preset UI in the AutoSync window to allow presets to be added to, overwritten or saved as copies.
    • Added AudioClip merging support to AutoSyncConversionUtility.
    • Added GetAnimationCurves() method to TransformAnimationCurve to support the animation clip export package.

    Fixes
    • Fixed bug where deleting pre-processed data from a clip didn't get serialized.
    • Fixed loading a preset on a prefab in prefab mode not forcing the prefab to save.
    • Fixed how the "Yes" and "No" options in the "Save File" dialog show up.

    Changes
    • Made some data in LipSyncClipSetup public so extensions can access them.
    • Removed some legacy code supporting Unity 5.3.5 and lower.
    • Removed old "BlendshapePreset" class that has been deprecated since 1.0.
     
    DesertJay and Safemilk like this.
  22. Safemilk

    Safemilk

    Joined:
    Dec 14, 2013
    Posts:
    16
    Hey, Rtyper! Great news on the new version, I'll do a clean install of Lip Sync Pro and see if the issue persists!

    Before the update though I'll send the error with the line locations of the error just for posterity, then I'll see how I fare with 1.52! Thanks again!

    Here's the full error.

    upload_2019-11-15_9-43-53.png
     
  23. Safemilk

    Safemilk

    Joined:
    Dec 14, 2013
    Posts:
    16
    Dang, no luck, I removed Lip Sync Pro and installed the new version fresh, when I opened the clip editor these 3 errors popped up:

    upload_2019-11-15_9-55-9.png

    So then I made a fresh project and brought 1.52 into it to see if there was something in my project interfering with it. A fresh project with 1.52 seems to work perfectly, until I restarted! Then the errors started coming back.

    Very strange. We are on 2019.1.2f1 Pro still.
     
    Last edited: Nov 15, 2019 at 5:24 PM
  24. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    No problem, and thanks for the line numbers! I've looked into the exception and it means there's some class in your project that isn't loading correctly. I'd guess it's possibly a DLL as you don't have any compiler errors. What other assets do you have in the project?

    Either way, I can add a check for this to that class that should make it usable, or at least help us figure out what's breaking it. I'll PM you a patch when I get it sorted.
     
    Safemilk and DesertJay like this.
  25. NawarRajab

    NawarRajab

    Joined:
    Aug 23, 2017
    Posts:
    18
    @Rtyper hey, did you get a chance to look into the bug I mentioned?
     
  26. DeadSeraph

    DeadSeraph

    Joined:
    Sep 8, 2017
    Posts:
    84
    Thanks for the reply. Wasn't sure if the asset had been deprecated or not. I will grab the latest update and see if that solves the issue I was having previously. If not, I'll post again with screenshots.
     
  27. DeadSeraph

    DeadSeraph

    Joined:
    Sep 8, 2017
    Posts:
    84
    So, I installed the new update and it still doesn't want to recognize the montreal forced aligner extension. I run the wizard, wait for it to install, it installs, compiles, then the wizard doesn't recognize it as installed, and the console displays the following (bottom most error, the others are unrelated):


    *Edit to add*: The settings for autosync in the clip editor do not show any way for me to point the plugin to the extension manually (only shows 1 for sox)
     

    Attached Files:

  28. DeadSeraph

    DeadSeraph

    Joined:
    Sep 8, 2017
    Posts:
    84
    Also, when I try to download the forced aligner extension manually, it says it's already installed.
     
  29. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    I think those other errors are actually the problem. It looks like there are errors in some script code (or possibly a DLL, I'm not 100% sure) that are preventing Unity from compiling all the scripts in your project. This would explain why the files for the MFA module could be in the project, but LipSync can't detect it because it hasn't compiled.
     
  30. Safemilk

    Safemilk

    Joined:
    Dec 14, 2013
    Posts:
    16
    Hey thanks a ton for the info, that definitely gives me a lead to look into, I'll see what I can turn up, we definitely have a ton of assemblies in our project so this could actually be code smell to a much larger issue!
     
  31. DeadSeraph

    DeadSeraph

    Joined:
    Sep 8, 2017
    Posts:
    84
    Restarting unity did the trick. This didn't work with the previous version, so I'm not sure what changed with the build you released today, but it's functioning now :)
     
  32. Dragonic8926

    Dragonic8926

    Joined:
    Mar 10, 2014
    Posts:
    28
    Hello,

    I can see from the roadmap on your website that French and German langage models for MFA should be available since August, but I can't find this extension in the plugin. How can I get this ?