Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

Discussion in 'Assets and Asset Store' started by Rtyper, Mar 11, 2015.

  1. minad2017

    minad2017

    Joined:
    Dec 1, 2016
    Posts:
    9
    I am in trouble because I do not know the chance of this trouble occurred.
    It worked without any problem when I verified it with other personal computer.
    On a computer with a problem, even if you install Unity separately and introduce Lipsyncpro, the same problem has occurred.
    By the way, when reinstalling Lipsyncpro, data such as shortcut keys were automatically inherited.
    Is this that the old data remains in the project?
    How can I delete all Lipsyncpro data once?
     
  2. minad2017

    minad2017

    Joined:
    Dec 1, 2016
    Posts:
    9
    I deleted all the windows temp folder
    As a result, Lipsyncpro now reads the file without problems.
    I do not know the root cause, but it seems there was some problem with Unity's temporary file.
    Currently it works steadily,
    thx.
     
  3. DerDicke

    DerDicke

    Joined:
    Jun 30, 2015
    Posts:
    181
    The WebGL demo doesn't work for me. Abraham is not moving his lips and sound cannot be stopped.
    On Win7, Firefox.
     
  4. juliemooster

    juliemooster

    Joined:
    Jun 13, 2017
    Posts:
    2
    Hi. I'm using LipSync with Unity 5.6 and I just have a quick question that I'm hoping that you can point me in the right direction of. I've tried to look for the answer via online searches, in the tutorial videos and documentation but can't find it anywhere.

    I have separate meshes for the body (which includes the face), the teeth and the tongue. These are all parented to a game object to which I've added the LipSync property. I've got the main Character Mesh as the body element, and then I've added 2 x Optional Other Meshes of: (1) the tongue and (2) the teeth.


    What I was expecting to see in the drop down list of blend shapes when I'm adding shapes to Phonemes though were options for the teeth and tongue, but they aren't there.



    Could I ask you please to point me in the right direction of where I'm going wrong.

    Many thanks in advance for your help.
     
    Last edited: Oct 9, 2017
  5. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Well I'm glad it worked out! Weirdly this is the second time in a few weeks I've heard of temporary files causing problems in LipSync. I'll have to keep an eye on this one!

    Thanks for letting me know. I'll look into it.

    This is a situation when you'd need to use the 'advanced blendshape blend system' in LipSync Pro, but it's not currently included with LipSync Lite.
    I'd read your email, and I was actually trying to figure out if there was a simple way to get it to work with Lite, but there are a few more recent updates in Pro that it needs to work. I'm working on an up-to-date version of Lite now, which will include it - It'll probably be about a week away. Otherwise, if you get Pro, it'll work right away.
     
    juliemooster likes this.
  6. Mbcburt

    Mbcburt

    Joined:
    Aug 11, 2014
    Posts:
    8
    I just bought the Pro version. I am really liking it, great product. I did notice a few things with the emotions updating correctly and adding new emotions updating in the editor correctly. Also with the emotions, I was expecting them to be additive to the Phonemes. currently when ever there is a phoneme key it overides the emotions. This is mostly obvious in the mouth areas. For this to work for me I need the emotions to overlay or be additive to the Phonemes. I am using Unity 2017.
     
  7. Rubychung33

    Rubychung33

    Joined:
    Oct 23, 2017
    Posts:
    1
    I'm sorry if you've answered this question already. Will this program work with clay animation?
     
  8. gsurla

    gsurla

    Joined:
    Sep 10, 2014
    Posts:
    5
    Was there ever a solution to this issue? I'm running into it fairly regularly on Pro 1.401
     
  9. ColtonK_VitruviusVR

    ColtonK_VitruviusVR

    Joined:
    Nov 27, 2015
    Posts:
    147
    Hey gsurla,

    I didn't end up sending a sample into the developers so nothing was done to my knowledge. I ended up just using a coroutine that waited in parallel to the Lipsync playing.

    Cheers,
    Colton
     
  10. gsurla

    gsurla

    Joined:
    Sep 10, 2014
    Posts:
    5
    Hi Colton,
    Thanks for the response. That's what I'm doing currently...just curious if there was a problem with my usage since we seem to be the only ones in the forums having the issue.
    Thanks,
    gsurla
     
  11. Foysal-Ahmed-Emon

    Foysal-Ahmed-Emon

    Joined:
    Mar 13, 2016
    Posts:
    8
    I need HELP ! I want to work with the AutoLipSync feature runtime. I can't find any documentation. Can you help me?
     
  12. davidosullivan

    davidosullivan

    Joined:
    Jun 9, 2015
    Posts:
    385
    I'm using LipSync with UMA in an AR project (LSP is flippin great BTW!)

    Problem I am having though is that I need Lipsync to 'preload' somehow. What is happening for me at the moment is that when my augmentation is detected, and the game objects are enabled, the whole thread stops as (I think) the audio files decompress and LipSync does its 'PrepareData'. So what I am after is some kind of 'Preload' checkbox on Lipsync that will make it do PrepareData when the scene starts. I can then disable the objects, and when I re-enable them, the lipsync should already be prepared and play without a pause.

    This is slightly complicated by the fact I have multiple 'augmentations' in my scene- sub-scenes if you will- that get turned on and off as their tracking images are discovered. But I think if I could just get LipSync to 'preload' that would solve it.

    I have tried to write a script of my own but it doesn't work:
    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using RogoDigital.Lipsync;
    5.  
    6. namespace Hurrar.LipSyncAddons
    7. {
    8.     /// <summary>
    9.     /// With long clips LipSync stops the main thread while is processes data. This is bad for AR because we need it to start immediately when
    10.     /// The augmentation is found. This component makes Lipsync 'Play' the given file at zero volume, which causes it to PrepareData. Then once
    11.     /// it is playing the playhead for the file is set back to zero, ready to 'Resume' and the AudioSource volume is reset to its starting level
    12.     /// </summary>
    13.     public class LipSyncPreloader : MonoBehaviour
    14.     {
    15.         public LipSync lipSync;
    16.         public AudioSource audioSource;
    17.         public LipSyncData dataToLoad;
    18.         public bool preloadOnAwake;
    19.  
    20.         private AudioClip originalAudioClip;
    21.         private float originalAudioSourceVolume;
    22.         private float originalAudioSourcePlaybackTime;
    23.         private bool originalAudioSourcePlayOnAwake;
    24.         private bool playOriginalAudioSource;
    25.  
    26.         private bool wasPreloaded;
    27.  
    28.         void Awake()
    29.         {
    30.          
    31.         }
    32.  
    33.         // Use this for initialization
    34.         void Start()
    35.         {
    36.             if (preloadOnAwake)//think we have to do this on start because LipSync is not ready on Awake
    37.             {
    38.                 PreloadLipSync();
    39.             }
    40.         }
    41.  
    42.         // Update is called once per frame
    43.         void Update()
    44.         {
    45.  
    46.         }
    47.  
    48.         void PreloadLipSync()
    49.         {
    50.             if (lipSync == null || audioSource == null || dataToLoad == null)
    51.                 return;
    52.  
    53.             if (audioSource.isPlaying)
    54.             {
    55.                 playOriginalAudioSource = true;
    56.                 originalAudioSourcePlaybackTime = audioSource.time;
    57.                 audioSource.Pause();
    58.             }
    59.             originalAudioClip = audioSource.clip;
    60.             originalAudioSourceVolume = audioSource.volume;
    61.             audioSource.volume = 0f;
    62.             lipSync.Play(dataToLoad);
    63.             Debug.Log("LipSyncPreloader preloaded " + dataToLoad.name);
    64.             StartCoroutine(PauseLipSyncAfterPreload());
    65.         }
    66.  
    67.         IEnumerator PauseLipSyncAfterPreload()
    68.         {
    69.             while(lipSync.CurrentTime == 0)
    70.             {
    71.                 Debug.Log("LipSyncDidn't start playing yet...");
    72.                 yield return null;
    73.             }
    74.             Debug.Log("LipSyncStartedPlaying- Pausing LipSync!");
    75.             wasPreloaded = true;
    76.             lipSync.Pause();
    77.             audioSource.clip = originalAudioClip;
    78.             audioSource.volume = originalAudioSourceVolume;
    79.             if (playOriginalAudioSource)
    80.             {
    81.                 audioSource.time = originalAudioSourcePlaybackTime;
    82.                 audioSource.Play();
    83.             }
    84.         }
    85.  
    86.         public void PlayPreloadedLipsync()
    87.         {
    88.             Debug.Log("LipSyncPreloader tried to play preloaded " + dataToLoad.name);
    89.             audioSource.clip = dataToLoad.clip;
    90.             lipSync.PlayFromTime(dataToLoad, 0f);
    91.         }
    92.  
    93.     }
    94. }
    95.  
    Basically somehow I need all my audio and Lipsync to play instantaneously when the augmentation happens because all the animation in it is synced to the audio!

    EDIT: Getting rid of the audioSource stuff in there seems to make that script kind of work...
     
    Last edited: Nov 1, 2017
  13. davidosullivan

    davidosullivan

    Joined:
    Jun 9, 2015
    Posts:
    385
    Also the viewer might view the augmentation again, so when lipSync ends I want it to keep its data ready for playing again... It looks like It might already do this actually- but calling Play always makes ProcessData happen, even if tthats alread been done... After more tests, its def this that causes my game to pause...

    So, to summarize, I want to ProcessData when the scene loads, and then be able to Play without 'ProcessData' happening again. and if the LipSync gets to the end, and the same lipsyncData gets played again, I want that to play without doing ProcessData again...
     
    Last edited: Nov 1, 2017
  14. ColtonK_VitruviusVR

    ColtonK_VitruviusVR

    Joined:
    Nov 27, 2015
    Posts:
    147
    Hey @Rtyper,

    This might tie into other reported problems of the lipsync finish event not being called which I and @gsurla have experienced. Currently we are having issues with a few characters in our scene (I think it is the same problem). We have a character who is smoking and uses lipsync for facial animations (exhaling smoke etc.). At the end of the smoking loop it checks LipSyncController.IsPlaying to see if it can smoke again. However, the IsPlaying is always true, which is why I think it might tie into the finish event not being called! This issue only happens in standalone builds and does not happen all the time (I think it has something to do with the character being culled while the lip sync is playing?). But once it locks up it will never go to false again. I have tried updating the skinned mesh when offscreen but that doesn't help. We have a demo event next week, is there anyway you could look into this for us?

    Cheers,
    Colton
     
  15. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Apologies for not actually replying to you sooner, but I've been working on this recently! I'll be putting it into the next update (but not 100% sure when that will be right now) and I can send you a patch with just this feature over private message tomorrow.

    As in 3D scanned clay models, or sprites based on clay models - sure. If you mean actual, physical clay animation you may be looking for the other Lip Sync Pro ;)

    @gsurla and @ColtonK_VitruviusVR This has always been an issue I've been unable to recreate, so this new info is definitely useful! The event is fired by the same part of the code that transitions back to the neutral pose after an animation and sets IsPlaying to false, so if you're finding that isn't being reset, it sounds likely that the two issues are linked. For what it's worth, those things won't happen if the component is set to loop (as it just goes back to the start of the animation), but I'm assuming that's not the case here. Culling is unlikely to affect it, unless the LipSync component is being deactivated as well (in which case the animation should stop playing too), but I'll consider it. Are you using Unity's occlusion culling?

    I'll look back into this asap, but until I can reproduce it I'm not sure how much help I can be. If you're able to put together a scene/project file you could send me so I can see it first hand that would be invaluable, but I get that can sometimes be a problem with legal departments etc!
     
  16. ColtonK_VitruviusVR

    ColtonK_VitruviusVR

    Joined:
    Nov 27, 2015
    Posts:
    147
    Im not really surprised you haven't been able to reproduce the issue, I just tried in a demo scene with no success. I have tried re-importing all lip sync with no success. I'll keep investigating!

    Have a good weekend,
    Colton
     
  17. ColtonK_VitruviusVR

    ColtonK_VitruviusVR

    Joined:
    Nov 27, 2015
    Posts:
    147
    Hey @Rtyper,

    So what I noticed is the LipSync.IsPlaying gets stuck on true, but the current time logs out at 0. For now I have replaced all of our LipSync.IsPlaying with LipSync.audioSource.isPlaying which seems to fix everything for now.

    Cheers,
    Colton
     
  18. Alex-3D

    Alex-3D

    Joined:
    May 21, 2013
    Posts:
    78
    Hi, Rtyper,
    I tried Lipsync Lite with AC and MCS. Doesn't work. I don't know how Pro works, but this Lite version is a real antiPR. I returned back to Salsa. Is there any reasons to change my mind?
     
    Last edited: Nov 7, 2017
  19. Devistute

    Devistute

    Joined:
    Aug 23, 2014
    Posts:
    32
    I'm having an pretty interesting problem with Lip Sync Pro, latest version.
    LipSync Pro won't recognize my Adobe Fuse CC character's Blendshapes at all. (They are same as in legacy Fuse).
    If i drag the "Body" child of the character to the "Character Mesh" slot, it gives the following error.

    If i however drag the same "Body" to the slot again, the error goes away.
    But even though it goes away, the blendshapes are not set up, not a single one.

    The blendshapes however, ARE THERE. The script just apparently doesn't recognize them.
    As said above, its most recent version imported literally 15 mins ago and my Unity version is 2017.2.0f3

    Any help with this?

    Edit; LipSync Pro did work with the same character before, but now it doesn't.
     
  20. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    OK, thanks for the info. I've just received another report of what sounds like the same issue, so hopefully I'll get to the bottom of it fairly soon!

    I've actually just submitted an update to LipSync Lite! You're right, it has been neglected for a long time and the current version wasn't compatible with Unity 5.4 or up. If you feel like giving it another try, the new version should be approved within a week. It's definitely compatible with MCS, and I believe it should work with AC (as Pro is) but I'll need to test that out.

    Once you've added your meshes, just pressing the "Continue" button will dismiss the warning and show the rest of the editor, adding the same mesh again just triggers some code that basically does the same thing. :) I can see how this might be a bit confusing though, I'll look at simplifying this for 2.0!

    As for the blendshapes just not being recognised, I'm not sure what the problem could be without seeing the model really. Is it just a Fuse character downloaded straight from the auto rigger, or has it been modified afterwards (in Max, C4D, Blender etc)? If it is, could you show me screenshots of the blendshapes list on the Skinned Mesh Renderer and where LipSync is failing to show them? I haven't come across any problem like this with Fuse characters before!
     
  21. Devistute

    Devistute

    Joined:
    Aug 23, 2014
    Posts:
    32
    Its downloaded straight from auto rigger, yes. It has not been modified on other programs.
    Here's some screenshots.




     
  22. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    You need to press "Add Blend Shape" to add a blend shape from your mesh to that pose. They're not just all added by default as you probably only need a small number of them per pose!
     
  23. Devistute

    Devistute

    Joined:
    Aug 23, 2014
    Posts:
    32
    Oh my god, i'm blind apparently. I didn't notice the preset button at all on right.
    Sorry, and thanks! :D
     
  24. spootymilk

    spootymilk

    Joined:
    Jun 9, 2017
    Posts:
    15
    Hi,
    I'm using unity 2017.2 on windows 10.

    all the time i quit a scene with a MCS character witch has a lipsyncpro and a eyecontroller scripts attached, the is an error saying: Capture.PNG

    what can i do to avoid it ?
    until today, i just ignore it, but if there is an error, it's for a good reason....

    thank s a lot for your help.
     
  25. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Just a quick news update first - we've just updated LipSync Lite to version 1.401! It's now on par with the current version of LipSync Pro (Phoneme Sets are no longer a Pro-only feature!) , and it's compatible with Unity 5.5 all the way up to 2017.2 (and probably 2017.3, though I haven't tested it). Sorry to everyone who's had problems with the old version for not updating this sooner, but there were some issues with the compiled code designed for 5.4 and lower not working on newer versions. In the end, it just made sense to drop support for those older versions.

    This new version is now the same as LipSync Pro 1.401 but without the following features:
    • Emotion Markers
    • Gesture Markers
    • AutoSync
    • Eye Controller component (can still be bought separately)
    As before, you can download it for free from the Asset Store:
    https://www.assetstore.unity3d.com/en/#!/content/61886



    No problem! It's pretty easy to miss - I'm trying to make it stand out a bit more in 2.0.

    Just done a quick test myself and I don't get the same errors when changing scenes. Those errors mean that the Eye Controller component is somehow unregistered from the Morph3D blend system it's using, but doesn't seem to realise that! Is there any chance you could send some test project or scene to me so I can look into it? Either as a PM here or to contact@rogodigital.com. Without that or at least some more info about your project/code there's not much else I can guess at.
     
  26. JoMaHo

    JoMaHo

    Joined:
    Apr 2, 2017
    Posts:
    94
    Hi @Rtyper - how is it going with AutoSync on Mac OS Sierra?
     
    Last edited: Nov 14, 2017
    kukuhbasuki and wetcircuit like this.
  27. arie1994

    arie1994

    Joined:
    Jun 30, 2016
    Posts:
    67
    Hello
    Is there integration with the new Timeline sequencer in unity 2017?
    Can the facial animation blend with body animation?
     
  28. zukinet

    zukinet

    Joined:
    Oct 30, 2016
    Posts:
    45
    Hi, i'm interested to this plugin. Can be used to create Talk Back function (just like Talking Tom) ?
     
  29. kukuhbasuki

    kukuhbasuki

    Joined:
    Apr 13, 2016
    Posts:
    28
    Does it still not support MacOS Sierra?
     
  30. kukuhbasuki

    kukuhbasuki

    Joined:
    Apr 13, 2016
    Posts:
    28
    Yes, I’m waiting for this too.
     
  31. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    857
    @Rtyper -- I've been wondering for a long time if this could be used to animate simple face textures on a model. For example, assuming I have various sets of eyes/mouths/etc. that change, would you be able to easily make it to where when setting up phonemes, I could pick a texture that would display on the model's face instead of blendshapes or bones?

    How long would it take to implement something like this? -- Could I do something to implement it faster myself?
     
  32. seamon67

    seamon67

    Joined:
    Jan 30, 2016
    Posts:
    3
    I get bombarded by this error message, like 999+ errors whenever I start Unity and I am unable to play any scene

    ArgumentOutOfRangeException: Argument is out of range.
    Parameter name: index
    System.Collections.Generic.List`1[RogoDigital.ContinuationManager+Job].get_Item (Int32 index) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633)
    RogoDigital.ContinuationManager.Update () (at Assets/Rogo Digital/Shared/Editor/ContinuationManager.cs:28)
    UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at C:/buildslave/unity/build/Editor/Mono/EditorApplication.cs:127)
     
  33. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    857
    What version of Unity are you using? -- I ask because it seems support might be lacking a bit... :(



    @Rtyper
    Hopefully you'll get around to answering my other question, but also, I wonder -- is Timeline/Cinemachine supported with this out of the box, or does one need a system like the Dialogue System asset to make this work with Timeline?
     
  34. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Still in progress, I'm afraid! I'm working with another developer (with more C++ experience) on solving the issue with the current set of .dylib libraries on Sierra. I had hoped to get an update from him on how long this would take before replying here, but it's taking longer than expected. I'm really sorry about this - I promise I'll have a patch ready as soon as I have the fixed files!

    As you both asked about Timeline support - the short answer is no, at present. I'm currently in the process of re-writing a lot of LipSync Pro for version 2, and the new design works with Timeline out of the box, but the current version's design just doesn't gel well with how Timeline works. I do have a fairly rough Timeline support extension that I wrote for another customer that I can send over if it's really urgent though - just bear in mind that there may be bugs/problems with it!

    @arie1994 I'm not really sure what you mean by your second question. LipSync Pro works separately to the Animator component, so if you want to have body animations playing on and Animator, then yes - LipSync's facial animations will work with that.

    @awesomedata That is something that could be set up using a custom Blend System. Specifically you should look at the built-in Sprite Blend System, as that does something similar only with Sprites. You'd want to have a manager component (like the Sprite Manager) that stores references to Texture2Ds and possibly the Material(s) they go with, and a Blend System class that sets the textures based on the highest-value blendable (again, look at the Sprite Blend System for this). It's not the most efficient way of doing things, as it has to create several animation curves which are basically ignored, but it's the only way at the moment without modifying LipSync Pro's core code. (This is something else I'm addressing in LipSync 2 though.)

    I'm not really familiar with that app, but I'm going to say probably not. It looks like that uses text-to-speech to generate audio at runtime, whereas LipSync Pro requires the audio to exist in the editor so it can be processed.

    Is this in the latest version of LipSync Pro? As awesomedata asked, which Unity version are you using? Does this happen as soon as you open the editor or do you have to do something first to trigger it? I know where the error is coming from, but I thought I had fixed it a couple of updates ago!
     
  35. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    857
    The next thing I want to ask is, if I purchase LipSync Pro now, will I have access to LipSync 2 as well? -- It seems like this should be the case, but I want to know since I want only a simple use of LipSync's feature set with changing the textures on my models, and I'm going to have to write this functionality myself anyway it seems before I'll even be able to use LipSync, but I want to know for sure.

    My needs are simple and there's not another package on the store that can do general LipSyncing as well as it seems LipSync can do, but I want to know if it would be worthwhile to purchase this.
     
    Last edited: Nov 26, 2017
  36. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    857
    Would you mind sharing with me a list of classes/scripts I'd need to alter/study/implement to be able to manage this with the least performance impact? -- I'm sure there's a lot to LipSync, but I don't have a ton of time to learn the backend of the system to get something like this up and running. I took a peek at the custom Blend System stuff, but I just didn't have time to delve into it. Any help pointing me in the right direction with info like the above would help immensely. Thanks!
     
    Last edited: Nov 26, 2017
  37. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    LipSync Pro 2 will be a paid upgrade, for about $10 (and the base price will go up to $40). I know of one company that bought LipSync for the clip editor and actually don't use the runtime component at all, but obviously your mileage may vary - that might not be worth it for you. As it's part of support though, I'd be happy to write the blend system myself if you do end up going with LipSync. It won't be a particularly complicated one to create, especially with the Sprite blend system to base it off!
     
  38. seamon67

    seamon67

    Joined:
    Jan 30, 2016
    Posts:
    3
    Yes, this is the latest version of Lipsync Pro v1.401.

    I am using Mixamo characters, I use the tutorial video and follow the steps exactly. After setup it doesn't work. It used to work with an earlier build + Unity 5.6.

    So after setup, it start giving this error. After I close Unity and reopen it, the error still pops up continuously and I can't enter play mode. The only solution is completely removing Lipsync from the project :/
     
  39. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    857
    That'd be great. I actually purchased LipSync already since I didn't expect v2 to be on the level of an entirely new asset, but that aside, what is the final cost of the new version going to look like, and when are you looking to release it by? -- I might consider upgrading when you release it if you'd be willing to help me deal with some things in this version.

    The few things I would need addressed if you did implement this system (compared w/ the current one) would include:

    1. First, would it be possible to remove some of the overhead of creating curves for nothing (to some extent at least?) w/ your implementation of a texture + uv swap mechanism in the current state of LipSync? Let me explain why I think this shouldn't be terribly hard:

      First, Unity 2017.x implemented Enum animation support, and I believe this is exactly the type of thing it was meant to solve (i.e. using curves for no reason) -- I would be very grateful to be able to use this new feature in the current version of LipSync. I would be totally willing to upgrade if you were willing to support Enums in your implementation of LipSync currently and allow creation of the animation curves to be 'optional' for both sprite and texture animation / blend-system types.

    2. Secondly, would you be willing to allow me to RMB to pick both a texture (with a preview) as well as an Enum that describes the uv offset for the current image (for the gameobject's material) from a pop-up menu where you would normally select phonemes in the timeline? I would think it would be nicer to preset these as 'emotions' instead of directly in the timeline though, which I will describe in #3 now:

    3. I tend to want to change other textures (and possibly materials) separately from the mouth (such as eyebrows) and would like you to make it possible to create 'emotions' based on these material+texture assignments, with only the 'mouth' or 'eye' materials' uv offset moving automatically (for example, one could select an eye 'blink' phoneme, which is actually just an animation across UV x for a number of frames via coroutine or something). Although it animates automatically based on phonemes, I'll need to automatically swap the mouth's texture (or UV y position) based on a chosen 'emotion' too, since I will have, say, an 'angry' mouth, or a 'sad' mouth, or a 'neutral' mouth -- this 'emotion' texture's UV animation x offsets would remain the same across all emotions (such as in the case of the mouth), but only the actual atlas texture image (or UV y value) would change when the 'emotion' is changed.

    4. Finally, I will want to change other aspects of the character's face textures or materials (i.e. wrinkles that tense up when 'angry' or allow the wrinkles to sag a lot when 'sad' in order to give a full range of expressiveness while otherwise talking automatically).

    I hope this is reasonable to ask for. This would be enough for anything I'd want to do with LipSync for now, plus, as it stands, no other asset out there has this sort of functionality, so it would be a great selling point for you anyway imo.

    Hopefully I would still retain the ability to use poses for emotions, just as with regular 3D characters -- just the method of facial animation would be slightly different. To be clear, it should be possible to change both the texture and the uv of the material via "emotion" animation at any point in the timeline in order to maintain proper emotional range. The only difference between this type of "emotion" and standard blendshapes is that there could be multiple materials affected in a single "emotion" and others additional textures/materials that are ignored (or included), depending upon what "emotions" include what materials on what gameobject. For example, only when the character gets "SUPER angry" do the wrinkles appear, while otherwise, just normal "angry" only changes the eyebrows and mouth materials' texture (or UV y offset, to conserve memory and drawcalls).

    You, sir, would have yourself a faithful customer for life if you would be willing to help me with this requirement! -- I've been waiting a long time to find a system to help me do what I outlined above.

    Do you think you could do it without too much effort?


    D:
     
    Last edited: Nov 27, 2017
  40. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    857
    @Rtyper

    Also, regarding the 2D uv/texture approach, I wanted to ask:



    Would it be possible to use a simple range slider for each of these so that they modify UV offsets instead of bone transforms instead?

    "Eye Turn Speed" wouldn't be necessary, but it would be nice to let it modify how slowly the eyes move toward a target (this animation would actually look better as a curve IMO, instead of a specific value, but I'm not picky).

    I would use a simple mask overlay for the eyelids to make them blink too, but in some characters, blinking wouldn't be necessary, so it'd be useful to be able to disable it.


    I really hope there's not much to making this work. -- I think there's a little more to what I want than a custom blend system can handle (maybe not hopefully!), but I really appreciate your offer to help with this! -- With everything I've outlined above, this system would be more than sufficient for what I want to do, and I can't find another LipSync system out there that can do any of this! D:
     
  41. joelcarlson

    joelcarlson

    Joined:
    Apr 7, 2014
    Posts:
    43
    Hi there! I'm getting the the same error message

    ArgumentOutOfRangeException: Argument is out of range.
    Parameter name: index
    System.Collections.Generic.List`1[RogoDigital.ContinuationManager+Job].get_Item (Int32 index) (at /Users/builduser/buildslave/mono/build/mcs/class/corlib/System.Collections.Generic/List.cs:633)
    RogoDigital.ContinuationManager.Update () (at Assets/Rogo Digital/Shared/Editor/ContinuationManager.cs:28)
    UnityEditor.EditorApplication.Internal_CallUpdateFunctions () (at C:/buildslave/unity/build/Editor/Mono/EditorApplication.cs:127)

    999+ errors in the console. I just downloaded and installed today from the purchase I made on the weekend.

    Lip Sync Pro version:
    Unity Version: 2017.2.0.f3
    Lip Sync Pro Version: 1.401
     
  42. joelcarlson

    joelcarlson

    Joined:
    Apr 7, 2014
    Posts:
    43
    Also, I've been trying to look for a solution to my problem. Is there a way to set up Lip Sync Pro to use a sort of blendtree of sprite animations. Here's an example of what I hope to achieve.

    MouthBlending.gif

    I would just like to have an animation for each phoneme and for Lip Sync Pro to manage the blending of those Sprite animations. Is this possible?

    Thanks!
     
  43. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    857
    <__<


    >__>


    D:


    Kinda feel like if the dev was on vacation, he'd at least have left a note...
     
  44. gaillath

    gaillath

    Joined:
    Apr 20, 2017
    Posts:
    10
    Hi,

    I'm getting exactly the same error.
    Is there a fix?

    Thanks
     
  45. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    416
    Sorry for the late response again, everyone - It's been a really busy couple of weeks!

    OK, the problem is that I haven't been able to recreate the issue myself before, but I have an idea that I'm looking into now. If any of you are able to send a project folder where the errors occur to me at contact@rogodigital.com, that'd be a real help! I'll make sure to post back here as soon as I have any update on the problem.

    Apologies for not getting back to you sooner on this! It's unfortunately not a trivial change to make it more efficient like that - the animation curve generation and sampling stuff is built into LipSync itself, all the blend systems do is get an index and a weight (as if dealing with blend shapes) and are expected to set some property with that info. Like I said, that's changing in the future, but for now any system that doesn't really need curves still has to use them, but just apply some kind of threshold.

    A lot of those other changes would also be out of the scope of a custom blend system. Some things (like changing the controls of Eye Controller) might be something I can make customisable in a future update, but most would require basically creating a custom version of the LipSync component just to handle this one specific use-case. I'm really sorry, but I can't really justify the amount of work for the price of one sale! I don't want to have misled you, so I can ask Unity to issue a refund if you don't think you can get an use out of LipSync Pro without these changes.

    That's quite an interesting idea. A blend-tree like system probably wouldn't be doable, as blending sprite animations isn't really possible (unless you do something with multiple sprite renderers and changing transparency), but the basic idea may be possible with just a custom blend system like the included sprite system, only using Animator triggers instead of swapping out sprites.
     
  46. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    857
    It sounds like I *may* have to ask you for a refund, but maybe there's still hope. --

    Although I was really hoping your asset would be the robust LipSyncing solution I was looking for, I unfortunately don't have enough time to study all your code in order to support all the features I need myself. I don't mind a lot of work, but most of that would be spent on trying to understand what already exists and what still needs to be programmed to support the features requested above.

    If you would be willing to at least provide me with the info and support as I need it to program this functionality myself, it's possible I wouldn't mind programming what I need since I indeed need some semblance of LipSyncing functionality.

    I did think the Custom Blend solution might fit with the scope I was looking for, but changing the expressions and eyes based on emotions would be necessary for me to accomplish what I'm looking to accomplish artistically.


    That said:

    The part about changing the editor portion to allow better workflow in the expression selections is something I could probably handle on my own if the core functionality was built in to swap materials and textures in the model based on LipSyncing audio data and emotion data (I don't mind writing the editor portion of that as long as it was supported under the hood somewhere), but that would be something I would have to leave up to you -- otherwise it's something I won't have much time for.

    Is this something you would be willing to compromise on with me?


    EDIT:

    Also, I have just considered perhaps using bones to offset the UVs of a given material/texture -- for example, the orientation of a floating bone (or bones, in my case) would assign a texture to various materials on the model for the eyes, mouth, etc. to change expressions.

    Would this be easier to implement with the existing system?
     
    Last edited: Dec 12, 2017
  47. ratonmalo

    ratonmalo

    Joined:
    Apr 12, 2010
    Posts:
    280
    Hi, been using different lipsync tools and no one have the option Im looking, wondering if lipsync-pro have it or not.

    Im looking to use the realtime lipsync In my character mouth using and Audio, till there is posible with any lipsync tool, from there I need to bake it into an animation and add it to my character face. so will not need the lipsynt tool later.

    It is possible to bake the face blandshape animated by the audio and LimpsyncPro in to an animation?

    I can pay extra for that option.

    thanks
     
    Last edited: Dec 13, 2017
  48. Ryan-Hayle

    Ryan-Hayle

    Joined:
    Feb 16, 2014
    Posts:
    69
    How do you setup LipSync Pro with the MCS Female? I am unable to get it to work.

    >No worries, I thought there was a pre-set for these type of characters, but you have to choose your own.
     
    Last edited: Dec 14, 2017
  49. Pemdora

    Pemdora

    Joined:
    Oct 18, 2017
    Posts:
    7
    Hello, I'm trying to use Lip Sync pro for dialogue generation with several Audio clips (LipSyncData) but when I call Play() function on a clip, it will only play the last clip.

    using UnityEngine;
    using RogoDigital.Lipsync;

    public class MyScript : MonoBehaviour {
    public LipSync lipsyncComponent;
    public LipSyncData clip1;
    public LipSyncData clip2;

    private void Start () {
    lipsyncComponent.Play(clip1);
    lipsyncComponent.Play(clip2); // plays only clip2
    }
    }
     
  50. gaillath

    gaillath

    Joined:
    Apr 20, 2017
    Posts:
    10
    @Rtyper
    FYI: The bug disappeared on my side. Maybe because I deleted the MCS asset? (not sure though).
    Anyway lipsync working fine with my character (non mcs) now.