Search Unity

  1. Looking for a job or to hire someone for a project? Check out the re-opened job forums.
    Dismiss Notice
  2. Unity 2020 LTS & Unity 2021.1 have been released.
    Dismiss Notice
  3. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

[RELEASED] LipSync Pro and Eye Controller - Lipsyncing and Facial Animation Tools

Discussion in 'Assets and Asset Store' started by Rtyper, Mar 11, 2015.

  1. Hazneliel

    Hazneliel

    Joined:
    Nov 14, 2013
    Posts:
    234
    I have an additional issue, it seems that emotions do not blend smoothly when changing clips, For example on the Demo scene if you press the Talk button it will start speaking, if you press it again, the emotion will abruptly change. Emotions should blend smoothly between different lipsync clips. Is there a configuration that can help me with this_
     
    Orbytel likes this.
  2. Hazneliel

    Hazneliel

    Joined:
    Nov 14, 2013
    Posts:
    234
    Further more, I keep observing some problems, on the video you can see that when playing different clips with emotions, if one clip is interrupted and you start playing another one (time 0:10 of the video), the emotion will not go back. This is even if you have the Keep Emotion When Finished flag unchecked.

    Additionally if you try to manually reset the emotion with the ResetEmotion(float time) it wont reset it (time 0:27 of the video). The only option is to use ResetDisplayedEmotions but this will abruptly reset it and doesnt look good.



    I hope you can take a look at this and the previous issues mentioned before and help me with this.

    I appreciate it, Thanks
     
  3. harjotmadhopuri

    harjotmadhopuri

    Joined:
    Jan 10, 2018
    Posts:
    1
    You are Copying Lipsync Component, its not allowed right now. You should add a new fresh LipSync component.
     
  4. pradf4i

    pradf4i

    Joined:
    Nov 7, 2017
    Posts:
    33
    Seeing the following error when trying to install autosync on MAC with Unity 2019.4

    "Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/Modules/Montreal Forced Aligner/ASMontrealPhonemeDetectionModule.cs(44,24): error CS0115: 'ASMontrealPhonemeDetectionModule.ProcessWithTemplates(LipSyncData, AutoSync.ASProcessDelegate, PhonemeMarker, EmotionMarker)': no suitable method found to override"

    Any guidance on how to fix this? Thanks
     
  5. colpolstudios

    colpolstudios

    Joined:
    Nov 2, 2011
    Posts:
    150
    Woah I really like the hair used, did you do this?
     
    Hazneliel likes this.
  6. faizan143143

    faizan143143

    Joined:
    Oct 24, 2017
    Posts:
    4
    AudioClip ModelClip= ModelLip.audioSource.clip;
    float dialougelengthFloat = ModelClip.length;
    List<PhonemeMarker> markers = new List<PhonemeMarker>();
    List<EmotionMarker> Emotion = new List<EmotionMarker>();
    List<GestureMarker> GestureAnim = new List<GestureMarker>();
    int k = 0;
    Phoneme phoneme = 0;
    int phonemeNum=0;
    for (float i = 0.01f; i <= dialougelengthFloat; i += .008f)
    {
    int j =RandomInt.Random(1,9);
    PhonemeMarker phonemeMarker = new PhonemeMarker(tempPhonemeState, i, 1f,false);
    markers.Add(phonemeMarker);
    // ModelLip.GetComponents<AutoSyncDataReady>();
    // AutoSyncDataReady(ModelLip.audioSource.clip, phonemeMarker);
    int z = RandomInt.Random(0, 9);

    if (z == 0)
    {
    tempPhonemeState = Phoneme.AI;
    phonemeNum = 0;
    }
    else if (z == 1)
    {
    tempPhonemeState = Phoneme.CDGKNRSThYZ;
    phonemeNum = 1;
    }
    else if (z == 2)
    {
    tempPhonemeState = Phoneme.E;
    phonemeNum = 2;
    }
    else if (z == 3)
    {
    tempPhonemeState = Phoneme.FV;
    phonemeNum = 3;
    }
    else if (z == 4)
    {
    tempPhonemeState = Phoneme.L;
    phonemeNum = 4;
    }
    else if (z == 5)
    {
    tempPhonemeState = Phoneme.MBP;
    phonemeNum = 5;
    }
    else if (z == 6)
    {
    tempPhonemeState = Phoneme.O;
    phonemeNum = 6;
    }
    else if (z == 8)
    {
    tempPhonemeState = Phoneme.U;
    phonemeNum = 8;
    }
    else if (z == 9)
    {
    tempPhonemeState = Phoneme.WQ;
    phonemeNum = 9;
    }
    if (phonemeNum ==7)
    {
    phoneme = 0;
    phonemeNum = 0;
    }
    phoneme++;
    phonemeNum++;
    k++;
    }
    lipSyncData.name = ModelLip.audioSource.clip.name;
    lipSyncData.clip = ModelLip.audioSource.clip;
    lipSyncData.length = ModelLip.audioSource.clip.length;
    lipSyncData.phonemeData = markers.ToArray();
    float EndTime= 0.01f;
    float StartTime;
    for (float i = 0.03f; i <= dialougelengthFloat; i += .06f)
    {
    EyesState tempEyesState = EyesState.CloseEyes;
    int l = RandomInt.Random(0, 2);

    if (l == 0)
    {
    tempEyesState = EyesState.CloseEyes;
    }
    else if (l == 1)
    {
    tempEyesState = EyesState.CloseEyes;
    }
    StartTime = EndTime+i+.08f;
    EndTime = StartTime+0.001f;
    EmotionMarker emotionMarker = new EmotionMarker(tempEyesState.ToString(), StartTime, EndTime, 0.025f,0f,false,false,false,false);
    Emotion.Add(emotionMarker);
    }
    EndTime = 0.01f;
    for (float i = 0.03f; i <= dialougelengthFloat; i += .06f)
    {
    EyesState tempEyesState = EyesState.LookingAround;
    int l = RandomInt.Random(0, 3);

    if (l == 0)
    {
    tempEyesState = EyesState.LookingAround;
    }
    if (l == 2)
    {
    tempEyesState = EyesState.EyebrowsUp;
    }
    else if (l == 1)
    {
    tempEyesState = EyesState.LookingAroundRight;
    }
    StartTime = EndTime + i + .10f;
    EndTime = StartTime + 0.01f;
    EmotionMarker emotionMarker = new EmotionMarker(tempEyesState.ToString(), StartTime, EndTime, 1f);
    Emotion.Add(emotionMarker);
    }
    for (float i = .01f; i <= dialougelengthFloat; i += .4f)
    {

    AnimState tempState = AnimState.Talking;
    int j = RandomInt.Random(0, 2);

    if (j == 0) {
    tempState = AnimState.TalkingTwo;
    }
    else if (j == 1)
    {
    tempState = AnimState.Talking;
    }
    // GestureMarker emotionMarker = new GestureMarker(tempState.ToString(), i);
    // GestureAnim.Add(emotionMarker);
    }
    lipSyncData.emotionData = Emotion.ToArray();
    lipSyncData.gestureData = GestureAnim.ToArray();
    lipSyncData.transcript = dialouge.text;
    ModelLip.Play(lipSyncData, 0f);

    }
     
  7. faizan143143

    faizan143143

    Joined:
    Oct 24, 2017
    Posts:
    4
    NullReferenceException: Object reference not set to an instance of an object
    RogoDigital.Lipsync.LipSync.SetEmotion (System.String emotion, System.Single blendTime) (at Assets/Rogo Digital/LipSync Pro/Components/LipSync.cs:611)
    EmotionSetToModel.GetlipSync (System.String Emotionstring) (at Assets/Game/Script/EmotionSetToModel.cs:69)
    EmotionSetToModel.TaskOnClick () (at Assets/Game/Script/EmotionSetToModel.cs:40)
    UnityEngine.Events.InvokableCall.Invoke () (at C:/buildslave/unity/build/Runtime/Export/UnityEvent.cs:166)
    UnityEngine.Events.UnityEvent.Invoke () (at C:/buildslave/unity/build/Runtime/Export/UnityEvent_0.cs:58)
    UnityEngine.UI.Button.Press () (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/Button.cs:66)
    UnityEngine.UI.Button.OnPointerClick (UnityEngine.EventSystems.PointerEventData eventData) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/UI/Core/Button.cs:108)
    UnityEngine.EventSystems.ExecuteEvents.Execute (UnityEngine.EventSystems.IPointerClickHandler handler, UnityEngine.EventSystems.BaseEventData eventData) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/ExecuteEvents.cs:50)
    UnityEngine.EventSystems.ExecuteEvents.Execute[T] (UnityEngine.GameObject target, UnityEngine.EventSystems.BaseEventData eventData, UnityEngine.EventSystems.ExecuteEvents+EventFunction`1[T1] functor) (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/ExecuteEvents.cs:261)
    UnityEngine.EventSystems.EventSystem:Update() (at C:/buildslave/unity/build/Extensions/guisystem/UnityEngine.UI/EventSystem/EventSystem.cs:377)
     
  8. EmagineVR

    EmagineVR

    Joined:
    Nov 1, 2018
    Posts:
    4
    Hey guys. I recently upgraded to 2020 from 2019 and I'm getting an error on the LipSync Pro component: "Phoneme does not exist in the chosen Phoneme set. You can change the Phoneme Sets from the Project Settings". Where abouts in the project settings is this setting?
     
  9. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    450
    Could you both make sure you're on the latest version from the Asset Store?

    @faizan143143 Can you give me any info about what you're doing to get that error? (Just basing off the post you've quoted there) If you're creating a LipSyncData object from a script make sure you set the "version" field to match the version of LipSync Pro you're using (probably 1.531) - LipSync uses this number to determine if the data needs updating from an old format.
     
  10. colpolstudios

    colpolstudios

    Joined:
    Nov 2, 2011
    Posts:
    150
    Hi all, thought id share a short video animation using lipsync pro with a standard Uma.



    It's my first one, like ever.

    comments and critic welcome
     
    bz_apps and Rtyper like this.
  11. forzabo

    forzabo

    Joined:
    Aug 28, 2015
    Posts:
    35
    Hi, FYI --I'm running into a null ref error when playing back emotions. On Unity 2019.3, MacOS 10.12, LSP 1.532.

    The null ref occurs at line 611 of LipSync.cs where it loops thru boneCurves. Adding a null check fixes the problem.
    Before:
    Code (CSharp):
    1. // Fix Quaternion rotations (Credit: Chris Lewis)
    2.                 foreach (TransformAnimationCurve curve in boneCurves)
    3.                 {
    4.                     curve.FixQuaternionContinuity();
    5.                 }
    After:
    Code (CSharp):
    1.  
    2. // Fix Quaternion rotations (Credit: Chris Lewis)
    3. //Hack fix for null ref
    4. if(boneCurves != null)
    5. {
    6.                 foreach (TransformAnimationCurve curve in boneCurves)
    7.                 {
    8.                     curve.FixQuaternionContinuity();
    9.                 }
    10. }
    11.  
     
  12. Vladislav-experience

    Vladislav-experience

    Joined:
    Dec 21, 2020
    Posts:
    2
    Hello everyone, I am making an automatic playback of a LipSync clip with audio recorded with a microphone. Everything is fine with the recording, no problems. But then I need to arrange phonemes using the AutoSync Window, and I can't find the functions I need and is it possible to do this at all. I would be very grateful if someone can provide some algorithm of actions or methods so that AutoSync can be done in code.

    Thank you very much!
     
  13. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    450
    Hello everyone - I've got a bit of news to announce. I recently got offered a new job that will take up much more of my time, and so I've been considering my options regarding what will happen with LipSync Pro and Cinetools going forward.

    Ideally, I'd like to be able to keep selling LipSync Pro, and publish the new Cinetools (v2) assets as planned - realistically though it looks like I just won't have the time to provide the required support in future, and I'll have a lot less free time to spend working on the Cinetools assets. Unfortunately, for this reason I've decided to stop selling the current version of LipSync Pro from the start of January 2021, when the New Year's sale ends. I will continue to provide support through email and discord when I can, until I've finished updating the documentation. I also plan to publish further patches as and when, to keep the final version stable and working.
    My long-term plan here is to release LipSync Pro under an open-source license (details TBD) so others can continue improving it if they wish, but I want to leave some time between deprecating the asset on the store and releasing it for free in fairness to anyone who has bought it recently.

    As for Cinetools (including LipSync Pro 2), I'm not cancelling it outright - I've put a lot of work into it so far - but it needs a fair bit more work to be usable. For now, I'm just going to put the project on ice and revisit it in future - there may be bits I can make available in some other way, we'll see.

    Regardless of what happens, I'd like to thank everyone who's bought and used LipSync over the last 5 and a half years. It's been amazing seeing all the varied projects you guys have made!

    Thanks for the heads up - I'll put it in a patch with a few other small fixes.

    I'm afraid AutoSync isn't available at runtime, as stated in the Asset Store description. You can access its functions from editor code though if that's any use - you can find the required methods in the RogoDigital.Lipsync.AutoSync class.
     
    julianr and bz_apps like this.
  14. Vladislav-experience

    Vladislav-experience

    Joined:
    Dec 21, 2020
    Posts:
    2
    Oh, i understand.
    Thanks a lot!
     
  15. Pourya-MDP

    Pourya-MDP

    Joined:
    May 18, 2017
    Posts:
    121
    Hey
    Somehow lipsyncpro component throw this error and nothing work,as you can see theres nothing in the inspector under the lipsync pro component
    Unity version 2019.4.10f1
    Lipsync pro latest version
    Untitled.png
     
  16. HeroYss

    HeroYss

    Joined:
    Jan 24, 2017
    Posts:
    3
    please add Mandarin (Chinese) recognition support, we are looking forward to it!
     
  17. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    450
    That's weird - can you tell me what happened before that occurred? I see you've got the Sprite Manager component attached, so I assume it worked at first until some point after choosing the blend system?

    I'm afraid there's no plans to add Mandarin support at this time - as explained above, I'm actually stopping active development on LipSync Pro for now.
     
  18. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    450
    And on that note - today's the day the LipSync Pro package is being deprecated from the Asset Store (the New Year's sale lasted longer than I expected it to). If you've just come to this thread after seeing it marked as deprecated, don't worry, this isn't the end of support yet!

    The full post is a little further up this page if you want more info, but the short version is I no longer have time to properly support it and continue development on v2 going forward due to a new job. I've taken it off sale as it doesn't seem fair to continue selling it without the expectation of long-term support, but I do plan to eventually release it under some open source license in the future. For now, I'll continue responding to questions when I can (most likely on weekends, so please be patient!). If you've only just bought it and aren't happy with the circumstances, I understand - my aim in not making it open source immediately was to give some value to brand-new customers, but if you haven't yet downloaded it and would prefer a refund, get in touch with your order number and I'll get that sorted with Unity.
     
    bz_apps, DragonShadow and MoMonay like this.
  19. MoMonay

    MoMonay

    Joined:
    May 9, 2018
    Posts:
    18
    Hey,

    So when I imported the newest package from package manager window in a new unity 2019.4.17f1 project. I then installed without problems until I had to download the montreal forced aligner package. After downloading it I got the same error as the other guys:

    Code (CSharp):
    1. "Assets/Rogo Digital/LipSync Pro/AutoSync/Editor/Modules/Montreal Forced Aligner/ASMontrealPhonemeDetectionModule.cs(44,24): error CS0115: 'ASMontrealPhonemeDetectionModule.ProcessWithTemplates(LipSyncData, AutoSync.ASProcessDelegate, PhonemeMarker, EmotionMarker)': no suitable method found to override"
    When I checked the code, the error seemed accurate; the
    ProcessWithTemplates virtual method is not declared in the AutosyncModule.cs.
    However, if I remove the ProcessWithTemplates implementation in ASMontrealDetectionModule.cs (it wasn't being used)
    it also gave me some errors, when I looked into it, it seems variables were using membernames that were already refactored in their corresponding classes. I changed them to the correct version: phonemeList.Count became phonemes.Length.

    Makes me think if the montreal forced aligner package maybe contains an outdated/wrong version of ASMontrealDetectionModule.cs?

    Either way, after changing those things in ASMontrealDetectionModule.cs everything works great!

    Thank you for your hard work and good luck with your new job! :D
     
  20. Rtyper

    Rtyper

    Joined:
    Aug 7, 2010
    Posts:
    450
    Hey, I'll take a look at this asap, but it looks more to me like Package Manager may be serving you an older version of LipSync Pro - can you confirm the version number you see in the top-right corner of the Clip Editor? I have seen other publishers talking about similar issues with the Package Manager & Asset Store packages, so it may be related.
     
  21. MoMonay

    MoMonay

    Joined:
    May 9, 2018
    Posts:
    18
    Ohhh wow you're right! Package manager is making me use 1.52 despite it saying it's 1.532.0. That's a super lame bug, considering their choice to force the use of package manager in unity 2020 and beyond :(

    I've downloaded it again in a new unity 2019 project through the asset store window and now I don't have any errors at all.

    So to avoid this unity bug for now, we just have to use unity 2019 or below because those still have the asset store window?
     
  22. TonyLi

    TonyLi

    Joined:
    Apr 10, 2012
    Posts:
    11,224
    Delete the old package from your computer's Asset Store cache folder:

    Windows: [user]\AppData\Roaming\Unity\Asset Store-5.x
    Mac: ~/Library/Unity/Asset Store-5.x

    Then you can download the latest version using the Package Manager window.
     
    MoMonay likes this.
  23. Swyfft

    Swyfft

    Joined:
    Mar 14, 2019
    Posts:
    49
    /sigh .. the pain of buying several assets at a big sale .. and JUST getting around to seeing what I want to run with them .. picked up Slate and was looking at this lip sync to go with it ..
    understand shifting prioroties .. especially these days ..
    will follow the progress here .. hope your new job is successful !!
     
  24. NotPlaying

    NotPlaying

    Joined:
    Feb 10, 2021
    Posts:
    1
    I just started a project and was looking for lipsync that I could try for to learn from.
    I saw that there was a "Lite" version to use for free, but I'm a bit late. Good luck on your new ventures.
     
  25. WizzleSWD

    WizzleSWD

    Joined:
    Nov 16, 2018
    Posts:
    7
    Dead Asset
     
    mrwhip likes this.
  26. AGregori

    AGregori

    Joined:
    Dec 11, 2014
    Posts:
    439
    OK what's going on? Rogo gave up on the asset again?...


    Edit: OK, I've just read the post from 9 Jan. RIP.
    To anyone who tried the Slate+LipsyncPro extension: does it work well with the final Lipsync version, or is it time to go for Salsa?
     
unityunity