Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

[RELEASED] Puppet Face - All-In-1 Facial Animation for Unity

Discussion in 'Assets and Asset Store' started by jamieniman, Dec 17, 2020.

  1. unity_a8B9wWbnDhO-Jw

    unity_a8B9wWbnDhO-Jw

    Joined:
    May 13, 2019
    Posts:
    2
    Thank you for your reply. So, if I decide not to use lip sync, the plugin will work in Linx, right?
    I'm mainly interested in face animation through face tracking from video: will it work in Linux?
     
  2. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    It hasn't been tested on linux - but I see know reason why it wouldn't work.
    (lip sync should still work, just the conversion needs to be done manually in a console)
     
  3. YTGameDevDave

    YTGameDevDave

    Joined:
    Jan 3, 2015
    Posts:
    12
    Hey Jamie, I've done the .exe thing with rhubarb through the lipsync component but it doesn't update after installing with the .exe ... It just keeps saying "Download LipSync Converter"

    upload_2022-3-6_17-48-50.png

    I have Unity 2020.3.24f1 LTS

    Using the lipsync demo scene in the package

    Edit: Solved it, just have to put the .exe in:

    /Tools/LipSync/rhubarb/rhubarb.exe

    Run it and then it works.
     
    Last edited: Mar 6, 2022
  4. YTGameDevDave

    YTGameDevDave

    Joined:
    Jan 3, 2015
    Posts:
    12
    Uhm Jamie, another problem, your scenes in playmode don't show any blendshape deformations... It's a fresh scene, fresh import... It's just the jawbone going up and down...

    Fixed: It definitely needs the rhubarb update but it still didn't work. Just had to launch the editor and click reload in the lipsync timeline and now it works.
     
    Last edited: Mar 6, 2022
  5. YTGameDevDave

    YTGameDevDave

    Joined:
    Jan 3, 2015
    Posts:
    12
    Another problem! Selecting the phonemes doesn't change the face pose.

    upload_2022-3-6_19-3-32.png

    And something janky now, selecting a different phoneme should get your out of the edit mode, but whilst it still does that, it changes other phonemes to completely different sounds. Now 3 phonemes changed to the "E" sound.
     
  6. YTGameDevDave

    YTGameDevDave

    Joined:
    Jan 3, 2015
    Posts:
    12
    upload_2022-3-6_19-6-1.png
    For no reason. And it's not undoable.

    For some reason it just started working by just clicking in and out of a few phonemes. This is very janky behavior...
     
  7. YTGameDevDave

    YTGameDevDave

    Joined:
    Jan 3, 2015
    Posts:
    12
    changing the list doesn't change the pose in edit mode:
    upload_2022-3-6_19-9-4.png

    and thats not even the O or Smile pose

    Okay so the lipsync timeline and phonemes do work properly if you just take great care and learn what breaks it so that it doesn't happen.

    Another issue is that the mesh collider somehow does not align with the model sometimes.
     
    Last edited: Mar 6, 2022
  8. jawasjnsdjn

    jawasjnsdjn

    Joined:
    Mar 7, 2013
    Posts:
    34
    proper forum etiquette would be to just edit your previous post as no one is responding to you and anyone who has a watch on this thread has gotten 5 emails already.
     
  9. YTGameDevDave

    YTGameDevDave

    Joined:
    Jan 3, 2015
    Posts:
    12
    I can't add anymore images to a comment after posting it, sorry

    And sorry for the spam but I don't see these questions answered anywhere else.
     
    Last edited: Mar 6, 2022
  10. YTGameDevDave

    YTGameDevDave

    Joined:
    Jan 3, 2015
    Posts:
    12
    @jamieniman How do we save a lipsync animation to an anim clip? Without using performance capture. I've tried it with "Capture" but it doesn't store all the blendshape animation. Only from what's detected on the webcam. But not the LipSync. How do we save that?

    Okay, so I've tried opening a new scene and dragging the cookiescout girl into it. Then I selected the head mesh containing the blendshapes and added the blendshape manager and lipsync components. I added the text file and audio file. Put the jawbone into the face bone array. Hit play, hit play in the inspector... Audio is playing but as for blendshape animations... Nothing... What am I doing wrong?
     
    Last edited: Mar 6, 2022
  11. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Apologies for not replying sooner.
    Are you able to give me steps to reproduce this error you are getting? Are you editing one phoneme and then clicking on another one (before applying the previous one)? This will assign the previous blendshape to the new Phoneme - which might explain why you are getting the 'O' in multiple places
     
  12. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    The lipsync animation is designed to be played using the LipSync component (Either by playing it in code or using the LipSync track on a Timeline). The reason I havent made a feature to bake this to a clip as this would be destructive and you would easily be able to use the lipsync editor to edit the animation.

    You need to assign all the blendshapes and set the jaw poses in the Lip Sync Editor
     
  13. SquareEyesTaco

    SquareEyesTaco

    Joined:
    Mar 25, 2013
    Posts:
    27
    Hello! thank you for the great asset - whenever i hit the create blend shape button it creates another head slightly to the right of where the head should be to edit the blend shape. Any idea why its not just creating the blend shape where the head is?
     
  14. SquareEyesTaco

    SquareEyesTaco

    Joined:
    Mar 25, 2013
    Posts:
    27
    also for some reason on another model when i try to sculpt a blend shape i get 2 circles like this
    upload_2022-3-23_1-7-1.png

    Why is that happening?
     
  15. SquareEyesTaco

    SquareEyesTaco

    Joined:
    Mar 25, 2013
    Posts:
    27
    Sorry i meant that the circles were off centered. the right one is too far from the left one
     
  16. FellowPlayer123

    FellowPlayer123

    Joined:
    Dec 23, 2016
    Posts:
    106
    I have BlendShapes for face skin.


    But I also have BlendShapes for other parts of the character mesh: eyebrows, bottom teeth, and so on.


    Currently, when the characters talk the mouth is moving while the bottom teeth remain unmoved.


    Question: how to connect those Blend Shapes?
     
  17. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Most likely the bindPose isnt centered. (Will also explain the mirroring issue you are getting - it mirrors in world space X)
    Try out these different settings to see if any of them work better for you:
    upload_2022-3-25_0-31-28.png
     
  18. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Give them all their own LipSync component, and assign the face poses. Then you can add these to the LipSyncs property of the main LipSync component. This lipsync component will then drive the other ones.
     
    FellowPlayer123 likes this.
  19. thanhle

    thanhle

    Joined:
    May 2, 2013
    Posts:
    155
    i open with unity 2021.2.14
    and i try to open BrownieGirl_PerformanceCapture.unity .... i got this error

    DllNotFoundException: OpenCvSharpExtern assembly:<unknown assembly> type:<unknown type> member:(null)
    OpenCvSharp.NativeMethods.TryPInvoke () (at Assets/PuppetFace/Tools/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:115)
    Rethrow as OpenCvSharpException: OpenCvSharpExtern assembly:<unknown assembly> type:<unknown type> member:(null)
    *** An exception has occurred because of P/Invoke. ***
    Please check the following:
    (1) OpenCV's DLL files exist in the same directory as the executable file.
    (2) Visual C++ Redistributable Package has been installed.
    (3) The target platform(x86/x64) of OpenCV's DLL files and OpenCvSharp is the same as your project's.
    System.DllNotFoundException: OpenCvSharpExtern assembly:<unknown assembly> type:<unknown type> member:(null)
    OpenCvSharp.NativeMethods.TryPInvoke () (at Assets/PuppetFace/Tools/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:115)
    OpenCvSharp.NativeMethods.TryPInvoke () (at Assets/PuppetFace/Tools/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:124)
    OpenCvSharp.NativeMethods..cctor () (at Assets/PuppetFace/Tools/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:59)
    Rethrow as TypeInitializationException: The type initializer for 'OpenCvSharp.NativeMethods' threw an exception.
    OpenCvSharp.FileStorage..ctor (System.String source, OpenCvSharp.FileStorage+Mode flags, System.String encoding) (at
     
  20. AGregori

    AGregori

    Joined:
    Dec 11, 2014
    Posts:
    511
    @jamieniman Please add text-to-lipsync ability to Puppet Face. Others have pointed out previously how many of our games simply don't always use spoken audio for dialogues or cutscenes.

    Text-to-lipsync would be the last hurdle to make this asset better than Salsa in most respects.
     
  21. yuda2010

    yuda2010

    Joined:
    May 17, 2017
    Posts:
    1
    Can this plugin support export blendshap to unity anim?Animator can easy to adj
     
  22. VikiSystems

    VikiSystems

    Joined:
    Mar 13, 2019
    Posts:
    3
    does it support 2d characters ?
     
  23. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Thank you for reporting this to me - I'll have to take a look.
     
  24. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Agreed, Its in my trello page
     
  25. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Apologies, not 100% sure what you're asking?
     
  26. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Not officially. There are little things you can do to hookup 2d characters to the lip sync.
    As an example I hooked up this 2D robot face to the lip sync for a game I worked on:
    upload_2022-5-2_12-58-58.gif
     
  27. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    So I tried it in 2021.3.1 and its working - have you tried it in a clean new project?
     
  28. thanhle

    thanhle

    Joined:
    May 2, 2013
    Posts:
    155
    I try version 2021.3.3f1
    still Error
    I use macbook M1( not window )
    upload_2022-5-22_3-22-45.png
     
  29. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Are you able to send me the rest of this error:
    upload_2022-5-23_9-4-35.png

    For interest does it work for you in an earlier Unity version? Also can you check its broken in an empty new clean project - just to rule anything else out.

    Kind Regards

    Jamie
     
  30. thanhle

    thanhle

    Joined:
    May 2, 2013
    Posts:
    155
    Ảnh chụp Màn hình 2022-05-26 lúc 3.25.01 SA.png
    this is empty new and clean project !
     
  31. aloneandsuffering101

    aloneandsuffering101

    Joined:
    Apr 21, 2018
    Posts:
    15
    is Daz3D support still coming along eventually?
     
  32. IconArdventure

    IconArdventure

    Joined:
    Dec 28, 2021
    Posts:
    1
    Hi,
    Sorry I am quite a noob to Unity. I just loaded the performance capture demo scene and cannot get my (virtual) webcam (or audio?) to work:
    upload_2022-5-29_0-51-11.png

    Here is how it is configured:
    upload_2022-5-29_0-52-53.png

    I see it's the issue is with this line of code:

    Code (CSharp):
    1. audioSource.clip = Microphone.Start(Microphone.devices[MicrophoneDeviceIndex], true, 10, Mathf.Min(MaxFrequency, 44100));
    Other mic's are disabled in windows system settings.

    Could you please help?

    Thanks.
    Arda
     

    Attached Files:

    Last edited: May 28, 2022
  33. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    What operating system are you using? If you search for OpenCvSharpExtern and find the one that is on your operating system - see if it is using the correct platform:
    upload_2022-6-7_9-35-13.png
     
  34. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    It already should work with Daz3D characters - are you referring to anything that you found not to work?
     
  35. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Very strange - could you add this line before that one so we can see if its this that is erroring:
    Code (CSharp):
    1. Debug.Log(Mircophone.devices[MicrophoneDeviceIndex];
     
  36. m1k311

    m1k311

    Joined:
    Apr 29, 2020
    Posts:
    1
    I couldn't find this on the forum if it's in here but do you recommend a specific web cam for Performance Capture and what specs does a web cam need to be used with Performance Capture?
     
    Last edited: Jul 2, 2022
  37. mrstruijk

    mrstruijk

    Joined:
    Jan 24, 2018
    Posts:
    49
    Hi, is there any news on wwise support?
     
  38. FellowPlayer123

    FellowPlayer123

    Joined:
    Dec 23, 2016
    Posts:
    106
    I have PuppetFace and it's great. Thank you for this wonderful tool.
    Question - If I already have tons of facial animations made in Blender can I export them to Unity or to PuppetFace and play them?
     
  39. keithsoulasa

    keithsoulasa

    Joined:
    Feb 15, 2012
    Posts:
    2,126
    Does the webcam to face animation work in runtime builds ?

    Would it be possible to get this running on an IPad ?
     
  40. kai_dev

    kai_dev

    Joined:
    Jun 29, 2015
    Posts:
    1
    Does it support IOS & Android?
     
  41. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    1,415
    Would you mind explaining how you did that? -- I'd love to have official support for this.
     
  42. zanarevshatyan

    zanarevshatyan

    Joined:
    Jul 15, 2021
    Posts:
    1
    Hi there, can somebody help me with this package? I'm getting extra bad mocap quality using this, I've tried 2 webcams, I've tried with light or without, I've tried even painting my mouth edge in black, I even changed the blendshapes looking alike with the blendshapes in puppet oficial video tutorial, and I get very bad mocap quality didn't recognizing my words from my mouth, this stupid doll is just chaotically opening her bitchy mouth and smiling like a retard.
    I also can't even lip sync the character, when I'm chossing the wav file, also I've choosed several files I get empty unity window



    What is happening with your package guys?
     
  43. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Are you doing the calibration? (Where you record all the different states based on your own face.
    You can see it being done here:

    Do you get any errors? Are you assigning the lipsync files at the same index as the audio file?
     
  44. AGregori

    AGregori

    Joined:
    Dec 11, 2014
    Posts:
    511
    So are any updates happening after almost 1 year of no updates at all?
    Puppet2D is even more problematic, with no updates for nearly 2 years.....

    You really need to implement a basic text-to-speech functionality, besides bugfixes, it's a staple these days. So many story-rich adventure games out there with no voice, only text used.
     
  45. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Yeah its been a while. In terms of bug fixes - I do these when I receive any bug reports from users, then role these into updates. I haven't really had any bug reports as of late - if you know of any bugs you are aware of please email me a description of the issue and I'll look into it. In terms of features - I have some ideas of what I would like to do next. Im finishing off another asset at the moment, and then Ill hopefully jump back onto Puppet Face. (I'm still supporting Puppet2D, but have no real plans to add new features).
    Yeah - I can see how useful this would be. I need to way it up with other things I want to improve on.
     
    AGregori likes this.
  46. alsomitra

    alsomitra

    Joined:
    Aug 31, 2017
    Posts:
    32
    How are your plans for the performance capture for Android coming along?
    When will it be released?
     
    awesomedata likes this.
  47. dprat0821

    dprat0821

    Joined:
    Dec 14, 2016
    Posts:
    2
    Hi,
    I'm using Puppet Face 1.5.1 on M1 Mac. When running the demo scene at /Demo/BrownieGirl/BrownieGirl_PerformanceCapture, i get the following error:


    [code=CSharp]DllNotFoundException: OpenCvSharpExtern assembly:<unknown assembly> type:<unknown type> member:(null)
    OpenCvSharp.NativeMethods.TryPInvoke () (at Assets/PuppetFace/Tools/OpenCV+Unity/Assets/Scripts/OpenCvSharp/PInvoke/NativeMethods.cs:115)
    Rethrow as OpenCvSharpException: OpenCvSharpExtern assembly:<unknown assembly> type:<unknown type> member:(null)
    *** An exception has occurred because of P/Invoke. ***
    Please check the following:
    (1) OpenCV's DLL files exist in the same directory as the executable file.
    (2) Visual C++ Redistributable Package has been installed.
    (3) The target platform(x86/x64) of OpenCV's DLL files and OpenCvSharp is the same as your project's.[/code]


    Do I need to add extra dependency on? I didn't find it in the doc.
    My environment:
    Mac: OS13.0.1, M1 Max
    Unity: 2021.3.14f1
    PuppetFace: 1.5.1

    Appriciated!
     

    Attached Files:

  48. dprat0821

    dprat0821

    Joined:
    Dec 14, 2016
    Posts:
    2
    Plus, the same setup works on my Windows machine. However, I noticed that the avatar's eyes are blinking rapidly and endlessly. How can it get fixed?
     
  49. snowcult

    snowcult

    Joined:
    Feb 6, 2014
    Posts:
    295
    Is the webcam capture something that can work in a build or is this editor only?
     
    Last edited: Jan 11, 2023
  50. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    It works in build - but recording only works editor time.
     
    snowcult likes this.