Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

[RELEASED] Puppet Face - All-In-1 Facial Animation for Unity

Discussion in 'Assets and Asset Store' started by jamieniman, Dec 17, 2020.

  1. EddieChristian

    EddieChristian

    Joined:
    Oct 9, 2012
    Posts:
    725
    I bought this when it first came out and I finally have a project where I get to use it. But I just wanted to verify it was capable of doing what I need before I start. I just need an application that uses the webcam and tracks a person face. I realized that maybe this tool only did this in the editor and thought I would ask if it can do this in builds as well.
     
  2. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Is this in the demo scene? Could you try disabling and then reenabling the BlendShapeManager component?
     
  3. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Are you able to send me the BuyCookies.xml it created?
     
  4. BarnimPL

    BarnimPL

    Joined:
    Nov 22, 2020
    Posts:
    7
    Sure.
     

    Attached Files:

  5. thatnzguy

    thatnzguy

    Joined:
    Jun 8, 2015
    Posts:
    17
    I had a few difficulties with the blendshape editor.

    Some notes or ideas:
    • The tool values keep resetting. Values are the same per tool, it would be nice if they were separate.
    • If the radius is ever 0 or -0 then the face usually breaks right away (typing in 0.02 requires typing in 0.0 first..)
    • Mirror is misaligned in RP_Eric model
    • Pressing Ctrl-Z for undo also triggers the use of the automatic modifier Ctrl, which is either Reset or Pull modes
    • I would like more control on the brush, the fall off etc.
    • The vert display mode is helpful but appears really bright/large, a different style would be helpful
    • I'm accidentally moving the teeth, and it's really hard to target them to erase the changes to them. I suspect the skinned mesh rendedrer's physics collision is used for the brush targeting, making it hard to get inside the mouth
    Here's it in action:
     
    Last edited: Apr 27, 2021
  6. pab

    pab

    Joined:
    Jul 2, 2013
    Posts:
    6
  7. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Thanks for pointing these out, I'll take a look.

    Did you know if you hold down the "B" button you can change the radius by moving the mouse left and right. I find it much more intuitive than setting it in the inspector.

    Is he at 0 in the world x position?

    ah - let me take a look.

    yes this would be a good future feature.

    ill have a think,
    Did you know you can choose a submesh - this way you can exclude the teeth when making the blendshape :)
     
  8. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Ok this might be a Unity version issue - thanks for giving me the Unity version so I can look into it
     
  9. pab

    pab

    Joined:
    Jul 2, 2013
    Posts:
    6
    Another thing to check. Use a Mecanim Humanoid character with an Animator component playing an Idle animation. I think the lipsync clip will not play whilst the Animator is enabled. Disable the component and the lipsync track plays albeit with the fault already reported.
     
  10. Unique_User_Name

    Unique_User_Name

    Joined:
    Apr 29, 2021
    Posts:
    1
    @jamieniman below errors that I have on Unity version 2021.1.1f1 and 2019.4.20f1



     
  11. unity_ErOys4agLfdvKA

    unity_ErOys4agLfdvKA

    Joined:
    May 22, 2020
    Posts:
    1
    @BarnimPL @jamieniman all reported errors related to incorrect format are due to a problem with parsing float values from XML files. You are using dot (".") to separate values (total value from a fraction) but in some regions a comma (",") is used instead. You just need a better method for parsing float values.
     
    BarnimPL likes this.
  12. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Thank you! That'll help me figure a solution :)
     
    BarnimPL likes this.
  13. Clawdelle

    Clawdelle

    Joined:
    Feb 26, 2017
    Posts:
    3
    Hi, no markers are showing under wave.
    Unity 2020.3.6f1


    <?xml version="1.0" encoding="utf-8"?>
    <rhubarbResult>
    <metadata>
    <soundFile>C:\Users\Delle\Spiel-Neue-Projekte\puppyFace2\Assets/PuppetFace/Demo/Audio/delle1.wav</soundFile>
    <duration>5.30</duration>
    </metadata>
    <mouthCues>
    <mouthCue start="0.00" end="0.74">X</mouthCue>
    <mouthCue start="0.74" end="0.82">A</mouthCue>
    <mouthCue start="0.82" end="0.90">C</mouthCue>
    <mouthCue start="0.90" end="0.97">E</mouthCue>
    <mouthCue start="0.97" end="1.11">F</mouthCue>
    <mouthCue start="1.11" end="1.27">A</mouthCue>
    <mouthCue start="1.27" end="1.55">B</mouthCue>
    <mouthCue start="1.55" end="1.69">D</mouthCue>
    <mouthCue start="1.69" end="2.38">B</mouthCue>
    <mouthCue start="2.38" end="2.65">F</mouthCue>
    <mouthCue start="2.65" end="2.69">C</mouthCue>
    <mouthCue start="2.69" end="2.77">A</mouthCue>
    <mouthCue start="2.77" end="2.94">F</mouthCue>
    <mouthCue start="2.94" end="3.15">G</mouthCue>
    <mouthCue start="3.15" end="3.57">F</mouthCue>
    <mouthCue start="3.57" end="4.32">X</mouthCue>
    <mouthCue start="4.32" end="4.37">B</mouthCue>
    <mouthCue start="4.37" end="4.63">F</mouthCue>
    <mouthCue start="4.63" end="4.70">G</mouthCue>
    <mouthCue start="4.70" end="5.30">X</mouthCue>
    </mouthCues>
    </rhubarbResult>

    upload_2021-4-29_20-59-36.png
     
  14. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Could you try this for me - open the xml in an editor and replace all . with ,
    Then tell me if it works?

    Next thing could you reset that file, and then try adding the following script to the project - I want to see if this works:
    Code (CSharp):
    1. using System.Globalization;
    2. using System.Threading;
    3. using UnityEngine;
    4. #if UNITY_EDITOR
    5. using UnityEditor;
    6. #endif
    7. #if UNITY_EDITOR
    8. [InitializeOnLoad]
    9. public static class FixCultureEditor
    10. {
    11.     static FixCultureEditor()
    12.     {
    13.         Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;
    14.     }
    15. }
    16. #endif
    17. public static class FixCultureRuntime
    18. {
    19.     [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
    20.     static void FixCulture()
    21.     {
    22.         Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;
    23.     }
    24. }
     
    Last edited: Apr 29, 2021
  15. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    I believe I have sorted this, if you email me at Puppet3Dunity@gmail.com I can send you the fix (the xml reader needed to be culture invariant)
     
  16. Clawdelle

    Clawdelle

    Joined:
    Feb 26, 2017
    Posts:
    3


    Cool, comma (,) is working, so it's a German notation thing. r/Detlef
     
  17. Clawdelle

    Clawdelle

    Joined:
    Feb 26, 2017
    Posts:
    3

    Hi Jamieniman, I placed the script in rhubarb folder and it seems to work. Old Man shows the markers without intervention.

    thx/Detlef
     
  18. Miltan_

    Miltan_

    Joined:
    Sep 7, 2013
    Posts:
    7
    Hello, first of all, great tool,
    I have wave audio as byte[]
    Can you provide an C# Code example of how your tool handles this kind of approach on runtime.
    Can your tool perform lipsync animation if i download wave file on runtime
     
  19. paulacs

    paulacs

    Joined:
    Mar 4, 2021
    Posts:
    4
    Hello,
    I've got the following problem, when I try to use the animation clip capture with the PerformanceCapture component in a animator controller the application doesn't run out the rutine and, I've got the following warning:

    "Binding warning: Some generic clip(s) animate transforms that are already bound by humanoid avatar. These transforms can only be changed by Humanoid clips."

    I'm using and Humanoid rig so I don't understand why it doesn't work. Maybe the answer it is very obvious but i'm new in this field.
    Thank you so much!!
     
  20. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Currently there are a few hoops to jump through to get humanoid and performance capture to work together:
    1. import the fbx exporter package
    2. right click on your char (with its anims on an animator) and export as an fbx (with model and anims)
    3. Set this fbx to a humanoid (with the same avatar)
    4. Duplicate all the animations and add them to your animator.
    5. Copy all the blend shape keyframes across from the pre-exported anims to the exported ones.
    It's not ideal. I want to make humanoid specific performance capture - its on my list.
     
  21. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    At the moment the audio to lipsync conversion is editor time only.
    If you take a look at the lipsync script at line around 490, the conversion uses an exe, and is currently wrapped in If UNITY_EDITOR defines.
    I haven't looked into this much though, so perhaps its easy enough to make it work. Its on the the list of todos.
     
  22. Davon92

    Davon92

    Joined:
    Oct 24, 2014
    Posts:
    12
    Could this handle doing simple transitions as well between mouth shapes with no blending in between? im looking for something with this level of flexibility but that can also do static quick changes.
     
    Davon_Allen_AofL likes this.
  23. paulacs

    paulacs

    Joined:
    Mar 4, 2021
    Posts:
    4
    Thank you!!
     
  24. appleboxman

    appleboxman

    Joined:
    Apr 17, 2020
    Posts:
    5
    Hi,
    I am new to Puppet Face. Using v1.4 today and I got some error messages:

    Assembly 'Library/ScriptAssemblies/Unity.PlasticSCM.Editor.dll' will not be loaded due to errors:
    Reference has errors 'unityplastic'.

    Assembly 'Packages/com.unity.collab-proxy/Lib/Editor/PlasticSCM/unityplastic.dll' will not be loaded due to errors:
    unityplastic references strong named System.Windows.Forms Assembly references: 4.0.0.0 Found in project: 2.0.0.0.
    Assembly Version Validation can be disabled in Player Settings "Assembly Version Validation"
     
  25. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Perhaps its an assembly definition issue.
    Could you try the following to see if it fixes it:
    1. Go to the player settings and make sure "Allow 'unsafe' Code is ticked on. (Allows the use of pointers)
    upload_2021-6-9_9-10-5.png
    2. Now search for the file "OpenCvSharp" and delete just this single file.

    Let me know if that fixes it?
     
  26. paulacs

    paulacs

    Joined:
    Mar 4, 2021
    Posts:
    4
    Hi! I've got a problem in the latest version, when I try to use the Blendshape manager.
    In the blenddshape manager when I try to create new blendshapes I don't get the circle that allows me to edit, it follows the normal mouse cursor, and even if I change pull/push/eraser, nothing, it doesn't change and it doesn't modify the mesh.
     
  27. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Make sure you have the gizmos button on (top right of the scene window) :)
     
  28. Xyth

    Xyth

    Joined:
    Aug 10, 2017
    Posts:
    3
    I am interested in this application to create facial blendshapes on 3D Unity based characters that are rigged but lack facial rigging. Is facial rigging required to create facial blendshapes? Also, I noticed this tool adds several new components in the editor. Do any of those components need to stay on the character during export to an asset file (.Unity3d for example). I would use this for adding characters to an existing game, and that game just uses only built in Unity components. Thanks!
     
  29. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Its not required, you can make the blendshapes on any mesh or skinned mesh.
    The lip sync component is required to drive the lip sync animation.
     
  30. Xyth

    Xyth

    Joined:
    Aug 10, 2017
    Posts:
    3
    Perfect. Thanks for the info!
     
  31. delany

    delany

    Joined:
    Nov 20, 2013
    Posts:
    10
    Hi. I've set up a lip sync with one clip - and it works very nicely ... I can edit and playback. Thanks!

    However, when I try and add a second clip (i.e I add a second Lip Sync File (in Element 1, previous is Element 0) and a second Audio Clip (in Element 1), I get a stream of Errors either when playing it in the scene or when trying to do things with it in the Editor. I can see the new audio file in the editor and I can see the associated Lip Syncs marked up - but when I play the audio the errors appear and no synching takes place. It's not the specific files - if I swap the two elements around, the new first one will work and the new second one will return the errors - so it's something about having that second clip.

    The errors all look like:

    ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
    Parameter name: index
    System.ThrowHelper.ThrowArgumentOutOfRangeException (System.ExceptionArgument argument, System.ExceptionResource resource) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
    System.ThrowHelper.ThrowArgumentOutOfRangeException () (at <eae584ce26bc40229c1b1aa476bfa589>:0)
    PuppetFace.LipSync.SetPhoneme (System.Int32 index, System.Single timeVal) (at Assets/PuppetFace/Scripts/LipSync.cs:468)
    PuppetFace.LipSync.EvaluateLipSync (System.Int32 index) (at Assets/PuppetFace/Scripts/LipSync.cs:206)
    PuppetFace.LipSync.LateUpdate () (at Assets/PuppetFace/Scripts/LipSync.cs:154)

    I'm using Unity 2020.3.1f1 on a Mac.

    Thanks for any help you can give!
     
  32. Davon_Allen_AofL

    Davon_Allen_AofL

    Joined:
    Jul 17, 2019
    Posts:
    6
    This may have gotten lost in the many questions but i would still like to know. Is this tool capable of doing quick changes ala south park or pose to pose animation. I'm looking into audio generated lipsync without actually blending/interpolating from one shape to another. similar to stop motion even.
     
  33. delany

    delany

    Joined:
    Nov 20, 2013
    Posts:
    10
    Just tried my same exercise as above using Unity 2019.4.23f1 and it worked fine - so it looks like an issue of Puppetface with Unity 2020.3.1f1.
     
  34. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    could you try the following to see if it fixes it -
    open LipSyncBehaviour.cs and comment out line 27, and replace line 35 with:
    Code (CSharp):
    1.  lipSync.SetPhoneme(input.LipSyncIndex, timeVal);
    2.  
     
  35. delany

    delany

    Joined:
    Nov 20, 2013
    Posts:
    10
    Yes! That seems to have worked. Thank you!
     
  36. photonic

    photonic

    Joined:
    Dec 29, 2012
    Posts:
    46
    Sorry @jamieniman,
    if this has been asked before, but the Blendshape Manager seems to have major issues with high poly models.
    After I hit the "create new blend shape button" the model gets completely distorted.
    I've tried all three TYPE-options ... any hint what I could do?
    Thanks
     
  37. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    For interest how many polys is it? Are you able to filter the sub-mesh to what you need to sculpt?
    What kind of distortion happens?
     
  38. photonic

    photonic

    Joined:
    Dec 29, 2012
    Posts:
    46
    I've tried with different single meshes and everything above roughly 10k up to 60k had single vertices or groups of vertices pushed way out along different axis before I even edited the mesh.
    How do I filter the sub mesh? Did not find anything about that in the manual?
     
  39. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    If you've got multiple materials on your mesh you can filter these:
    upload_2021-6-25_11-49-56.png

    I haven't experienced the error you're reporting - are you able to email me (puppet3dunity@gmail.com) an example that breaks, and let me know which version of Unity you are using?
     
    Last edited: Jun 25, 2021
  40. photonic

    photonic

    Joined:
    Dec 29, 2012
    Posts:
    46
    Thanks, the problem occurs even with one mesh and one material (so I can't select anything else in the Sub Mesh dropdown).
    Unity Version is 2019.4.18f1 (and 2019.4.21f1)

    Mail with example should be in your inbox.
     
  41. photonic

    photonic

    Joined:
    Dec 29, 2012
    Posts:
    46
    Thank you @jamieniman for your fast support!

    Just to clear up matters for anyone interested here on the forum:

    Blend Shape Manager can currently handle objects up to 120k tris (which is quite reasonable).
    - be aware, numbers in my above post were vertices, thus in some cases having significantly more triangles than 120k ;-)

    I just use meshes with higher resolution for up-close hero shots for cinematics and thus went over that limit.

    So, for 'normal' game assets all is well and I've tested it with several different objects.
    Cheers!
     
    Last edited: Jun 25, 2021
    jamieniman likes this.
  42. mcroswell

    mcroswell

    Joined:
    Jan 6, 2010
    Posts:
    73
    You product looks interesting and fun!

    Anyway, I've got a bunch of questions relating to your product and particularly in terms of requirements.

    In the description (of Puppet Face at the Asset Store) the last line says:

    (Requires Windows for Lip Sync Converter)

    What is Lip Sync Converter? Who makes it? Where do I get it?

    What does this line refer to, the whole product or an aspect?

    If just an aspect, does this product work with Mac or additional benefit with Windows?

    Thank you,
    Mike
     
    Last edited: Jun 28, 2021
  43. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Hi Mike, good question -
    There is a program called Rhubarb that Puppet Face runs that does the initial conversion of the wav files to lipsync data. There is a Mac version on their site which you run manually in a terminal. What the "Require's Windows" was saying is with Windows you don't need to do this manually - Puppet Face has a button to do it in Unity. (Which just runs the terminal command under the hood).
    Apart from this inconvenience everything else works on Mac.
     
    mcroswell likes this.
  44. appleboxman

    appleboxman

    Joined:
    Apr 17, 2020
    Posts:
    5
    Not fix and keep spawn the same error message.
    Assembly 'Library/ScriptAssemblies/Unity.PlasticSCM.Editor.dll' will not be loaded due to errors:
    Reference has errors 'unityplastic'.
    Assembly 'Packages/com.unity.collab-proxy/Lib/Editor/PlasticSCM/unityplastic.dll' will not be loaded due to errors:
    unityplastic references strong named System.Windows.Forms Assembly references: 4.0.0.0 Found in project: 2.0.0.0.
    Assembly Version Validation can be disabled in Player Settings "Assembly Version Validation"
     
  45. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Are you sure this is a Puppet Face caused error? If you delete Puppet Face from the project does it go away?
     
  46. appleboxman

    appleboxman

    Joined:
    Apr 17, 2020
    Posts:
    5
    I have opened a new project (build-in 3D) and only imported Puppet Face. Then I run your demo, the error msg pop in the console. As You said, after I deleted the Puppet face folder, no error msg anymore.

    BTW, I am using the latest unity version 2020.3.12f1.
     
  47. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    I'm not getting the error.
    It's something to do with plastic (a Unity package - not sure what it does)
    The error suggest you disable Assembly Validation in the player settings:

    upload_2021-7-2_16-32-3.png

    Does that fix it?
     
  48. LTHendrix

    LTHendrix

    Joined:
    Jul 29, 2013
    Posts:
    17
    When I bring in audio and click reload, it produces a lot of phonemes on the timeline. Is there a way to clear them all at once instead of deleting them one by one?

    Thank you.
     
  49. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Open up the XML file it generated and delete all the lines that look like this:
    Code (CSharp):
    1.     <mouthCue start="0.00" end="0.04">X</mouthCue>
    2.  
    then click reload an it should remove them all :)
     
  50. LTHendrix

    LTHendrix

    Joined:
    Jul 29, 2013
    Posts:
    17
    Thanks for the quick reply.

    Now I'm having trouble getting the Lip Sync Track to work in Timeline. I placed it but when I scrub the TL it won't play even when I change the LipSync Index.
    pf_issue01.JPG
    What am i missing?