Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

[RELEASED] Puppet Face - All-In-1 Facial Animation for Unity

Discussion in 'Assets and Asset Store' started by jamieniman, Dec 17, 2020.

  1. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Can you confirm for me your using Puppet Face v1.4.1 ?

    Also does your animation file have any keyframes on the face bones/ blendshapes?
     
  2. tonnynevel

    tonnynevel

    Joined:
    Oct 19, 2018
    Posts:
    5
  3. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
  4. LTHendrix

    LTHendrix

    Joined:
    Jul 29, 2013
    Posts:
    17
    NVM Got it to work. Thanks
     
  5. delany

    delany

    Joined:
    Nov 20, 2013
    Posts:
    10
    Hi. On 2020.3.1f1 on a Mac, when I edit the placement, range or assigned mouth shape in the Lip Sync Editor, it works while I play the clip in the Lip Sync Editor but resets back to what I had before when I e.g. play the Scene. Is there some way to save these changes - or does that not work on a Mac because Rhubarb isn't integrated? (Note, Editing the Bone Poses themselves works fine).

    The documentation mentions save and reload buttons - but I don't see those. Just the 'Convert Audio Files' button at the top right of the pane.

    Thanks!
     
    Last edited: Jul 20, 2021
  6. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    You should see a save button. If not expand the window really wide (I think there is a dual monitor bug on mac where this centers to the wrong width)
    upload_2021-7-20_14-28-13.png
     
  7. delany

    delany

    Joined:
    Nov 20, 2013
    Posts:
    10
    Thanks - but unfortunately, I don't see that save button. Just the 'Convert Audio Files' button always at the very far right side of the pane, no matter how wide I make the pane (up to the size of the screen). I'm using only one screen on a laptop.
     
  8. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    can you try this:
    open PuppetFaceEditor.cs and comment out lines 153 and 185
    Code (CSharp):
    1. GUILayout.FlexibleSpace();
    change to
    Code (CSharp):
    1. //GUILayout.FlexibleSpace();
     
  9. delany

    delany

    Joined:
    Nov 20, 2013
    Posts:
    10
    That seems to make half the next (maybe reload) button visible at all widths. Here's a screenshot at maximum width after making those changes:

    Screen Shot 2021-07-20 at 12.49.42 PM.png
     
  10. delany

    delany

    Joined:
    Nov 20, 2013
    Posts:
    10
    FYI - you may already know - it seems to be the Screen.width that's returning the wrong value. If I change that enclosingBeginArea from:
    GUILayout.BeginArea(new Rect(10, 10, Screen.width - 20, 70), EditorStyles.helpBox);
    To:
    GUILayout.BeginArea(new Rect(10, 10, 800 - 20, 70), EditorStyles.helpBox);

    And restore those FlexibleSpace() calls ...Then I see all the buttons in a row - in a fixed-width Area of 800 width like this:

    Screen Shot 2021-07-20 at 5.44.28 PM.png
     
  11. metaphysician

    metaphysician

    Joined:
    May 29, 2012
    Posts:
    165
    hey there Jamie, on a Mac (High sierra 10.13.6) and definitely seem to be experiencing issues with the LipSync. saw the post about downloading the Mac OS CLI version of rhubarb so DLd it and ran it, which did seem to process but appears to have created a blank XML file. also the Lip Sync component has no awareness of the update XML file added, so while the audio file can be loaded visually now, there's no phoneme data. more importantly, adding phonemes manually does work, but saving doesn't seem to, so when you open things again, everything's blank. i did sort of try manually adding associations with the Text asset and the audio file but that didn't seem to get anywhere.

    so basically i'm dead in the water. on the good side the tool handles Blendshape creation very well for the most part, though renaming them easily would be nice. would also definitely like the ability to add more phonemes or blendshapes other than the default ones. sort of surprising its missing the s,z phonemes for example. anyway that's my observations for the moment.
     
  12. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    for interest what command did you run?
    Are you clicking the "save" button?
    upload_2021-8-5_8-53-54.png
    Adding new blendshapes is on my feature list
     
  13. b1gry4n

    b1gry4n

    Joined:
    Sep 11, 2013
    Posts:
    120
    I looked through these forums and watched the tutorial video, but couldnt find an answer to a question I have about puppet face. I see the tool is mostly aimed towards lip sync and animations, but what about character creation blend shapes commonly found in games when creating a character? Am I able to create different blend shapes (nose size, lip size, cheek size, jaw width, etc) and export these blendshapes? Are the blendshapes being "baked" into a new fbx file that you could later use somewhere else without the need for the puppet face scripts? (moving the sliders in the skinned mesh renderer for the blend shapes youve created with puppet face)
     
  14. SuperDanOsbourne

    SuperDanOsbourne

    Joined:
    Oct 3, 2017
    Posts:
    45
    Hey Jamie, any progress on getting the performance capture to work properly on a macbook pro?
    Also we were talking about the possibility of loading pre-recorded video files to capture.
     
  15. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Hi you can make these blend shapes and then use Unity's FBX Exporter to export your character with the blendshapes to use outside of Unity.
     
  16. merpheus

    merpheus

    Joined:
    Mar 5, 2013
    Posts:
    193
    Is there documentation on which phenomes are needed for face capture? I need to send a list of phenomes created as blendshapes to the character artist. )I am asking all the phenomes that would make the capture good, not just the bare limited/required ones -- in case there is a distinction) @jamieniman
     
    Last edited: Sep 10, 2021
  17. jawasjnsdjn

    jawasjnsdjn

    Joined:
    Mar 7, 2013
    Posts:
    34
    could load the video into something obs's virtual camera to play it back for puppetface to record later, would have to have the performer do the poses to configure it before they start then you would only need a camera for capture.
     
  18. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Hi, just saw your review - my apologies for not replying to this. I moved house in September - and must have seen the message and planned on replying but completely forgot (then without new messages on the forum I didnt get a reminder to reply). Ordinarily I try to be quick with support. I'm very sorry.
    I saw in your review you had other issues that hopefully I can help with.
    Starting with the error you are getting:
    Do you get a line number in the error? What version of Unity were you using?
     
  19. merpheus

    merpheus

    Joined:
    Mar 5, 2013
    Posts:
    193
    Thanks for returning back, I would love to correct my review once the problems I got resolved.
    So as a response, I am using 2021.2 latest beta. As the whole error, this is what I get:

    Code (CSharp):
    1. ArgumentException: Value must be a Com object.
    2. Parameter name: o
    3. System.Runtime.InteropServices.Marshal.ReleaseComObject (System.Object o) (at <e03f4e9558c44d56aad5167e5810a238>:0)
    4. System.Runtime.InteropServices.Marshal.FinalReleaseComObject (System.Object o) (at <e03f4e9558c44d56aad5167e5810a238>:0)
    5. Ookii.Dialogs.VistaFileDialog.RunFileDialog (System.IntPtr hwndOwner) (at <e3ade0b9a63043c89b5fbc325cc98b60>:0)
    6. Ookii.Dialogs.VistaFileDialog.RunDialog (System.IntPtr hwndOwner) (at <e3ade0b9a63043c89b5fbc325cc98b60>:0)
    7. System.Windows.Forms.CommonDialog.ShowDialog (System.Windows.Forms.IWin32Window owner) (at <1d2e5fd5dacd4b00b29699699894aec6>:0)
    8. (wrapper remoting-invoke-with-check) System.Windows.Forms.CommonDialog.ShowDialog(System.Windows.Forms.IWin32Window)
    9. SFB.StandaloneFileBrowserWindows.OpenFilePanel (System.String title, System.String directory, SFB.ExtensionFilter[] extensions, System.Boolean multiselect) (at Assets/PuppetFace/Tools/Editor/StandaloneFileBrowser/StandaloneFileBrowserWindows.cs:38)
    10. SFB.StandaloneFileBrowser.OpenFilePanel (System.String title, System.String directory, SFB.ExtensionFilter[] extensions, System.Boolean multiselect) (at Assets/PuppetFace/Tools/Editor/StandaloneFileBrowser/StandaloneFileBrowser.cs:51)
    11. PuppetFace.PuppetFaceEditor.ConvertAudioToLipSync () (at Assets/PuppetFace/Scripts/Editor/PuppetFaceEditor.cs:475)
    12. PuppetFace.PuppetFaceEditor.OnGUI () (at Assets/PuppetFace/Scripts/Editor/PuppetFaceEditor.cs:171)
     
  20. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Ah - you're using the beta Unity, We support only actual full releases of Unity (tend to find Unity breaks things and then puts them back on final release). In this case the issue looks like it could be with a 3rd party asset browser we use called the StandaloneFileBrowser. I think I could make a quick fix for that. If you email me at puppet3Dunity@gmail.com I can email it to you.
     
  21. imtomgamedev

    imtomgamedev

    Joined:
    Jan 11, 2019
    Posts:
    3
    Hi Jamie, I just updated Puppet Face to v1.4.2 and am encountering this console error as well.

    "Assembly 'Packages/com.unity.collab-proxy/Lib/Editor/PlasticSCM/unityplastic.dll' will not be loaded due to errors:
    unityplastic references strong named System.Windows.Forms Assembly references: 4.0.0.0 Found in project: 2.0.0.0.
    Assembly Version Validation can be disabled in Player Settings "Assembly Version Validation"


    Enabled run unsafe code and deleted the single file "OpenCvSharp" as suggested, which does not fix the issue. I'm running Unity v2020.3.16f1 on Mac. Up until this new update, there were no Puppet Face console errors. Any suggestions?
     
  22. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    So I just discovered what is causing this - its the StandaloneFileBrowser that Puppet Face uses. For now can you delete this whole folder from the project and in the PuppetFaceEditor.cs replace the following:
    Code (CSharp):
    1. var extensions = new[] {
    2.             new ExtensionFilter("Sound Files", "mp3", "wav" ),
    3.             };
    4.             string[] files = StandaloneFileBrowser.OpenFilePanel("Select Wav Files", "", extensions, true);
    5.             string path = EditorUtility.OpenFilePanel("Select Wav File", "", "wav");
    6.             if (files.Length != 0)
    7.             {
    8.                 foreach (string path in files)
    9.                 {
    with :
    Code (CSharp):
    1. string path = EditorUtility.OpenFilePanel("Select Wav File", "", "wav");
    2.             {
    3.                 {
    Also comment out this line:
    Code (CSharp):
    1. //using SFB;
     
  23. imtomgamedev

    imtomgamedev

    Joined:
    Jan 11, 2019
    Posts:
    3
    Thank you Jamie for the guidance. That appears to have suppressed the issue.

    Question regarding this fix, the code editor suggests there's a syntax error possibly (I'm no programmer). See image, anything I need to do?

    Last, one suggestion regarding my setup that may play a part...I don't know. I've setup a separate project with only a few assets installed among them are puppet face and puppet 3D and some others. However, my main project that produced the console errors upon updating Puppet Face actively utilizes FMOD. Since the code you had me remove appears to pertain to sound, could there be some interplay between Puppet Face and FMOD causing this issue? Just a suggestion. Thanks, Tom!
     
  24. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    1,415
    This is such a great tool.

    Any chance at adding emotion support? -- Also how about cartoon characters that have/use 2D textures for parts of their faces? -- So far, I've not seen a solution for animating textures for mouth/eye shapes, but I think this system would be great for that too, assuming emotion support was added. D:
     
  25. FiveFingers

    FiveFingers

    Joined:
    Oct 15, 2009
    Posts:
    541
    Hi Jamie, great package.
    We have just bought it from the Asset Store.
    We are wondering if you plan to add more blend shapes mapping for Performance Capture because OpenMouth, O-mouth and smile are too few to make facial expression.

    How hard would it be to modify performancecapture.cs to implement other face points to work together with other blend shapes, just like Lip Synch does and more ?

    Thank you in advance
     
    Last edited: Nov 24, 2021
  26. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    I'm working on supporting emotions for the next update (as a start it will allow you to add custom blendshapes to the lipsync animation)
     
    awesomedata likes this.
  27. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    74
    For clarification, can I drive facial animation in real-time at runtime? This would be awesome for "avatar interviews" and podcasts.
     
  28. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    It's not extremely difficult, just a bit of a manual process at the moment (adding all the variables, hooking them up etc).
    I'm currently working on the lipSync feature of PuppetFace. My next plan is to revisit the Performance Capture, probably I want to first integrate MediaPipe for improved tracking, and next would be to expose more feature tracking than it does currently.
     
  29. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Grab the latest version of Puppet Face (v1.5.0)- you can now add other blendshapes to lipsync animation
     
  30. awesomedata

    awesomedata

    Joined:
    Oct 8, 2014
    Posts:
    1,415
    Nice. Thanks!

    Just a quick question -- I didn't notice at a glance, but is it possible to overlay multiple blendshapes at once on the face for emotions (i.e. angry eyes blend, but smirking mouth blend)?

    Btw, this would be a HUGE improvement. This would be what most users expect IMO. Please add support for this, if possible, asap. :)
     
  31. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    yup it would

    noted :)
     
  32. NonhumanStoltz

    NonhumanStoltz

    Joined:
    Dec 6, 2020
    Posts:
    19

    Hello. I'm interested in the asset but pretty much all of my animations are humanoid and have no expressions when moving about. This seems like a good tool to add those animations/expressions in by motion capture or hand, so I was wondering if humanoid support was ever added? I did a search to find out and came across this post back in may. I'd like to avoid blending the mesh as much as possible since my 3d animation skills are, lets say rudimentary at best, so there might be something I'm missing since I'm more of a 2d guy.

    Edit: Actually, turns out I got this asset already. Must have been from some humble bundle. Neat! Still wondering if adding facial animations to humanoid animations is possible though.
     
    Last edited: Dec 7, 2021
  33. SkandYxyz

    SkandYxyz

    Joined:
    Mar 13, 2015
    Posts:
    77
    Hey.

    I saw that automatic eye movement does not work with the character laying on the side, as the rotation is still horizontal. It would make sense to save local and not global transforms here.
    But mabe I'm doing sth. wrong.

    Keep up the good work,
    Andre
     
  34. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    This could be something I've overlooked.
    Trying it on the demo character I get this, which I can see isn't perfect:

    upload_2021-12-9_10-55-8.gif

    Thanks for pointing it out to me, I'll take a look into it
     
    awesomedata likes this.
  35. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    The Lipsync feature is fine with humanoid.
    The Performance capture feature however requires a few more steps to get working with humanoids. Here's what to do:
    The performance capture controls the head animation - to get this to work on humanoid rigs you will need to export all the animations that get generated as fbx files. You can do this using Unity's FBXexporter.
    Hopefully I can make this simpler in future, but this is the way around it for now.
     
  36. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    74
    Can existing blend shapes in a model (from Character Creator or Ready Player Me for example) be connected to Puppet Face Blend Shapes in BlendShapeManager, then refined?
    or is it assumed perhaps that blendshapes are always made from a neutral face?
     
  37. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    74
    I notice that my model's head tracking is reversed. If I pivot my head upwards, the model looks down and vice versa.
    Can this be reversed?
     
  38. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    I haven't got experience with those assets, but as long as the skinned mesh they create is fixed, you can edit it with the BlendshapeManager. (By fixed I mean there isn't an editor script that can dynamically change the Mesh that is generated and assigned to the SkinnedMeshRenderer)
     
    XyrisKenn likes this.
  39. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Try putting negative numbers in the Rotation Amount:
    upload_2021-12-15_9-30-13.png
     
    XyrisKenn likes this.
  40. XyrisKenn

    XyrisKenn

    Joined:
    Dec 8, 2015
    Posts:
    74
    I hope that eye tracking and blinks become possible with Puppet Face. This software is truly enjoyable to use!
     
  41. frankadimcosta

    frankadimcosta

    Joined:
    Jan 14, 2015
    Posts:
    195
    1) Noob question: can I reset a blend shape in lipsync editor ? Not in mesh modifier using erase tool. TNX !

    2) A request: can you change the color of name of blendshape in blenshape manager from white to a dark color ? White on grey is hard to read. TNX

    3) Can you implement a sort of "slow motion" in lipsync editor, setting tiimescale and picth of audiosource ?

    4) Building the project i have this error:
    "Assets\PuppetFace\Scripts\AfterAllPostprocessor.cs(7,35): error CS0246: The type or namespace name 'AssetPostprocessor' could not be found (are you missing a using directive or an assembly reference?)"
    I suppose this script should not be included in building, you should move it into editor folder.



    2 done myself:
    file BlendShapeManagerEditor.cs
    guistyleLabel.normal.textColor = Color.black; //white;
     
    Last edited: Dec 20, 2021
  42. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Apologies not sure what you are asking here? :)
    Oh, cool I'll add that to the features list
    Thanks so much for pointing this out - this was an error in the previous update. I've fixed it in v1.5.1 now
     
    frankadimcosta likes this.
  43. frankadimcosta

    frankadimcosta

    Joined:
    Jan 14, 2015
    Posts:
    195
    I'd like to reset the blendshape to "new" state. Or copy it from REST blendshape.

    Another question: it's possibile to add bones to lipsync face bones array AFTER editing ? The button "edit bone pose" shows only the first bone in array, in the inspector.

    TNX for support !
     
    Last edited: Dec 21, 2021
  44. dmenefee

    dmenefee

    Joined:
    Oct 14, 2019
    Posts:
    136
    @jamieniman Hi. Have you looked further into UMA integration?
     
  45. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    You make a new blendshape with the same name and setting it will overwrite the old one.
    Just click "Edit Bone Pose" on the Rest pose (after assigning the new bones). You will then need to resetup all the bone poses for the rest of the phonemes.
     
  46. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    Not had a chance I'm afraid - other features and fixes have been higher on my todo list
    I havent had any experience with UMA - what features does PuppetFace need to allow it to work with it?
     
  47. jawasjnsdjn

    jawasjnsdjn

    Joined:
    Mar 7, 2013
    Posts:
    34
    I think i found a bug or i don't understand something i am using unity 2021.2.8f1 on win10
    The performance capture options show when i have a virtual camera selected but when i choose a physical camera everything from the skin selector down to the capture button completely disappears. it does show the video on the plane in play mode and it does track but is unusable as it can't be calibrated.
    -+-+-
    also the manual could be more fleshed out on playing multiple lipsyncs and from an index and could use a demo scene with multiple lipsyncs hooked up to a button
     
  48. dmenefee

    dmenefee

    Joined:
    Oct 14, 2019
    Posts:
    136
    I don’t know, yet, not having purchased PF (all my characters are UMA and I will pick it up if it works with them). UMA characters have their own way of managing blend shapes, so there may need to be changes to support that. I have LipSync, and it provides a UMA component to control facial expressions…
     
  49. unity_a8B9wWbnDhO-Jw

    unity_a8B9wWbnDhO-Jw

    Joined:
    May 13, 2019
    Posts:
    2
    Hello! I was almost buying the package, but I read that some parts of it are only compatible with windows. I only run on linux, so what can I do and what cannot I do with Puppet face?
     
  50. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    955
    It uses Rhubarb to convert the lip sync. (it runs and exe which is why windows only) But this can be done manually and then loaded into Unity using PuppetFace (You will just need to manually assign the html file to the lipSync component)
    Here is link to the linux version of Rhubarb - it will need to run in command line
    https://github.com/DanielSWolf/rhub...oad/v1.11.0/Rhubarb-Lip-Sync-1.11.0-Linux.zip