Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[RELEASED] Puppet Face - All-In-1 Facial Animation for Unity

Discussion in 'Assets and Asset Store' started by jamieniman, Dec 17, 2020.

  1. AGregori

    AGregori

    Joined:
    Dec 11, 2014
    Posts:
    527
    This is great for a new asset, and is already making a name for itself after being just 2 months on the store. However, it needs to support DAZ, UMA and perhaps other 3rd party character creators. Again, cool asset though.
     
    Last edited: Feb 27, 2021
  2. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Android is supported. I should say though, the performance capture is made for pre-recording animations. At the moment it's not made for live face capture on mobile. (I'm looking to implement AR foundation at some point)
     
  3. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985

    1. Such a great idea - I'm going to add this to my trello feature list.

      This would also be a lovely feature. I'd need to have a think how best to implement this so that it would be a useful tool for a lot of people.

    There's certainly room for PuppetFace to go a lot deeper with its face rigging tools. Thanks for the great suggestions.
     
    awesomedata likes this.
  4. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Totally agree, I'm working on improving the workflow with DAZ currently.
     
    JonasLuz likes this.
  5. hean

    hean

    Joined:
    Jul 21, 2013
    Posts:
    20
    Hey, I'm getting an error when trying to convert to lip sync an exported audio file(wav) from audacity
    upload_2021-2-28_4-34-0.png
     
  6. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Is your audio file somewhere inside the projects assets folder? (it should be)
     
  7. netpost

    netpost

    Joined:
    May 6, 2018
    Posts:
    388
    @jamieniman

    I think you skipped my questions, because you answered all of them except mines so please see a copy of my previous post below : ;)

    Congratulations! Puppet Face seems like a great asset. I have a few questions if you don't mind.

    1-Do you have any plans to offer the option to record the facial using the iphone for AI face recognition?

    At the moment, I am recording with an iphone in a third party app (Reallusion Live Face) and exporting the fbx animations to Unity which is a pita.

    2-Do you think Puppet Face can edit the recorded blenshapes and exported from T_pose FBX files?

    Specifically , Is it compatible with Character Creator 3 (CC3)?
    Meaning, if I record blenshapes and voices in a third party application (Reallusion Live Face) can the CC3 recorded blendshape be imported and modify in Face Puppet?

    Thanks!
     
  8. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Apologies, accidentally missed it!

    I have thoughts to implement ARKit at somepoint, (and maybe ARCore)

    You can make other meshes into blendshapes as long as the topology is completely identical to the original model. (Then you would add the meshModifier component and set it as a blend shape on the the skinned model.).
     
  9. netpost

    netpost

    Joined:
    May 6, 2018
    Posts:
    388
    @jamieniman Thanks for the quick reply

    Sorry...I am not sure I understand. If blendshapes have been recorded previously in a third party app and export to fbx with the the character model can you import and edit these blendshapes in Face Puppet? Is the blendshapes import process automatic or do you have to recreate all blendshapes? Thanks!
     
  10. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I think I misunderstood you - if your character already has blendshapes you can use PuppetFace to edit them however you want using the blend shape manager. (I thought you were asking about making another unskinned model a blend shape of your skinned character)
     
  11. netpost

    netpost

    Joined:
    May 6, 2018
    Posts:
    388
    @jamieniman

    Great! Buying now. Tanks again for the quick reply and support.
     
    TymNetwork likes this.
  12. jnbbender

    jnbbender

    Joined:
    May 25, 2017
    Posts:
    486
    @jamieniman Never worked with blend shapes before but what if my character does not have blend shapes? It has a jaw (of course) and transforms for eyebrows, lids, lips, etc. How will I be able to use Puppet Face to manipulate my character if it is simply a single Skinned Mesh Renderer. Didn't quite get that from the tut video.
     
  13. netpost

    netpost

    Joined:
    May 6, 2018
    Posts:
    388
    @jamieniman Doing a test with Reallusion CC3 Character, The blendshapes manager is working very well at auto finding the blendshapes. But I have a small problem when attempting to edit a blenshape, when clicking on a blendshape to edit it I see some strange things... please see attached picture. I am using a Tpose character with a layer animation (with a head mask) for the facial capture animation. Any help would be greatly appreciated. Using Unity 2019.4.19f1. Thank you.
     

    Attached Files:

  14. netpost

    netpost

    Joined:
    May 6, 2018
    Posts:
    388
    @jamieniman This is another test with a simpe T-pose. Do I need to work with an A_pose? It seems like Face Puppet is using a A_pose by default.
     

    Attached Files:

  15. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    You can use PuppetFace to create new blend shapes, and/or you can also make use of your bones to set face poses for the lip sync.
     
  16. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    The mesh you see when you make a blend shape will be the "Bind Pose" of your character. This means when the character was first skinned to the bones this will be the model that was skinned. What is likely is the Reallusion character was skinned in an A-pose and then the bones were transformed into a Tpose. From the looks of things the mesh collider isn't matching the blendshape. Is there anyway you can send me a test character (puppet3dunity@gmail.com) I want to fix this issue.
     
  17. netpost

    netpost

    Joined:
    May 6, 2018
    Posts:
    388
    @jamieniman Thanks again for the quick reply.

    This makes sense. I just found this info on Reallusion's website:
    The two previous sent characters were CC3 Base+ . I made another test with an old CC3 base character (not the CC3 Base+) and the binding pose is fine. Unfortunately I work only with CC3+ .
     
  18. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Making the blendshape from the bind pose should be working though, I would like to look at fixing it, are you able to send me the zombie character so I can see if I can get the mesh collider to match the bindpose?
     
  19. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Actually - I believe I have a fix for the issue. Could you email me (puppet3dunity@gmail.com) and I'll send you the update.
     
  20. netpost

    netpost

    Joined:
    May 6, 2018
    Posts:
    388
    @ jamieniman I will do.
     
  21. dmenefee

    dmenefee

    Joined:
    Oct 14, 2019
    Posts:
    142
    This looks like an excellent tool. As asked by someone else earlier, is there UMA support? Thanks!
     
  22. staddonl

    staddonl

    Joined:
    Feb 8, 2013
    Posts:
    4
    I just bought this and I'm having trouble in Unity 2020.2.6f1 using the HDRP with lip sync. Whenever I run convert audio files on any audio, including the shipped sample audio, a completely blank file gets produced for the lip sync mouth cues.

    It happens on all Skinned Mesh Renderers with all audio. Any ideas?
     
  23. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I just tried and it worked for me. Can you tell me a few things:
    - are you using windows?
    - do you get an error in the console?
    - do you see a black window showing the conversion happening?
    - are you selecting files in the project itself (below the Assets folder)?
     
    staddonl likes this.
  24. staddonl

    staddonl

    Joined:
    Feb 8, 2013
    Posts:
    4
    - I'm using Windows
    - There are no errors in the console
    - No, I'm not seeing the black window showing the conversion
    - The files themselves are below the Assets folder (specifically Assets/Audio/Voice)

    EDIT: I've retried and the black processing window does come up, albeit so quickly that I can't see what's appearing on it.
     
    Last edited: Mar 2, 2021
  25. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I'm currently looking into UMA, I'll let you know when I know :)
     
    Last edited: Mar 3, 2021
    JonasLuz and dmenefee like this.
  26. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Are they wav files, and are there any spaces in the name?(it should be a wav with no spaces in the name)
     
  27. staddonl

    staddonl

    Joined:
    Feb 8, 2013
    Posts:
    4
    They're all WAVs with alphanumeric names I'm afraid. I get the same result even if using any of the three sample WAVs that ship with the package.

    I'm just trying it out in 2019.3.12f1 to see if I have the same issues.

    Thanks for the speedy replies btw, it's much appreciated :)
     
  28. staddonl

    staddonl

    Joined:
    Feb 8, 2013
    Posts:
    4
    I can confirm that I don't get the issue in 2019.3.12f1. From my own standpoint, I'm happy to work in 2019.3 instead of 2020, as this is only for a tech demo for our next project. I'll try starting a new 2020 project from scratch to see if I can recreate, in case it's something to do with how I'd set it up.

    Thanks very much for your help :)
     
  29. netpost

    netpost

    Joined:
    May 6, 2018
    Posts:
    388
    @jamieniman Thanks for your help today. Great support as always!
     
    jamieniman likes this.
  30. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I'd be interested to find out the results. I realised I tried 2020.2.3f1 HDRP (as opposed to 2020.2.6f1) Perhaps something broke in the small updates between then, seems unlikely but I'll download the newer version to be doubly sure.
    I suppose for now, as a workaround for that project, you could convert files in a separate project and assign them to the LipSync component in that project.
     
    staddonl likes this.
  31. dmenefee

    dmenefee

    Joined:
    Oct 14, 2019
    Posts:
    142
    Awesome. If it works or if it’ll work very soon, it’s going in my shopping cart. :)
     
  32. jnbbender

    jnbbender

    Joined:
    May 25, 2017
    Posts:
    486
    @jamieniman Went ahead and purchased. I was hoping the fix regarding the 10k verts mentioned by a reviewer would have been fixed (Guess I should have looked at the Release history). Anyway, having the same issue. Any idea on the time frame of the update?
     
    Last edited: Mar 7, 2021
  33. TymNetwork

    TymNetwork

    Joined:
    Jan 16, 2014
    Posts:
    84
    I need to learn more about making my own character as well.
    This is a good learning tool for me. I will ask questions as I go.
    - I did noticed that most of the tutorial videos are now unavailable.
     
  34. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I can send you the fix, I need to to test it a bit more before releasing it. (Just email me at Puppet3dUnity@gmail.com and I 'll send it to you)
     
  35. waltercl

    waltercl

    Joined:
    Dec 30, 2018
    Posts:
    47
    When I click on a character to set up the blendshapes I see a large green mesh covering my character and when I try to adjust the lips it only pulls down one small portion no matter what size brush I'm using. Does this have something to do with the issue with too many polygons?

    If I had a character with no rigging at all, would I be able to use your other tool (Puppet3D) for rigging the face and preparing it for this tool so I can do the blendshapes?
     
  36. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    could be to do with the bad bind pose issue - which I have also hopefully fixed for the next release. Would you like to try it earlier? (Email Puppet3DUnity@gmail.com) It would be good to see that it covers your issue.

    Yup :)
     
  37. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I believe I've emailed it to you (I think you're the same person who emailed me - otherwise please email me puppet3dunity@gmail.com and I'll send you the fix)
     
  38. dmenefee

    dmenefee

    Joined:
    Oct 14, 2019
    Posts:
    142
    Howdy. Have you had time to look into UMA integration?
     
  39. bluemoon

    bluemoon

    Joined:
    Dec 14, 2012
    Posts:
    85
    I am just scratching the surface when it comes to learning about blend shapes so forgive these are obvious questions.
    1) Will puppet face effect other shape keys for example like those used in a character creator. Nose and mouth size sliders?
    2) Could I use Puppet faces shape key sculpting to create shapes a character creator?

    Thanks.
     
  40. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Yes, you can edit ones that have already been made.
    Yes, you can make new blendshapes on the character creator characters.
     
  41. waltercl

    waltercl

    Joined:
    Dec 30, 2018
    Posts:
    47
    I'm really enjoying the ability to create Blendshapes, and it's working well for me. One question I have though is that on some models when I click on the Create New Blendshape button the head of my model shrinks to about 1/100th the size of the model and i have to try and get in really tight to edit the mouth for blendshapes. I'm sure this has more to do with the model than Puppet Face, but I'm wondering if there's any way to improve this.

    Edit: Found an answer. I just increase the scale on the blendshape and it makes it big enough to edit the mouth. I do have to set the radius to a very small number but it works.
     
    Last edited: Mar 24, 2021
  42. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Does your skinned model have any transforms on it?
     
  43. squallfgang

    squallfgang

    Joined:
    Sep 13, 2016
    Posts:
    21
    Hey just bought Puppet Face, looks great so far bu two problems occured:

    1. tried to Record and Animation (including Eyebrow movement) by using the camera with the peformance capture system. But even though there is an option on the performance capture component that i enabled the eybrow didn't change. I gave the eybrow mesh its own blendshape manager and created some blendshapes but it still didn't work. How do i set this up?

    2.The camera loses tracking of the head extremely fast. I have a high quality Kinect camera with great image quality. I use studio ,lights but still the tracking does not tolerate the slightest rotation. Is there something i can do?
     
  44. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Currently the performance capture requires the eyebrows to be part of the same mesh as the head - are your eyebrows separate meshes?
    Puppet Face uses OpenCV for its face detection (which is why you don't need an expensive iPhone to use it)- I find it helps to make sure your face is in the center of the camera feed - and there is a sweet spot in terms distance from camera that allows the most rotation.
     
  45. Barnimsson

    Barnimsson

    Joined:
    Nov 22, 2020
    Posts:
    7
    Hi!

    Just bought PuppetFace, but need some help.
    Each time I try to go through the tutorial on youtube I stuck on converting an audio file with this message in the console:

    FormatException: Input string was not in a correct format.
    System.Number.ParseSingle (System.String value, System.Globalization.NumberStyles options, System.Globalization.NumberFormatInfo numfmt) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    System.Single.Parse (System.String s, System.Globalization.NumberStyles style, System.Globalization.NumberFormatInfo info) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    System.Single.Parse (System.String s) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    PuppetFace.LipSync.GetPhonemes (System.Int32 index) (at Assets/PuppetFace/Scripts/LipSync.cs:345)
    PuppetFace.PuppetFaceEditor.ReloadNodes (UnityEngine.Vector2 mousePosition) (at Assets/PuppetFace/Scripts/Editor/PuppetFaceEditor.cs:1125)
    PuppetFace.PuppetFaceEditor.OnGUI () (at Assets/PuppetFace/Scripts/Editor/PuppetFaceEditor.cs:177)
    UnityEditor.HostView.InvokeOnGUI (UnityEngine.Rect onGUIPosition, UnityEngine.Rect viewRect) (at <7587a42c8322471cbc42b27c9b8eab3c>:0)
    UnityEditor.DockArea.DrawView (UnityEngine.Rect viewRect, UnityEngine.Rect dockAreaRect) (at <7587a42c8322471cbc42b27c9b8eab3c>:0)
    UnityEditor.DockArea.OldOnGUI () (at <7587a42c8322471cbc42b27c9b8eab3c>:0)
    UnityEngine.UIElements.IMGUIContainer.DoOnGUI (UnityEngine.Event evt, UnityEngine.Matrix4x4 parentTransform, UnityEngine.Rect clippingRect, System.Boolean isComputingLayout, UnityEngine.Rect layoutSize, System.Action onGUIHandler, System.Boolean canAffectFocus) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.IMGUIContainer.HandleIMGUIEvent (UnityEngine.Event e, UnityEngine.Matrix4x4 worldTransform, UnityEngine.Rect clippingRect, System.Action onGUIHandler, System.Boolean canAffectFocus) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.IMGUIContainer.HandleIMGUIEvent (UnityEngine.Event e, System.Action onGUIHandler, System.Boolean canAffectFocus) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.IMGUIContainer.HandleIMGUIEvent (UnityEngine.Event e, System.Boolean canAffectFocus) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.IMGUIContainer.SendEventToIMGUIRaw (UnityEngine.UIElements.EventBase evt, System.Boolean canAffectFocus, System.Boolean verifyBounds) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.IMGUIContainer.SendEventToIMGUI (UnityEngine.UIElements.EventBase evt, System.Boolean canAffectFocus, System.Boolean verifyBounds) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.IMGUIContainer.HandleEvent (UnityEngine.UIElements.EventBase evt) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.CallbackEventHandler.HandleEventAtTargetPhase (UnityEngine.UIElements.EventBase evt) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.MouseCaptureDispatchingStrategy.DispatchEvent (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.EventDispatcher.ApplyDispatchingStrategies (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel, System.Boolean imguiEventIsInitiallyUsed) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.EventDispatcher.ProcessEvent (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.EventDispatcher.ProcessEventQueue () (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.EventDispatcher.OpenGate () (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.EventDispatcherGate.Dispose () (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.EventDispatcher.ProcessEvent (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.EventDispatcher.Dispatch (UnityEngine.UIElements.EventBase evt, UnityEngine.UIElements.IPanel panel, UnityEngine.UIElements.DispatchMode dispatchMode) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.BaseVisualElementPanel.SendEvent (UnityEngine.UIElements.EventBase e, UnityEngine.UIElements.DispatchMode dispatchMode) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.UIElementsUtility.DoDispatch (UnityEngine.UIElements.BaseVisualElementPanel panel) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.UIElementsUtility.UnityEngine.UIElements.IUIElementsUtility.ProcessEvent (System.Int32 instanceID, System.IntPtr nativeEventPtr, System.Boolean& eventHandled) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.UIEventRegistration.ProcessEvent (System.Int32 instanceID, System.IntPtr nativeEventPtr) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.UIElements.UIEventRegistration+<>c.<.cctor>b__1_2 (System.Int32 i, System.IntPtr ptr) (at <16a864d8d57443bb82561171867685b1>:0)
    UnityEngine.GUIUtility.ProcessEvent (System.Int32 instanceID, System.IntPtr nativeEventPtr, System.Boolean& result) (at <6ddf8eac3856492ab1b8cf42618915cc>:0)

    upload_2021-4-16_21-0-36.png

    Windows, Unity 2020.3.4f1
     
  46. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    What happens when you click "Reload"?
     
  47. Barnimsson

    Barnimsson

    Joined:
    Nov 22, 2020
    Posts:
    7
    Nothin
    Nothing
     
  48. paulacs

    paulacs

    Joined:
    Mar 4, 2021
    Posts:
    4
    Hello, i've just buy it and try to use it, but iv've got the following error:

    NullReferenceException: Object reference not set to an instance of an object
    PuppetFace.BlendShapeManagerEditor.OnInspectorGUI () (at Assets/PuppetFace/Scripts/Editor/BlendShapeManagerEditor.cs:133)
    UnityEditor.UIElements.InspectorElement+<>c__DisplayClass59_0.<CreateIMGUIInspectorFromEditor>b__0 () (at <420b942f64794d4da66420603f126f7b>:0)
    UnityEngine.GUIUtility:processEvent(Int32, IntPtr, Boolean&)

    Thank you!
     
  49. Barnimsson

    Barnimsson

    Joined:
    Nov 22, 2020
    Posts:
    7
    Hi @jamieniman,

    Any thoughts about it?

    Tried few times and when I click to reload the result is like that (as I showed before):

    upload_2021-4-23_18-24-21.png

    Fresh Unity installation, new project, package imported, new empty scene with BrownieGirl model and BlendShapeManage attached to face_GEO + LipSync + rhubarub.exe in Tools/LipSync/rhubarb folder.
    Unity 2020.3.5f1 Windows 10.

    Thanks,
    Maciej
     
    Last edited: Apr 23, 2021
  50. dmenefee

    dmenefee

    Joined:
    Oct 14, 2019
    Posts:
    142
    Still curious about UMA. I’d like to pick this up, especially when on sale, since LipSync is no longer supported... UMA blendshapes are probably weird.