Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[RELEASED] Puppet Face - All-In-1 Facial Animation for Unity

Discussion in 'Assets and Asset Store' started by jamieniman, Dec 17, 2020.

  1. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Introducing Puppet Face - All-In-1 Facial Animation for Unity!



    Features included:

    Blend Shape Sculpting
    • Create new blend shapes & edit old ones.
    • A sculpting tool that respects topology (eg. you can move your top lip without affecting the bottom lip!).
    • Using the Blend Shape Manager you can see little snaps of your blend shapes.
    Audio to Lip Sync
    • Convert audio files to lip sync animation
    • Simple and quick setup.
    • Phoneme poses can be made from blend shapes, bones or a mix of the two.
    • Audio scrubbing makes it easy to edit and refine your phoneme animation for added perfectionism.
    Web Cam Performance Capture
    • Use your web cam to use your face and head motion to control your character.
    • Record your performance with audio.
    • Simple and quick setup
    Eye Motion
    • Automate eye motion and blinking.
    • Eye lids adjust with up and down, to give added realism to your eyes.

    Find it here:
    https://assetstore.unity.com/packages/slug/181312 .

    Any questions welcome!
     
    Last edited: Dec 18, 2020
  2. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    Does the lipsync support other languages than English?
     
  3. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    What would be the best webcam to use for motion capture?
    (min cam resolution and fps)
     
    AthrunVLokiz and Pourya-MDP like this.
  4. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Good questions!
    The speech recognition is setup to be phonetic so it works on other languages.
    I don't have exact specs for web cam. My one is a low end 1080 if that helps
     
    Last edited: Dec 31, 2020
    atomicjoe likes this.
  5. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    I currently use Salsa Lipsync Suite and I'm pretty happy with it.
    It gives really great results and matches any audio in realtime, so it's great for multi-language setups, since you don't have to prerecord the lipsync animation and will match any language. (although it's true that your plugin should match phonemes more accurately)
    But I like your plugin very much for the blendshape sculptor and webcam motion capture for the head animation.
    I would like to use it along with Salsa but I don't know how the two would work together.
    It would be nice to have an integration path for the two, since Salsa seems to be the standard right now and it's honestly working flawlessly for lipsync alone right now.
    Something like using the webcam motion capture to record the movements of the head + triggering emotion blendshapes.

    Edit: Does the webcam capture record the movement of each eyebrow independently or does it have the ability to recognize custom expressions to blend-in custom blendshapes? (like "happy", "sad", "surprised"... and blend them in-between)
     
    Last edited: Dec 19, 2020
    CrandellWS and Pourya-MDP like this.
  6. ChangoXP

    ChangoXP

    Joined:
    Aug 3, 2017
    Posts:
    4
    I don’t have much need for the facial features right now but the blend shape sculpting is very intriguing. If it works the way I assume it does, it would be a huge time saver for tweaking existing morphs and creating new one without having to switch out of the editor. I’m actually surprised I haven’t seen a tool like that for Unity before. There are a lot of blend shape mixers and controllers but nothing for sculpting per se.

    If you are planning to create any new videos, I’d like to see a demonstration of the blend shape sculpting and manager specifically. I understand the asset is new and has just been published, but more examples or documentation would be nice so I can see how this could work within my pipeline before purchasing it. (Also, you might want to create a link to this forum in the description so we don’t have to hunt it down.)
     
    atomicjoe likes this.
  7. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Puppet Face can be used to support other assets - so you could use it to improve your blend shapes using the sculpting feature, or you could use Puppet Face's lip sync for your hero/ cutscene where you want the quality of the facial animation to be much better. You can use the performance capture to animate the head rotation, and it currently registers smiles so you can use it to animate the smile blend shape and use it in conjunction with the other lipsync. I'm hoping to track more expressions as I further develop this feature.
     
    Last edited: Dec 21, 2020
  8. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Agreed, I have actually submitted an update with links in the description, just waiting for Unity to make it live.
    I should be making more tutorials going forward
     
    ChangoXP likes this.
  9. ChangoXP

    ChangoXP

    Joined:
    Aug 3, 2017
    Posts:
    4
    After watching the tutorial video a few times I think I'll go and give this a try now. Might even be able to use the head tracking and eye motion features to breathe extra life into my current assets.
     
    jamieniman likes this.
  10. reggie_sgs

    reggie_sgs

    Joined:
    Jun 29, 2014
    Posts:
    276
    I have some 3d models that were created without blendshapes as we weren't sure we were going to need facial animation. Now that the project will require the characters to talk, we need to add blendshapes and are currently looking for someone to create them. I saw this tool and wondered if this would indeed allow us to take a character and create new blendshapes when none were created originally. The characters are realistic humanoids and your video shows a cartoony styled character so I wondered how well this worked on other styles.
     
  11. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Yup, thats what puppet face is for. Can be used on any art style. Just check the mouth edges are open (not welded shut)
     
  12. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I'm currently looking into why for some people the InitialPuppetFaceLaunchEditor isn't running on import (this will cause it to error).
    Its easy to fix by just re-importing this script. If you've experienced this please get into contact - I'm trying to figure out what the cause is so I can fix it in the first place.
     
  13. Pourya-MDP

    Pourya-MDP

    Joined:
    May 18, 2017
    Posts:
    145
    Hi there
    Is it possible to have eyebrow movements in performance capture?
    I mean more control over face using this system
    Maybe some emotion presets?
    Angry
    Happy
    Sad
    Etc?
     
  14. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Yup, I've added these to my trello feature list.
     
  15. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    Will that expressions be able to mix smoothly or will it be more like a trigger? (ON/OFF each)
     
    Pourya-MDP likes this.
  16. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Hopefully smooth, it should work like smile does currently (this is the happy expression)
     
    Pourya-MDP and atomicjoe like this.
  17. ajaxlex

    ajaxlex

    Joined:
    Jul 15, 2016
    Posts:
    38
    Is the webcam feature tracking a module that can be used in-game, or only in editor?
     
  18. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    It works in-game, though it's been designed to be for capturing specifically.
     
    ajaxlex likes this.
  19. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    What would be better for PuppetFace webcam motion capture?
    1- 1920x1080 at 30fps
    or
    2- 640x480 at 60fps
     
  20. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I would go for the 1080p I think
     
    atomicjoe likes this.
  21. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I got this wrong earlier - the released version works phonetically so it works in other languages as well
     
    atomicjoe likes this.
  22. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I've update Puppet Face to v1.1, it now includes the brownie girl demo, and a couple of fixes. Happy new year!
    upload_2020-12-31_15-16-48.png
     
    atomicjoe likes this.
  23. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Hello. I like this asset very much. But there was a serious bug.
    It works fine until I load the second WAV file.
    Rhubarb's analysis information is correct when the third WAV file is registered. However, the WAV file referenced by Puppet Face's lip sync has become the second WAV file.
    After that, all refer to the previous WAV file. My mouth doesn't move. Please check and correct.
     
  24. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    I don't know if it's the direct cause, but I also got an error when importing and it didn't work.
    When I turned on "Project Settings> Player> Allow'unsafe' Code", all the errors disappeared and the asset worked.
     
  25. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Hello.
    Updated to the latest Version 1.1.
    The above problem has been resolved.
    Thank you.
     
    AthrunVLokiz likes this.
  26. siumanchun

    siumanchun

    Joined:
    Apr 7, 2013
    Posts:
    44
    Hi jamieniman :

    Please Help ! but I also got an error !
    I Version Unity 2019.4.0f1


    System.IO.IOException: Failed to Move File / Directory from 'Assets/PuppetFace/Gizmos' to 'Assets/Gizmos'.
    at UnityEditor.FileUtil.MoveFileOrDirectory (System.String source, System.String dest) [0x00041] in <af78ad00a40d4f61b82afc6cafb0a416>:0
    at PuppetFace.InitialPuppetFaceLaunchEditor..cctor () [0x00013] in F:\VT\New Unity Project TB\Assets\PuppetFace\Scripts\Editor\InitialPuppetFaceLaunchEditor.cs:15
    UnityEditor.EditorAssembliesrocessInitializeOnLoadAttributes(Type[])

    Thank You !
     
  27. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Hi, if you look in the inspector, do the lip sync files and wav files correspond correctly? (If not - quick fix would be to se tthem correctly there). Also dont forget to hit "reload" after you convert a file.
    Is it something you can find steps to reproduce? That would be really helpful for me.
    (By the way thank you for the review - I'm really pleased it works well in Japanese ;-) )
     
  28. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Aha! This error is what must have been causing the above issue I mentioned. Glad i separated it out in v1.1. There's a very easy work around, please move the gizmo folder into the assets folder directly.
     
  29. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Thank you. It works fine.

    Is it possible to use a lip sync file for the timeline?
    I read the manual but didn't know how to use it except while playing the game.
    I want to work with lip sync files on the timeline and other cinematic sequencers.
    Please tell me how.
    Thank you.
     
    JonasLuz likes this.
  30. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    That's a must have feature, it's on the top of my list.
    In the meantime you need to have a script on a game object that plays the lip sync which you enable when it needs to play.
     
    JonasLuz likes this.
  31. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Thank you for the reply.
    I'm looking forward to the version upgrade.
    Thank you.
     
  32. somosky

    somosky

    Joined:
    Feb 15, 2014
    Posts:
    138
    Hi have you tested this with Reallusion Character Creator 3? I recently got it and would love to see the results using their characters and your asset.
     
  33. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Looks like a nice asset - Puppet Face works with any character as long as the mouth isn't sewed shut if (it should at least be possible to move the upper and lower lip apart) It would be extremely shocking if Reallusion had glued shut mouths!!
    (Also just check if the character has eye bones - if not, I have a rigging tool I'd recommend Puppet3D :) )
     
  34. somosky

    somosky

    Joined:
    Feb 15, 2014
    Posts:
    138
    Sounds great. I'll be picking this tool up later this week.
     
  35. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Hello.

    I created a new project for a simple test.
    Added Puppet Face (version 1.1) to the clean project.
    I get an error message every time I run it after importing.


    The message is as follows.

    System.IO.IOException: Failed to Move File / Directory from'Assets / PuppetFace / Gizmos' to'Assets / Gizmos'.
    at UnityEditor.FileUtil.MoveFileOrDirectory (System.String source, System.String dest) [0x00041] in <2f1c602eae0d45c293fff3e3aef759fa>: 0
    at PuppetFace.InitialPuppetFaceLaunchEditor..cctor () [0x00013] in C: \ Unity_Develop \ Slate_Lip_Test \ Assets \ PuppetFace \ Scripts \ Editor \ InitialPuppetFaceLaunchEditor.cs: 15
    UnityEditor.EditorAssemblies: ProcessInitializeOnLoadAttributes (Type [])


    The project under construction has deleted version 1.0 assets and imported version 1.1, so it seems that no error has occurred.
    Please confirm.
     
  36. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Yes, the gizmo folder is missing in 1.1 by mistake. All it means is there wont be little icons on the components for PuppetFace, and there is the error on the first load of the project.
    For now you can copy that Gizmo folder over from the other project, (or just delete the InitialPuppetFaceLaunchEditor.cs)
     
  37. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Thank you for confirmation.
    OK. I was relieved to find that there was no problem with the system.
    Thank you.
     
  38. Xbionic93

    Xbionic93

    Joined:
    Nov 14, 2020
    Posts:
    5
    Is it possible to animate Mixamo characters with this?
     
  39. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Yup!
     
  40. somosky

    somosky

    Joined:
    Feb 15, 2014
    Posts:
    138
    Hi I purchased your asset today and imported it into a clean project. Upon installation it gives me several error messages saying to "Enable Unsafe Code" through the player tab which I have done and that gets rid of the error messages. Is that intended?

    Also when trying to use Capture mode I'm getting the error message

    Could not connect pins - RenderStream()
    UnityEngine.WebCamTexture:play()
    OpenCvSharp.Demo.WebCamera:set_DeviceName(String) (at Assets/PuppetFace/Tools/OpenCV+Unity/Demo/Scripts/WebCamera.cs:71)
    OpenCvSharp.Demo.WebCamera:Awake() (at Assets/PuppetFace/Tools/OpenCV+Unity/Demo/Scripts/WebCamera.cs:114)
    PuppetFace.PerformanceCapture:Awake() (at Assets/PuppetFace/Scripts/PerformanceCapture.cs:100)
     
  41. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I've just submitted v1.2 with a fix for the first issue (It was meant to change that setting automatically - so what you did was correct)

    The second one will have something to do with your webcam setup. Do you have a webcam connected?

    (Also v1.2 now has a dropdown where you can choose the device. If that device is being used by another application you will need to shut that application. For example if you use OBS Studio - if you are using a video capture source this can take control of your webcam and so stop Unity from being able to use it.)
     
  42. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    I've made a specific LipSync Track for Timeline for v1.2 (just been submitted to the asset store)
    Keep an eye out for the update when it comes :)
     
  43. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Hello. Thank you very much.
    I'm looking forward to it.
    I'm currently scripting and using it to work on the cinematic sequencer Slate. The test went well.
    I didn't like LipSync Pro. I was very happy when I found this asset.
    Thank you.
     
  44. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Hello.
    I have a new request.
    I would like a button to add a new blend shape to the state created by manipulating some of the blend shape bars in the blend shape manager.
    Please consider it.
    Thank you.
     
    JonasLuz likes this.
  45. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    That's a nice idea, I'll add it to the list :)
     
    JonasLuz and atomicjoe like this.
  46. hdguevara94

    hdguevara94

    Joined:
    Oct 15, 2019
    Posts:
    1
    Hi Man I'd like to try this asset before to buyit! :s it looks like very good
     
    castana1962 likes this.
  47. jamieniman

    jamieniman

    Joined:
    Jan 7, 2013
    Posts:
    985
    Puppet v1.2 is out now!
    It has a lip sync track for Unity Timeline:



    Changes:
    • New - Lip Sync Timeline Track
    • New - Eyebrows performance tracking
    • New - Select desired Webcam
    • Fix - Added Assembly Definition
    • Fix - Gizmo folder back
     
    JonasLuz and AthrunVLokiz like this.
  48. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Hello. I tested the Lip Sync Timeline Track.

    No matter where you place the lip-sync clip on the track, the starting point will be 0 time.
    And the lip sync index is called 0 no matter what number you enter.
    I tried arranging two lip sync clips on a track, but the second lip motion didn't play.

    I used Slate to create a sequence.
    I was able to call my own script at the actor's event and play lip sync.
    However, the audio file and the lip-sync XML file were out of sync.
    Maybe it's because Slate has its own time management.
    Is there any way to improve this?
    Do you have any plans to support Slate in the future?
    I like Slate because it's easy to incorporate cinematic scenes.

    I love this asset,but I decided not to hire it for the project I'm currently working on.

    It will be adopted after the next work.
    We will continue to cooperate in testing and trial and error.
    Thank you.
     
    atomicjoe likes this.
  49. atomicjoe

    atomicjoe

    Joined:
    Apr 10, 2013
    Posts:
    1,869
    I'm using slate too. Seems like the best cinematics editor currently for Unity.
     
  50. Junya-Ogihara

    Junya-Ogihara

    Joined:
    Sep 20, 2017
    Posts:
    12
    Hello.
    Rogo Design's LipSync Pro has recently been hidden from the Asset Store.
    It seems that development was interrupted due to busy schedule.
    It seems that the replacement and support of the stable version will continue for a while, but it seems that the development of version 2 was abandoned halfway.
    Expectations are increasing for puppet faces in the future.
    Thank you.