[RELEASED] Puppet Face - All-In-1 Facial Animation for Unity

Introducing Puppet Face - All-In-1 Facial Animation for Unity!

Features included:

Blend Shape Sculpting

• Create new blend shapes & edit old ones.
• A sculpting tool that respects topology (eg. you can move your top lip without affecting the bottom lip!).
• Using the Blend Shape Manager you can see little snaps of your blend shapes.

Audio to Lip Sync

• Convert audio files to lip sync animation
• Simple and quick setup.
• Phoneme poses can be made from blend shapes, bones or a mix of the two.
• Audio scrubbing makes it easy to edit and refine your phoneme animation for added perfectionism.

Web Cam Performance Capture

• Use your web cam to use your face and head motion to control your character.
• Record your performance with audio.
  
• Simple and quick setup

Eye Motion

• Automate eye motion and blinking.
• Eye lids adjust with up and down, to give added realism to your eyes.

Find it here:
https://assetstore.unity.com/packages/slug/181312 .

Any questions welcome!

 

Does the lipsync support other languages than English?

 

What would be the best webcam to use for motion capture?
(min cam resolution and fps)

 

Good questions!
The speech recognition is setup to be phonetic so it works on other languages.
I don't have exact specs for web cam. My one is a low end 1080 if that helps

 

I currently use Salsa Lipsync Suite and I'm pretty happy with it.
It gives really great results and matches any audio in realtime, so it's great for multi-language setups, since you don't have to prerecord the lipsync animation and will match any language. (although it's true that your plugin should match phonemes more accurately)
But I like your plugin very much for the blendshape sculptor and webcam motion capture for the head animation.
I would like to use it along with Salsa but I don't know how the two would work together.
It would be nice to have an integration path for the two, since Salsa seems to be the standard right now and it's honestly working flawlessly for lipsync alone right now.
Something like using the webcam motion capture to record the movements of the head + triggering emotion blendshapes.

Edit: Does the webcam capture record the movement of each eyebrow independently or does it have the ability to recognize custom expressions to blend-in custom blendshapes? (like "happy", "sad", "surprised"... and blend them in-between)

 

I don’t have much need for the facial features right now but the blend shape sculpting is very intriguing. If it works the way I assume it does, it would be a huge time saver for tweaking existing morphs and creating new one without having to switch out of the editor. I’m actually surprised I haven’t seen a tool like that for Unity before. There are a lot of blend shape mixers and controllers but nothing for sculpting per se.

If you are planning to create any new videos, I’d like to see a demonstration of the blend shape sculpting and manager specifically. I understand the asset is new and has just been published, but more examples or documentation would be nice so I can see how this could work within my pipeline before purchasing it. (Also, you might want to create a link to this forum in the description so we don’t have to hunt it down.)

 

Puppet Face can be used to support other assets - so you could use it to improve your blend shapes using the sculpting feature, or you could use Puppet Face's lip sync for your hero/ cutscene where you want the quality of the facial animation to be much better. You can use the performance capture to animate the head rotation, and it currently registers smiles so you can use it to animate the smile blend shape and use it in conjunction with the other lipsync. I'm hoping to track more expressions as I further develop this feature.

 

Agreed, I have actually submitted an update with links in the description, just waiting for Unity to make it live.
I should be making more tutorials going forward

 

After watching the tutorial video a few times I think I'll go and give this a try now. Might even be able to use the head tracking and eye motion features to breathe extra life into my current assets.

 

I have some 3d models that were created without blendshapes as we weren't sure we were going to need facial animation. Now that the project will require the characters to talk, we need to add blendshapes and are currently looking for someone to create them. I saw this tool and wondered if this would indeed allow us to take a character and create new blendshapes when none were created originally. The characters are realistic humanoids and your video shows a cartoony styled character so I wondered how well this worked on other styles.

 

Yup, thats what puppet face is for. Can be used on any art style. Just check the mouth edges are open (not welded shut)

 

I'm currently looking into why for some people the InitialPuppetFaceLaunchEditor isn't running on import (this will cause it to error).
Its easy to fix by just re-importing this script. If you've experienced this please get into contact - I'm trying to figure out what the cause is so I can fix it in the first place.

 

Hi there
Is it possible to have eyebrow movements in performance capture?
I mean more control over face using this system
Maybe some emotion presets?
Angry
Happy
Sad
Etc?

 

Yup, I've added these to my trello feature list.

 

Will that expressions be able to mix smoothly or will it be more like a trigger? (ON/OFF each)

 

Hopefully smooth, it should work like smile does currently (this is the happy expression)

 

Is the webcam feature tracking a module that can be used in-game, or only in editor?

 

It works in-game, though it's been designed to be for capturing specifically.

 

What would be better for PuppetFace webcam motion capture?
1- 1920x1080 at 30fps
or
2- 640x480 at 60fps

 

I would go for the 1080p I think

 

I got this wrong earlier - the released version works phonetically so it works in other languages as well

 

I've update Puppet Face to v1.1, it now includes the brownie girl demo, and a couple of fixes. Happy new year!

 

Hello. I like this asset very much. But there was a serious bug.
It works fine until I load the second WAV file.
Rhubarb's analysis information is correct when the third WAV file is registered. However, the WAV file referenced by Puppet Face's lip sync has become the second WAV file.
After that, all refer to the previous WAV file. My mouth doesn't move. Please check and correct.

 

I don't know if it's the direct cause, but I also got an error when importing and it didn't work.
When I turned on "Project Settings> Player> Allow'unsafe' Code", all the errors disappeared and the asset worked.

 

Hello.
Updated to the latest Version 1.1.
The above problem has been resolved.
Thank you.

 

Hi jamieniman :

Please Help ! but I also got an error !
I Version Unity 2019.4.0f1


System.IO.IOException: Failed to Move File / Directory from 'Assets/PuppetFace/Gizmos' to 'Assets/Gizmos'.
at UnityEditor.FileUtil.MoveFileOrDirectory (System.String source, System.String dest) [0x00041] in <af78ad00a40d4f61b82afc6cafb0a416>:0
at PuppetFace.InitialPuppetFaceLaunchEditor..cctor () [0x00013] in F:\VT\New Unity Project TB\Assets\PuppetFace\Scripts\Editor\InitialPuppetFaceLaunchEditor.cs:15
UnityEditor.EditorAssembliesrocessInitializeOnLoadAttributes(Type[])

Thank You !

 

Hi, if you look in the inspector, do the lip sync files and wav files correspond correctly? (If not - quick fix would be to se tthem correctly there). Also dont forget to hit "reload" after you convert a file.
Is it something you can find steps to reproduce? That would be really helpful for me.
(By the way thank you for the review - I'm really pleased it works well in Japanese ;-) )

 

Aha! This error is what must have been causing the above issue I mentioned. Glad i separated it out in v1.1. There's a very easy work around, please move the gizmo folder into the assets folder directly.

 

Thank you. It works fine.

Is it possible to use a lip sync file for the timeline?
I read the manual but didn't know how to use it except while playing the game.
I want to work with lip sync files on the timeline and other cinematic sequencers.
Please tell me how.
Thank you.

 

That's a must have feature, it's on the top of my list.
In the meantime you need to have a script on a game object that plays the lip sync which you enable when it needs to play.

 

Thank you for the reply.
I'm looking forward to the version upgrade.
Thank you.

 

Hi have you tested this with Reallusion Character Creator 3? I recently got it and would love to see the results using their characters and your asset.

 

Looks like a nice asset - Puppet Face works with any character as long as the mouth isn't sewed shut if (it should at least be possible to move the upper and lower lip apart) It would be extremely shocking if Reallusion had glued shut mouths!!
(Also just check if the character has eye bones - if not, I have a rigging tool I'd recommend Puppet3D )

 

Sounds great. I'll be picking this tool up later this week.

 

Hello.

I created a new project for a simple test.
Added Puppet Face (version 1.1) to the clean project.
I get an error message every time I run it after importing.


The message is as follows.

System.IO.IOException: Failed to Move File / Directory from'Assets / PuppetFace / Gizmos' to'Assets / Gizmos'.
at UnityEditor.FileUtil.MoveFileOrDirectory (System.String source, System.String dest) [0x00041] in <2f1c602eae0d45c293fff3e3aef759fa>: 0
at PuppetFace.InitialPuppetFaceLaunchEditor..cctor () [0x00013] in C: \ Unity_Develop \ Slate_Lip_Test \ Assets \ PuppetFace \ Scripts \ Editor \ InitialPuppetFaceLaunchEditor.cs: 15
UnityEditor.EditorAssemblies: ProcessInitializeOnLoadAttributes (Type [])


The project under construction has deleted version 1.0 assets and imported version 1.1, so it seems that no error has occurred.
Please confirm.

 

Yes, the gizmo folder is missing in 1.1 by mistake. All it means is there wont be little icons on the components for PuppetFace, and there is the error on the first load of the project.
For now you can copy that Gizmo folder over from the other project, (or just delete the InitialPuppetFaceLaunchEditor.cs)

 

Thank you for confirmation.
OK. I was relieved to find that there was no problem with the system.
Thank you.

 

Is it possible to animate Mixamo characters with this?

 

Yup!

 

Hi I purchased your asset today and imported it into a clean project. Upon installation it gives me several error messages saying to "Enable Unsafe Code" through the player tab which I have done and that gets rid of the error messages. Is that intended?

Also when trying to use Capture mode I'm getting the error message

Could not connect pins - RenderStream()
UnityEngine.WebCamTexturelay()
OpenCvSharp.Demo.WebCamera:set_DeviceName(String) (at Assets/PuppetFace/Tools/OpenCV+Unity/Demo/Scripts/WebCamera.cs:71)
OpenCvSharp.Demo.WebCamera:Awake() (at Assets/PuppetFace/Tools/OpenCV+Unity/Demo/Scripts/WebCamera.cs:114)
PuppetFace.PerformanceCapture:Awake() (at Assets/PuppetFace/Scripts/PerformanceCapture.cs:100)

 

I've just submitted v1.2 with a fix for the first issue (It was meant to change that setting automatically - so what you did was correct)

The second one will have something to do with your webcam setup. Do you have a webcam connected?

(Also v1.2 now has a dropdown where you can choose the device. If that device is being used by another application you will need to shut that application. For example if you use OBS Studio - if you are using a video capture source this can take control of your webcam and so stop Unity from being able to use it.)

 

I've made a specific LipSync Track for Timeline for v1.2 (just been submitted to the asset store)
Keep an eye out for the update when it comes

 

Hello. Thank you very much.
I'm looking forward to it.
I'm currently scripting and using it to work on the cinematic sequencer Slate. The test went well.
I didn't like LipSync Pro. I was very happy when I found this asset.
Thank you.

 

Hello.
I have a new request.
I would like a button to add a new blend shape to the state created by manipulating some of the blend shape bars in the blend shape manager.
Please consider it.
Thank you.

 

That's a nice idea, I'll add it to the list

 

Hi Man I'd like to try this asset before to buyit! :s it looks like very good

 

Puppet v1.2 is out now!
It has a lip sync track for Unity Timeline:



Changes:

• New - Lip Sync Timeline Track
• New - Eyebrows performance tracking
• New - Select desired Webcam
• Fix - Added Assembly Definition
• Fix - Gizmo folder back

 

Hello. I tested the Lip Sync Timeline Track.

No matter where you place the lip-sync clip on the track, the starting point will be 0 time.
And the lip sync index is called 0 no matter what number you enter.
I tried arranging two lip sync clips on a track, but the second lip motion didn't play.

I used Slate to create a sequence.
I was able to call my own script at the actor's event and play lip sync.
However, the audio file and the lip-sync XML file were out of sync.
Maybe it's because Slate has its own time management.
Is there any way to improve this?
Do you have any plans to support Slate in the future?
I like Slate because it's easy to incorporate cinematic scenes.

I love this asset,but I decided not to hire it for the project I'm currently working on.

It will be adopted after the next work.
We will continue to cooperate in testing and trial and error.
Thank you.

 

I'm using slate too. Seems like the best cinematics editor currently for Unity.

 

Hello.
Rogo Design's LipSync Pro has recently been hidden from the Asset Store.
It seems that development was interrupted due to busy schedule.
It seems that the replacement and support of the stable version will continue for a while, but it seems that the development of version 2 was abandoned halfway.
Expectations are increasing for puppet faces in the future.
Thank you.