[RELEASED] Puppet Face - All-In-1 Facial Animation for Unity

I bought this when it first came out and I finally have a project where I get to use it. But I just wanted to verify it was capable of doing what I need before I start. I just need an application that uses the webcam and tracks a person face. I realized that maybe this tool only did this in the editor and thought I would ask if it can do this in builds as well.

 

Is this in the demo scene? Could you try disabling and then reenabling the BlendShapeManager component?

 

Are you able to send me the BuyCookies.xml it created?

 

Sure.

 

I had a few difficulties with the blendshape editor.

Some notes or ideas:

• The tool values keep resetting. Values are the same per tool, it would be nice if they were separate.
• If the radius is ever 0 or -0 then the face usually breaks right away (typing in 0.02 requires typing in 0.0 first..)
• Mirror is misaligned in RP_Eric model
• Pressing Ctrl-Z for undo also triggers the use of the automatic modifier Ctrl, which is either Reset or Pull modes
• I would like more control on the brush, the fall off etc.
• The vert display mode is helpful but appears really bright/large, a different style would be helpful
• I'm accidentally moving the teeth, and it's really hard to target them to erase the changes to them. I suspect the skinned mesh rendedrer's physics collision is used for the brush targeting, making it hard to get inside the mouth

Here's it in action:

 

As reported before by others, the timeline function does not appear to work correctly. See linked video. https://drive.google.com/file/d/11q8xHmRPWjZXPN_sT3VBEiu73__qNuS_/view?usp=sharing. The lipsync clip plays at time zero, and then when it should as per the timeline - but it only plays the bone movements and not the blendshapes. Only using a demo scence. Unity 2020.3.3f1 HDRP on Windows 10.

 

Thanks for pointing these out, I'll take a look.

Did you know if you hold down the "B" button you can change the radius by moving the mouse left and right. I find it much more intuitive than setting it in the inspector.

Is he at 0 in the world x position?

ah - let me take a look.

yes this would be a good future feature.

ill have a think,

Did you know you can choose a submesh - this way you can exclude the teeth when making the blendshape

 

Ok this might be a Unity version issue - thanks for giving me the Unity version so I can look into it

 

Another thing to check. Use a Mecanim Humanoid character with an Animator component playing an Idle animation. I think the lipsync clip will not play whilst the Animator is enabled. Disable the component and the lipsync track plays albeit with the fault already reported.

 

@jamieniman below errors that I have on Unity version 2021.1.1f1 and 2019.4.20f1

 

@BarnimPL @jamieniman all reported errors related to incorrect format are due to a problem with parsing float values from XML files. You are using dot (".") to separate values (total value from a fraction) but in some regions a comma (",") is used instead. You just need a better method for parsing float values.

 

Thank you! That'll help me figure a solution

 

Hi, no markers are showing under wave.
Unity 2020.3.6f1


<?xml version="1.0" encoding="utf-8"?>
<rhubarbResult>
<metadata>
<soundFile>C:\Users\Delle\Spiel-Neue-Projekte\puppyFace2\Assets/PuppetFace/Demo/Audio/delle1.wav</soundFile>
<duration>5.30</duration>
</metadata>
<mouthCues>
<mouthCue start="0.00" end="0.74">X</mouthCue>
<mouthCue start="0.74" end="0.82">A</mouthCue>
<mouthCue start="0.82" end="0.90">C</mouthCue>
<mouthCue start="0.90" end="0.97">E</mouthCue>
<mouthCue start="0.97" end="1.11">F</mouthCue>
<mouthCue start="1.11" end="1.27">A</mouthCue>
<mouthCue start="1.27" end="1.55">B</mouthCue>
<mouthCue start="1.55" end="1.69">D</mouthCue>
<mouthCue start="1.69" end="2.38">B</mouthCue>
<mouthCue start="2.38" end="2.65">F</mouthCue>
<mouthCue start="2.65" end="2.69">C</mouthCue>
<mouthCue start="2.69" end="2.77">A</mouthCue>
<mouthCue start="2.77" end="2.94">F</mouthCue>
<mouthCue start="2.94" end="3.15">G</mouthCue>
<mouthCue start="3.15" end="3.57">F</mouthCue>
<mouthCue start="3.57" end="4.32">X</mouthCue>
<mouthCue start="4.32" end="4.37">B</mouthCue>
<mouthCue start="4.37" end="4.63">F</mouthCue>
<mouthCue start="4.63" end="4.70">G</mouthCue>
<mouthCue start="4.70" end="5.30">X</mouthCue>
</mouthCues>
</rhubarbResult>

 

Could you try this for me - open the xml in an editor and replace all . with ,
Then tell me if it works?

Next thing could you reset that file, and then try adding the following script to the project - I want to see if this works:

Code (CSharp):

1. using System.Globalization;
2. using System.Threading;
3. using UnityEngine;
4. #if UNITY_EDITOR
5. using UnityEditor;
6. #endif
7. #if UNITY_EDITOR
8. [InitializeOnLoad]
9. public static class FixCultureEditor
10. {
11.     static FixCultureEditor()
12.     {
13.         Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;
14.     }
15. }
16. #endif
17. public static class FixCultureRuntime
18. {
19.     [RuntimeInitializeOnLoadMethod(RuntimeInitializeLoadType.BeforeSceneLoad)]
20.     static void FixCulture()
21.     {
22.         Thread.CurrentThread.CurrentCulture = CultureInfo.InvariantCulture;
23.     }
24. }

 

I believe I have sorted this, if you email me at Puppet3Dunity@gmail.com I can send you the fix (the xml reader needed to be culture invariant)

 

Cool, comma (,) is working, so it's a German notation thing. r/Detlef

 

Hi Jamieniman, I placed the script in rhubarb folder and it seems to work. Old Man shows the markers without intervention.

thx/Detlef

 

Hello, first of all, great tool,
I have wave audio as byte[]
Can you provide an C# Code example of how your tool handles this kind of approach on runtime.
Can your tool perform lipsync animation if i download wave file on runtime

 

Hello,
I've got the following problem, when I try to use the animation clip capture with the PerformanceCapture component in a animator controller the application doesn't run out the rutine and, I've got the following warning:

"Binding warning: Some generic clip(s) animate transforms that are already bound by humanoid avatar. These transforms can only be changed by Humanoid clips."

I'm using and Humanoid rig so I don't understand why it doesn't work. Maybe the answer it is very obvious but i'm new in this field.
Thank you so much!!

 

Currently there are a few hoops to jump through to get humanoid and performance capture to work together:

1. import the fbx exporter package
2. right click on your char (with its anims on an animator) and export as an fbx (with model and anims)
3. Set this fbx to a humanoid (with the same avatar)
4. Duplicate all the animations and add them to your animator.
5. Copy all the blend shape keyframes across from the pre-exported anims to the exported ones.

It's not ideal. I want to make humanoid specific performance capture - its on my list.

 

At the moment the audio to lipsync conversion is editor time only.
If you take a look at the lipsync script at line around 490, the conversion uses an exe, and is currently wrapped in If UNITY_EDITOR defines.
I haven't looked into this much though, so perhaps its easy enough to make it work. Its on the the list of todos.

 

Could this handle doing simple transitions as well between mouth shapes with no blending in between? im looking for something with this level of flexibility but that can also do static quick changes.

 

Thank you!!

 

Hi,
I am new to Puppet Face. Using v1.4 today and I got some error messages:

Assembly 'Library/ScriptAssemblies/Unity.PlasticSCM.Editor.dll' will not be loaded due to errors:
Reference has errors 'unityplastic'.

Assembly 'Packages/com.unity.collab-proxy/Lib/Editor/PlasticSCM/unityplastic.dll' will not be loaded due to errors:
unityplastic references strong named System.Windows.Forms Assembly references: 4.0.0.0 Found in project: 2.0.0.0.
Assembly Version Validation can be disabled in Player Settings "Assembly Version Validation"

 

Perhaps its an assembly definition issue.
Could you try the following to see if it fixes it:
1. Go to the player settings and make sure "Allow 'unsafe' Code is ticked on. (Allows the use of pointers)

2. Now search for the file "OpenCvSharp" and delete just this single file.

Let me know if that fixes it?

 

Hi! I've got a problem in the latest version, when I try to use the Blendshape manager.
In the blenddshape manager when I try to create new blendshapes I don't get the circle that allows me to edit, it follows the normal mouse cursor, and even if I change pull/push/eraser, nothing, it doesn't change and it doesn't modify the mesh.

 

Make sure you have the gizmos button on (top right of the scene window)

 

I am interested in this application to create facial blendshapes on 3D Unity based characters that are rigged but lack facial rigging. Is facial rigging required to create facial blendshapes? Also, I noticed this tool adds several new components in the editor. Do any of those components need to stay on the character during export to an asset file (.Unity3d for example). I would use this for adding characters to an existing game, and that game just uses only built in Unity components. Thanks!

 

Its not required, you can make the blendshapes on any mesh or skinned mesh.

The lip sync component is required to drive the lip sync animation.

 

Perfect. Thanks for the info!

 

Hi. I've set up a lip sync with one clip - and it works very nicely ... I can edit and playback. Thanks!

However, when I try and add a second clip (i.e I add a second Lip Sync File (in Element 1, previous is Element 0) and a second Audio Clip (in Element 1), I get a stream of Errors either when playing it in the scene or when trying to do things with it in the Editor. I can see the new audio file in the editor and I can see the associated Lip Syncs marked up - but when I play the audio the errors appear and no synching takes place. It's not the specific files - if I swap the two elements around, the new first one will work and the new second one will return the errors - so it's something about having that second clip.

The errors all look like:

ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection.
Parameter name: index
System.ThrowHelper.ThrowArgumentOutOfRangeException (System.ExceptionArgument argument, System.ExceptionResource resource) (at <eae584ce26bc40229c1b1aa476bfa589>:0)
System.ThrowHelper.ThrowArgumentOutOfRangeException () (at <eae584ce26bc40229c1b1aa476bfa589>:0)
PuppetFace.LipSync.SetPhoneme (System.Int32 index, System.Single timeVal) (at Assets/PuppetFace/Scripts/LipSync.cs:468)
PuppetFace.LipSync.EvaluateLipSync (System.Int32 index) (at Assets/PuppetFace/Scripts/LipSync.cs:206)
PuppetFace.LipSync.LateUpdate () (at Assets/PuppetFace/Scripts/LipSync.cs:154)

I'm using Unity 2020.3.1f1 on a Mac.

Thanks for any help you can give!

 

This may have gotten lost in the many questions but i would still like to know. Is this tool capable of doing quick changes ala south park or pose to pose animation. I'm looking into audio generated lipsync without actually blending/interpolating from one shape to another. similar to stop motion even.

 

Just tried my same exercise as above using Unity 2019.4.23f1 and it worked fine - so it looks like an issue of Puppetface with Unity 2020.3.1f1.

 

could you try the following to see if it fixes it -
open LipSyncBehaviour.cs and comment out line 27, and replace line 35 with:

Code (CSharp):

1.  lipSync.SetPhoneme(input.LipSyncIndex, timeVal);
2.  

 

Yes! That seems to have worked. Thank you!

 

Sorry @jamieniman,
if this has been asked before, but the Blendshape Manager seems to have major issues with high poly models.
After I hit the "create new blend shape button" the model gets completely distorted.
I've tried all three TYPE-options ... any hint what I could do?
Thanks

 

For interest how many polys is it? Are you able to filter the sub-mesh to what you need to sculpt?
What kind of distortion happens?

 

I've tried with different single meshes and everything above roughly 10k up to 60k had single vertices or groups of vertices pushed way out along different axis before I even edited the mesh.
How do I filter the sub mesh? Did not find anything about that in the manual?

 

If you've got multiple materials on your mesh you can filter these:


I haven't experienced the error you're reporting - are you able to email me (puppet3dunity@gmail.com) an example that breaks, and let me know which version of Unity you are using?

 

Thanks, the problem occurs even with one mesh and one material (so I can't select anything else in the Sub Mesh dropdown).
Unity Version is 2019.4.18f1 (and 2019.4.21f1)

Mail with example should be in your inbox.

 

Thank you @jamieniman for your fast support!

Just to clear up matters for anyone interested here on the forum:

Blend Shape Manager can currently handle objects up to 120k tris (which is quite reasonable).
- be aware, numbers in my above post were vertices, thus in some cases having significantly more triangles than 120k ;-)

I just use meshes with higher resolution for up-close hero shots for cinematics and thus went over that limit.

So, for 'normal' game assets all is well and I've tested it with several different objects.
Cheers!

 

You product looks interesting and fun!

Anyway, I've got a bunch of questions relating to your product and particularly in terms of requirements.

In the description (of Puppet Face at the Asset Store) the last line says:

(Requires Windows for Lip Sync Converter)

What is Lip Sync Converter? Who makes it? Where do I get it?

What does this line refer to, the whole product or an aspect?

If just an aspect, does this product work with Mac or additional benefit with Windows?

Thank you,
Mike

 

Hi Mike, good question -
There is a program called Rhubarb that Puppet Face runs that does the initial conversion of the wav files to lipsync data. There is a Mac version on their site which you run manually in a terminal. What the "Require's Windows" was saying is with Windows you don't need to do this manually - Puppet Face has a button to do it in Unity. (Which just runs the terminal command under the hood).
Apart from this inconvenience everything else works on Mac.

 

Not fix and keep spawn the same error message.
Assembly 'Library/ScriptAssemblies/Unity.PlasticSCM.Editor.dll' will not be loaded due to errors:
Reference has errors 'unityplastic'.
Assembly 'Packages/com.unity.collab-proxy/Lib/Editor/PlasticSCM/unityplastic.dll' will not be loaded due to errors:
unityplastic references strong named System.Windows.Forms Assembly references: 4.0.0.0 Found in project: 2.0.0.0.
Assembly Version Validation can be disabled in Player Settings "Assembly Version Validation"

 

Are you sure this is a Puppet Face caused error? If you delete Puppet Face from the project does it go away?

 

I have opened a new project (build-in 3D) and only imported Puppet Face. Then I run your demo, the error msg pop in the console. As You said, after I deleted the Puppet face folder, no error msg anymore.

BTW, I am using the latest unity version 2020.3.12f1.

 

I'm not getting the error.
It's something to do with plastic (a Unity package - not sure what it does)
The error suggest you disable Assembly Validation in the player settings:



Does that fix it?

 

When I bring in audio and click reload, it produces a lot of phonemes on the timeline. Is there a way to clear them all at once instead of deleting them one by one?

Thank you.

 

Open up the XML file it generated and delete all the lines that look like this:

Code (CSharp):

1.     <mouthCue start="0.00" end="0.04">X</mouthCue>
2.  

then click reload an it should remove them all

 

Thanks for the quick reply.

Now I'm having trouble getting the Lip Sync Track to work in Timeline. I placed it but when I scrub the TL it won't play even when I change the LipSync Index.

What am i missing?