[RELEASED] Puppet Face - All-In-1 Facial Animation for Unity

Can you confirm for me your using Puppet Face v1.4.1 ?

Also does your animation file have any keyframes on the face bones/ blendshapes?

 

Hi i'm trying to add new blend shape and it's not working, if i'm wrong somewhere?
https://drive.google.com/file/d/1Ep-jLjxzbDp0JtNX3VsXoycwG3S2sRV6/view?usp=sharing - video

 

Ok, I haven't had seen that happen - what version of Unity are you using?

 

NVM Got it to work. Thanks

 

Hi. On 2020.3.1f1 on a Mac, when I edit the placement, range or assigned mouth shape in the Lip Sync Editor, it works while I play the clip in the Lip Sync Editor but resets back to what I had before when I e.g. play the Scene. Is there some way to save these changes - or does that not work on a Mac because Rhubarb isn't integrated? (Note, Editing the Bone Poses themselves works fine).

The documentation mentions save and reload buttons - but I don't see those. Just the 'Convert Audio Files' button at the top right of the pane.

Thanks!

 

You should see a save button. If not expand the window really wide (I think there is a dual monitor bug on mac where this centers to the wrong width)

 

Thanks - but unfortunately, I don't see that save button. Just the 'Convert Audio Files' button always at the very far right side of the pane, no matter how wide I make the pane (up to the size of the screen). I'm using only one screen on a laptop.

 

can you try this:
open PuppetFaceEditor.cs and comment out lines 153 and 185

Code (CSharp):

1. GUILayout.FlexibleSpace();

change to

Code (CSharp):

1. //GUILayout.FlexibleSpace();

 

That seems to make half the next (maybe reload) button visible at all widths. Here's a screenshot at maximum width after making those changes:

 

FYI - you may already know - it seems to be the Screen.width that's returning the wrong value. If I change that enclosingBeginArea from:
GUILayout.BeginArea(new Rect(10, 10, Screen.width - 20, 70), EditorStyles.helpBox);
To:
GUILayout.BeginArea(new Rect(10, 10, 800 - 20, 70), EditorStyles.helpBox);

And restore those FlexibleSpace() calls ...Then I see all the buttons in a row - in a fixed-width Area of 800 width like this:

 

hey there Jamie, on a Mac (High sierra 10.13.6) and definitely seem to be experiencing issues with the LipSync. saw the post about downloading the Mac OS CLI version of rhubarb so DLd it and ran it, which did seem to process but appears to have created a blank XML file. also the Lip Sync component has no awareness of the update XML file added, so while the audio file can be loaded visually now, there's no phoneme data. more importantly, adding phonemes manually does work, but saving doesn't seem to, so when you open things again, everything's blank. i did sort of try manually adding associations with the Text asset and the audio file but that didn't seem to get anywhere.

so basically i'm dead in the water. on the good side the tool handles Blendshape creation very well for the most part, though renaming them easily would be nice. would also definitely like the ability to add more phonemes or blendshapes other than the default ones. sort of surprising its missing the s,z phonemes for example. anyway that's my observations for the moment.

 

for interest what command did you run?

Are you clicking the "save" button?

Adding new blendshapes is on my feature list

 

I looked through these forums and watched the tutorial video, but couldnt find an answer to a question I have about puppet face. I see the tool is mostly aimed towards lip sync and animations, but what about character creation blend shapes commonly found in games when creating a character? Am I able to create different blend shapes (nose size, lip size, cheek size, jaw width, etc) and export these blendshapes? Are the blendshapes being "baked" into a new fbx file that you could later use somewhere else without the need for the puppet face scripts? (moving the sliders in the skinned mesh renderer for the blend shapes youve created with puppet face)

 

Hey Jamie, any progress on getting the performance capture to work properly on a macbook pro?
Also we were talking about the possibility of loading pre-recorded video files to capture.

 

Hi you can make these blend shapes and then use Unity's FBX Exporter to export your character with the blendshapes to use outside of Unity.

 

Is there documentation on which phenomes are needed for face capture? I need to send a list of phenomes created as blendshapes to the character artist. )I am asking all the phenomes that would make the capture good, not just the bare limited/required ones -- in case there is a distinction) @jamieniman

 

could load the video into something obs's virtual camera to play it back for puppetface to record later, would have to have the performer do the poses to configure it before they start then you would only need a camera for capture.

 

Hi, just saw your review - my apologies for not replying to this. I moved house in September - and must have seen the message and planned on replying but completely forgot (then without new messages on the forum I didnt get a reminder to reply). Ordinarily I try to be quick with support. I'm very sorry.
I saw in your review you had other issues that hopefully I can help with.
Starting with the error you are getting:

Do you get a line number in the error? What version of Unity were you using?

 

Thanks for returning back, I would love to correct my review once the problems I got resolved.
So as a response, I am using 2021.2 latest beta. As the whole error, this is what I get:

Code (CSharp):

1. ArgumentException: Value must be a Com object.
2. Parameter name: o
3. System.Runtime.InteropServices.Marshal.ReleaseComObject (System.Object o) (at <e03f4e9558c44d56aad5167e5810a238>:0)
4. System.Runtime.InteropServices.Marshal.FinalReleaseComObject (System.Object o) (at <e03f4e9558c44d56aad5167e5810a238>:0)
5. Ookii.Dialogs.VistaFileDialog.RunFileDialog (System.IntPtr hwndOwner) (at <e3ade0b9a63043c89b5fbc325cc98b60>:0)
6. Ookii.Dialogs.VistaFileDialog.RunDialog (System.IntPtr hwndOwner) (at <e3ade0b9a63043c89b5fbc325cc98b60>:0)
7. System.Windows.Forms.CommonDialog.ShowDialog (System.Windows.Forms.IWin32Window owner) (at <1d2e5fd5dacd4b00b29699699894aec6>:0)
8. (wrapper remoting-invoke-with-check) System.Windows.Forms.CommonDialog.ShowDialog(System.Windows.Forms.IWin32Window)
9. SFB.StandaloneFileBrowserWindows.OpenFilePanel (System.String title, System.String directory, SFB.ExtensionFilter[] extensions, System.Boolean multiselect) (at Assets/PuppetFace/Tools/Editor/StandaloneFileBrowser/StandaloneFileBrowserWindows.cs:38)
10. SFB.StandaloneFileBrowser.OpenFilePanel (System.String title, System.String directory, SFB.ExtensionFilter[] extensions, System.Boolean multiselect) (at Assets/PuppetFace/Tools/Editor/StandaloneFileBrowser/StandaloneFileBrowser.cs:51)
11. PuppetFace.PuppetFaceEditor.ConvertAudioToLipSync () (at Assets/PuppetFace/Scripts/Editor/PuppetFaceEditor.cs:475)
12. PuppetFace.PuppetFaceEditor.OnGUI () (at Assets/PuppetFace/Scripts/Editor/PuppetFaceEditor.cs:171)

 

Ah - you're using the beta Unity, We support only actual full releases of Unity (tend to find Unity breaks things and then puts them back on final release). In this case the issue looks like it could be with a 3rd party asset browser we use called the StandaloneFileBrowser. I think I could make a quick fix for that. If you email me at puppet3Dunity@gmail.com I can email it to you.

 

Hi Jamie, I just updated Puppet Face to v1.4.2 and am encountering this console error as well.

"Assembly 'Packages/com.unity.collab-proxy/Lib/Editor/PlasticSCM/unityplastic.dll' will not be loaded due to errors:
unityplastic references strong named System.Windows.Forms Assembly references: 4.0.0.0 Found in project: 2.0.0.0.
Assembly Version Validation can be disabled in Player Settings "Assembly Version Validation"

Enabled run unsafe code and deleted the single file "OpenCvSharp" as suggested, which does not fix the issue. I'm running Unity v2020.3.16f1 on Mac. Up until this new update, there were no Puppet Face console errors. Any suggestions?

 

So I just discovered what is causing this - its the StandaloneFileBrowser that Puppet Face uses. For now can you delete this whole folder from the project and in the PuppetFaceEditor.cs replace the following:

Code (CSharp):

1. var extensions = new[] {
2.             new ExtensionFilter("Sound Files", "mp3", "wav" ),
3.             };
4.             string[] files = StandaloneFileBrowser.OpenFilePanel("Select Wav Files", "", extensions, true);
5.             string path = EditorUtility.OpenFilePanel("Select Wav File", "", "wav");
6.             if (files.Length != 0)
7.             {
8.                 foreach (string path in files)
9.                 {

with :

Code (CSharp):

1. string path = EditorUtility.OpenFilePanel("Select Wav File", "", "wav");
2.             {
3.                 {

Also comment out this line:

Code (CSharp):

1. //using SFB;

 

Thank you Jamie for the guidance. That appears to have suppressed the issue.

Question regarding this fix, the code editor suggests there's a syntax error possibly (I'm no programmer). See image, anything I need to do?

Last, one suggestion regarding my setup that may play a part...I don't know. I've setup a separate project with only a few assets installed among them are puppet face and puppet 3D and some others. However, my main project that produced the console errors upon updating Puppet Face actively utilizes FMOD. Since the code you had me remove appears to pertain to sound, could there be some interplay between Puppet Face and FMOD causing this issue? Just a suggestion. Thanks, Tom!

 

This is such a great tool.

Any chance at adding emotion support? -- Also how about cartoon characters that have/use 2D textures for parts of their faces? -- So far, I've not seen a solution for animating textures for mouth/eye shapes, but I think this system would be great for that too, assuming emotion support was added. D:

 

Hi Jamie, great package.
We have just bought it from the Asset Store.
We are wondering if you plan to add more blend shapes mapping for Performance Capture because OpenMouth, O-mouth and smile are too few to make facial expression.

How hard would it be to modify performancecapture.cs to implement other face points to work together with other blend shapes, just like Lip Synch does and more ?

Thank you in advance

 

I'm working on supporting emotions for the next update (as a start it will allow you to add custom blendshapes to the lipsync animation)

 

For clarification, can I drive facial animation in real-time at runtime? This would be awesome for "avatar interviews" and podcasts.

 

It's not extremely difficult, just a bit of a manual process at the moment (adding all the variables, hooking them up etc).
I'm currently working on the lipSync feature of PuppetFace. My next plan is to revisit the Performance Capture, probably I want to first integrate MediaPipe for improved tracking, and next would be to expose more feature tracking than it does currently.

 

Grab the latest version of Puppet Face (v1.5.0)- you can now add other blendshapes to lipsync animation

 

Nice. Thanks!

Just a quick question -- I didn't notice at a glance, but is it possible to overlay multiple blendshapes at once on the face for emotions (i.e. angry eyes blend, but smirking mouth blend)?

Btw, this would be a HUGE improvement. This would be what most users expect IMO. Please add support for this, if possible, asap.

 

yup it would

noted

 

Hello. I'm interested in the asset but pretty much all of my animations are humanoid and have no expressions when moving about. This seems like a good tool to add those animations/expressions in by motion capture or hand, so I was wondering if humanoid support was ever added? I did a search to find out and came across this post back in may. I'd like to avoid blending the mesh as much as possible since my 3d animation skills are, lets say rudimentary at best, so there might be something I'm missing since I'm more of a 2d guy.

Edit: Actually, turns out I got this asset already. Must have been from some humble bundle. Neat! Still wondering if adding facial animations to humanoid animations is possible though.

 

Hey.

I saw that automatic eye movement does not work with the character laying on the side, as the rotation is still horizontal. It would make sense to save local and not global transforms here.
But mabe I'm doing sth. wrong.

Keep up the good work,
Andre

 

This could be something I've overlooked.
Trying it on the demo character I get this, which I can see isn't perfect:



Thanks for pointing it out to me, I'll take a look into it

 

The Lipsync feature is fine with humanoid.
The Performance capture feature however requires a few more steps to get working with humanoids. Here's what to do:
The performance capture controls the head animation - to get this to work on humanoid rigs you will need to export all the animations that get generated as fbx files. You can do this using Unity's FBXexporter.
Hopefully I can make this simpler in future, but this is the way around it for now.

 

Can existing blend shapes in a model (from Character Creator or Ready Player Me for example) be connected to Puppet Face Blend Shapes in BlendShapeManager, then refined?
or is it assumed perhaps that blendshapes are always made from a neutral face?

 

I notice that my model's head tracking is reversed. If I pivot my head upwards, the model looks down and vice versa.
Can this be reversed?

 

I haven't got experience with those assets, but as long as the skinned mesh they create is fixed, you can edit it with the BlendshapeManager. (By fixed I mean there isn't an editor script that can dynamically change the Mesh that is generated and assigned to the SkinnedMeshRenderer)

 

Try putting negative numbers in the Rotation Amount:

 

I hope that eye tracking and blinks become possible with Puppet Face. This software is truly enjoyable to use!

 

1) Noob question: can I reset a blend shape in lipsync editor ? Not in mesh modifier using erase tool. TNX !

2) A request: can you change the color of name of blendshape in blenshape manager from white to a dark color ? White on grey is hard to read. TNX

3) Can you implement a sort of "slow motion" in lipsync editor, setting tiimescale and picth of audiosource ?

4) Building the project i have this error:
"Assets\PuppetFace\Scripts\AfterAllPostprocessor.cs(7,35): error CS0246: The type or namespace name 'AssetPostprocessor' could not be found (are you missing a using directive or an assembly reference?)"
I suppose this script should not be included in building, you should move it into editor folder.



2 done myself:
file BlendShapeManagerEditor.cs
guistyleLabel.normal.textColor = Color.black; //white;

 

Apologies not sure what you are asking here?

Oh, cool I'll add that to the features list

Thanks so much for pointing this out - this was an error in the previous update. I've fixed it in v1.5.1 now

 

I'd like to reset the blendshape to "new" state. Or copy it from REST blendshape.

Another question: it's possibile to add bones to lipsync face bones array AFTER editing ? The button "edit bone pose" shows only the first bone in array, in the inspector.

TNX for support !

 

@jamieniman Hi. Have you looked further into UMA integration?

 

You make a new blendshape with the same name and setting it will overwrite the old one.

Just click "Edit Bone Pose" on the Rest pose (after assigning the new bones). You will then need to resetup all the bone poses for the rest of the phonemes.

 

Not had a chance I'm afraid - other features and fixes have been higher on my todo list
I havent had any experience with UMA - what features does PuppetFace need to allow it to work with it?

 

I think i found a bug or i don't understand something i am using unity 2021.2.8f1 on win10
The performance capture options show when i have a virtual camera selected but when i choose a physical camera everything from the skin selector down to the capture button completely disappears. it does show the video on the plane in play mode and it does track but is unusable as it can't be calibrated.
-+-+-
also the manual could be more fleshed out on playing multiple lipsyncs and from an index and could use a demo scene with multiple lipsyncs hooked up to a button

 

I don’t know, yet, not having purchased PF (all my characters are UMA and I will pick it up if it works with them). UMA characters have their own way of managing blend shapes, so there may need to be changes to support that. I have LipSync, and it provides a UMA component to control facial expressions…

 

Hello! I was almost buying the package, but I read that some parts of it are only compatible with windows. I only run on linux, so what can I do and what cannot I do with Puppet face?

 

It uses Rhubarb to convert the lip sync. (it runs and exe which is why windows only) But this can be done manually and then loaded into Unity using PuppetFace (You will just need to manually assign the html file to the lipSync component)
Here is link to the linux version of Rhubarb - it will need to run in command line
https://github.com/DanielSWolf/rhub...oad/v1.11.0/Rhubarb-Lip-Sync-1.11.0-Linux.zip