Facial animation for Unity 5

Hello,

i planning to do cut scenes, where i need facial animations. I would be happy to have a similar solution like it is in Mafia 2. All tools i found are not really good or suitable for indie developer:

1. FaceFX
For 900$ too expensive for indie developers, but looks good.

2. Maximo
Has a facial animation, but i guess it can just be captured by camera and cannot be configured with just an audio file as it is possible in FaceFX and Poser.

3. Poser Pro Dev.
This asset is often discounted, but has lot of bad reviews. Can be bought often from 150$ to 2xx$. Now the price for Unity 11 user is 225$.

I would like to know more about these tool, some experiences would be very helpful. If someone has another facial solution, which can be configured by an audio file, or maybe for a certain language with predefined speaker, where i just have to write in text what they should say, please post it in this forum.

Thank you in advance!

Edited: The unity asset store contains two inside Unity lip sync in assets:

- lipsync
Has just tutorial videos, he never started the animation ... Looks like it is a more manually configurable facial animation tool, which could be sometimes better than a bad automated one.

- SALSA
Automated facial animation and lip sync. The advantage is, it is in Unity, no need for time killing conversion in another tool and back. In my case it would hurt more, because i use the Blender files and let Unity handle them, convert them in background to FBX.

What tool would you use as indie developer?

 

There's also Cheshire, which is free.

As an indie developer, I would use SALSA for lipsync that doesn't involve high-detail extreme closeups. SALSA does lipsync approximation, which isn't quite as accurate as true phoneme-based lipsync matching. But the workflow is the absolute easiest, and the support is excellent. I believe they have tutorials for Blender.

For high-detail closeups, I would use either FaceFX (gotta bite the bullet for high quality) or Mixamo FacePlus. The cost savings of using FacePlus would be offset by having to hire face actors to record on a webcam. Mixamo as a company is in transition right now, since they were just acquired by Adobe. I'm not sure what this means for FacePlus.

 

To expand on what @TonyLi said regarding FacePlus, the capability is currently only compatible with Unity 4.x and further development is on hold. However, I have had success creating animations in a Unity 4.6 project and simply importing the .anim files into a Unity 5 project.

 

At the time i develop the game alone, and i plan to assign a voice acting studio at the end, if i find one for indie developer. So, all what i can do now is writing text, which later then should be spoken. And for this reason FacePlus (cannot make lipsync and facial animation from a sound clip) and FaceFX (too expensive) are out and from the Poser (bad rating, no unity forum) package i would just need the facial animation + lipsync. Poser has a bad rating, they didn't really care about Unity i guess, so they are out too.

SALSA looks exactly what i need, has a good rating and has a Unity forum, but has just few ratings. LipSync in Unity asset store looks in its way good too, sometimes a non automated facial animation assignment is better. I guess, i will buy both, if the time comes to make my cut scenes.

There is a very small chance, that i am not satisfied with all these tools and i end up with FaceFX, if my game is ready for launch. But now, even if FaceFX would have a 100$ indie version, i would at first give a try for SALSA and LipSync, because they are inside of unity.

 

Sounds good working with SALSA. But don't forget, by FaceFX is the workflow much more longer, because it is an external tool. I would like the quality like in Mafia II, which was made by FaceFX... But the game is old, and i hope SALSA can provide the quality. FacePlus doesn't suite in my workflow, but maybe when it comes to cut scenes, it will.

 

SALSA doesn't have the same quality as FaceFX, but it's still pretty good. On the other hand, SALSA's advantages are:

1. It's integrated in Unity.
2. You don't have to preprocess audio clips to generate animation files like you do in FaceFX.
3. At runtime, you don't have to tie animation clips to audio clips. SALSA generates the animation on the fly based on the audio clip.
4. It's less expensive.

If the camera is relatively far -- for example, showing the character's entire head and torso -- you might not be able to tell a difference in quality between SALSA and FaceFX, especially if you distract the player with full body motion and camera movement at the same time. But if you're doing a closeup on the character's face, you'll tell a difference. It's up to you to decide whether it's good enough or not. After all, FaceFX isn't perfectly lifelike either; it's just somewhat more accurate.

 

That's not entirely true.

@TonyLi Thanks for the quick shout-out! Always appreciate anyone throwing a little attention Cheshire's way.

And on that note, one of the big draws of Cheshire is that it allows you to take an audio file, feed it through the program, and get a fully-realized Unity animation file with the speech in the audio file converted into animation keyframes. Now, it does not do facial animation along with the lip-sync animation. Your character's face will move with the words, but the rest of the face will be static unless you make additional animations yourself. But the dream of converting a standard sound-file into a full-on lip-sync animation is what Cheshire is all about.

The currently available version of Cheshire uses an application that you have to download and use outside of Unity. (also free) The next big upgrade to Cheshire that I'm working on right now bundles the necessary application with the plugin, and will run that program behind-the-scenes right through the Unity interface. So you'll be able to rip the lip-sync timings for your animation without ever leaving Unity. (to improve workflow)

 

Have you tried NaturalFront Unity 3D face animation plugin?

The tutorial is very impressive.

I have tried their Free-version, 3D facial animation and talking heads are made much simpler than the other solutions suggested here.

The 3D animation created by the software is really good. Best of all, the modeling and animation are made really easy, easier than all the other solutions suggested here.

See for yourself if it is what you need.

 

The NaturalFront plugin is quite nifty. However, it is also a bit more limited than some of the other solutions. It seems that it is designed to be used in conjunction with pre-built 3D models that have already been constructed and had blend shapes added to them. This allows for a faster/easier workflow, as no actual content needs to be created by the user. But it limits what the user can do with the plugin, as the user is tied down to the pre-fabricated 3D models on offer. It is nice that you have the option of importing a picture to use as a UV map, and that there are handles for altering the imported image to stretch it to the correct positions of the UV map. But this is not the same as a fully customized mesh, and the flexibility it provides.

So while it is a solid turn-key solution, I would not necessarily recommend it for anyone who intends to use their own 3D art.

 

I am afraid what NaturalFront Unity plugin actually does is exactly the opposite of what you said.

I have tried NaturalFront Unity plugin. The NaturalFront Unity plugin can build a 3D-model that resembles whatever person in a photo, in less than two minutes. Best of all, the 3D-model will have built-in 3D-animation that is also high-quality!

I have built about two dozen 3D-models using NaturalFront software, the 3D geometry of each model also resembles different person in a photo, assuming your photo meets a few simple requirements. So not only is the UV as you said.

I don't know your situation, but I am an indie, and also my own boss and marketer.

What I have found is that, within a few minutes, the NaturalFront Unity plugin produces much better result than anything else we have discussed in this thread, in fact, orders better.

Please, Richard, please prove this is different otherwise. I mean really prove, as you would prove to your boss or clients.


I finally made up my mind to write these, since I just found this video:

It is quite funny and keeps me "oh!" for most of the time. It certainly contradicts to what Richard said about their software.

 

And what happens when you want to add facial animation to a character that looks like a dog? Or a cat? Or an exaggerated cartoon character? How about an anthropomorphized car?

This is what I'm referring to when I point out the limitations of the software in question. If you want to make something that fairly closely resembles a realistic human, it's good to go. And I have no doubt that it can handle a wide variety of facial types. But can it automatically animate custom 3D art that is not strictly structured like a human face, with standard human face proportions?

The sale's pitch they offer is convincing. And they're even right about several points. Being able to do these things quickly is very appealing. But don't be surprised if this software suite limits your stylistic choices. You will be making vaguely-realistic human heads. If that's fine for the type of project you're working on, good luck to you, and have fun. If you need a bit more visual range, you might want to consider other solutions.

 

Hi everyone,

Thanks for your interest in NaturalFront software.

I am working for NaturalFront. What I am writing here are from myself and not necessarily NaturalFront's viewpoints.

You are right that currently NaturalFront Unity plugin only models and animates realistic 3D human-faces.

But don't forget this is one of the most challenging tasks in the animation industry. Because of this challenge, the industry has chosen to offer a disproportionately large number of exaggerated or fancy characters to its user! Since once studios aim the realistic end of the spectrum, they run the huge risk of ruining their animation, see this recent example.

With NaturalFront software, this situation is fundamentally changed, since users can create realistic 3D-face-models very easily, e.g. from a photo. Once created, the 3D-face-model will have high-quality animation with it. The whole process is mostly automatic and can be completed very quickly.

In addition, the limitation of the current version doesn't mean the underlying technology is less suitable for the task.

I don't know on what basis you created this opinion. If you load a good photo that shows details of a face, these details will be converted into the computed 3D-face-model, including both texture and geometry.

One little trick is that, in the plugin UI, you can choose this setting

"More tolerance of imprecise point-positioning at the cost of speed"

to get more detailed texture at the cost of a little more time.

Thus, the end result might be even better than one you create manually, not to mention the huge time saving.

Try out for Free.


Thank you for your interest in NaturalFront software.

 

Dear Friends

Does anyone here can upload FACEPLUS plugin somewhere ??
This plugin is lost in space. No where to find it anymore. And i need it again.

:/

 

Hi, Richard. A lot of people keep asking me about Your Cheshire plugin for Unity, since 2014 or 2015. Are You going to reboot it and keep working, or have other plans regarding that? ( I hope my tutorials series is not the reason You suddenly removed it from Asset Store and close down Your main site )

 

No, it was removed from the Asset Store automatically because I hadn't updated it in so long. Life has intervened in my personal development efforts. I just haven't had as much time as I would have wanted to focus on efforts such as this. I do actually have an updated version of the plug-in, but I haven't had the time to create content or documentation for it.

Also, I was never entirely satisfied with the auto-detection of timing. The solution I was using was an open-source command line program from 2005. It worked decently enough, but it was a Windows-only solution. I did some research into using some of the more modern services from Apple and Google, but haven't come up with something yet that delivers satisfactory consistency.

I put a quickie demo together to test the Apple voice recognition software, but I found that the results it returns don't include individual phoneme timings. It will give you a solid prediction of the spoken words, and it will give you a word-timing breakdown, that seems to be pretty accurate. But it won't give you specific phoneme timing, which limits its utility for animation. From the research I did into the Google voice recognition, it functions in a similar fashion and returns comparable results. The only way to use this is to wrap in a dictionary of standardized timings, and match them to the word results. (a fairly time-intensive solution) And again, time is at a premium for me.

Also, and this part is important, for the longest time I've bee wanting to actually make a game. While Cheshire is a fun project for me, it is a tool, not a game. I have a habit of getting really enthusiastic about building tools, and then not doing anything with them. One of my objectives in making Cheshire was to use it to produce my own animations. But I got so wrapped up in the tool that I never got around to creating products with it. One of my current projects is prototyping an unrelated game, a game that will likely not use Cheshire much, if at all. So part of the reason I haven't been prioritizing it is simply through a shift in focus.

 

I actually appreciate you chiming in on this. I had dug it back up a month or two ago with the intention of giving it another go, but de-prioritized it because of how Unity handles object selection.

When I upgraded the plug-in, I wanted to make it more flexible, allowing people to incorporate a greater number of approaches. The first version of Cheshire focused only on using sprites or 3D shape-keys. You essentially had to choose between the two approaches. If you wanted to use bones, you were out of luck. If you wanted to use separate game objects, you were out of luck. The plug-in would only work with the specific object it had been applied to. This helped to keep things very simple, but it also severely limited how it could be applied.

But when I began researching the possibilities of expanding object selection, I found that Unity does not necessarily play nice with that sort of thing. There is no one generalized way of selecting individual properties for applying animations. Given the component structure of Unity, this is hardly surprising. First you would have to select an object, then select an instance of a component, then select the property you wanted to animate, then possibly select a sub-property of that property. Trying to come up with a generalized approach simply wasn't feasible.

But being reminded of this project helped my brain to stew on the issue, and I realized I was trying to do things the hard way again. I don't need to cover every base with a generalized solution. I just need to cover the most common bases, and the most common solutions. I can leave the rest to individual developers, and their distinct use-cases. As long as I've provided the needed flexibility to expand on my initial work, I can be satisfied that I've done my due-diligence. The rest of the focus should be on making the solution easy and simple to work with. With that approach, I can create a manageable interface that covers what is essential.