Search Unity

  1. Unity 2020.1 has been released.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

FaceFX Unity - Facial animation integration

Discussion in 'Made With Unity' started by facefx, Oct 12, 2010.

  1. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    After several requests from Unity developers, we've created a sample integration with FaceFX, our audio-based facial animation product. You can check out the demo here.

    The integration is just an editor script to import a FaceFX XML file on your character and a controller script based on the Character Animation documentation. The integration and demo use bones-based animations.

    Using curves to drive bone poses as opposed to importing FBX files for facial animations was very effective at reducing the amount of data needed to store the facial animations. After some quick tests, we estimated that we were saving over 90% versus default FBX export. Obviously, that figure can change based on the number of bones, number of curves, FBX compression, etc. But for anyone using bones that has a lot of facial animations to play, the Character Animation documentation is a good place to start. A custom controller script also helps keep the animation in synch with the audio.

    Thanks to the Unity team for putting together such a great product. We're looking forward to using Unity more internally.

    For anyone curious, FaceFX is a leading audio-based facial animation solution. It's been used in over 150 triple-A titles. The solution available on a per-title basis, as a standalone product, or as plugins for Max, Maya, XSI, and MotionBuilder.

    Let us know if anyone has any questions, comments, etc.
     
  2. Londy

    Londy

    Joined:
    Jan 24, 2010
    Posts:
    108
    My god this is epic!
     
  3. santiainen

    santiainen

    Joined:
    Oct 9, 2010
    Posts:
    9
    My god this is expensive!

    EDIT: Seriously, is there a uh... indie version of this, which would be.... like.... cheaper, or absolutely free?

    EDIT2: Also, can you add paypal support? I've had bad experiences with google checkout. I might actually buy this.
     
    Last edited: Oct 12, 2010
  4. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    FaceFX plugins are $199, which is pretty cheap. And the FaceFX License Manager lets you easily transfer your license to other computers. FaceFX Studio Professional is $1,995.

    You can use the cheaper plugins with this integration, but there is one "gotcha". The unity integration requires a curve for each bone pose in the animation. The Face Graph is capable of driving multiple poses with a single curve, so prior to XML export into Unity, we require that you publish your FaceFX actor to collapse the Face Graph. Publishing is only available in FaceFx Studio Professional, so if you want to use the plugins you would need a Face Graph with no links and curves for each bone pose you want to drive. Then you would generate your animations from the plugin, load up the .facefx file in "FaceFX Studio Free" (available from the evaluation installer) and export an XML file from there.

    I think there is a lot more we can do with Unity, and I recognize that there is a large community of developers that would need a free or almost free option to make use of our technology. We're looking into it, and if we can figure out a way to make it work, we will.

    As far as paypal support, I'll look into that. We can process your order manually too.
     
  5. Mani

    Mani

    Joined:
    Jan 12, 2009
    Posts:
    117
    This is sweet!
     
  6. Derydoca

    Derydoca

    Joined:
    May 13, 2010
    Posts:
    189
    I have no use for a tool like this right now, but when I do I will be checking out this plugin again. Looks very awesome!
     
  7. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    Awesome stuff!
    So correct me if I am wrong - I can't produce the XML files except via FaceFx Studio?
    If I already have the FaceFx plugin - what else do I require aside from the Unity integration files ?

    I guess my question is how do I export the necessary XML files?

    cheers,
    David.
     
  8. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    KITT, once you have generated an animation with the plugin and saved the .facefx file, you open the .facefx file in "FaceFX Studio Free" which was installed along with the plugin in the evaluation installer. FaceFX Studio Free can export the XML file.
     
  9. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    Many thanks!
     
  10. tatoforever

    tatoforever

    Joined:
    Apr 16, 2009
    Posts:
    4,053
    So in order to get FaceFX in Unity, i need:
    - FaceFX plugins (200$)
    - FaceFX studio (2000$/free?)
    - The Unity FaceFX XML reader/converter (free?)
    I'm i right?
     
  11. rahuxx

    rahuxx

    Joined:
    May 8, 2009
    Posts:
    537
    where i can get more reference and documents regarding FaceFX and Unity.
     
  12. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    You need a FaceFX plugin *or* FaceFX Studio Professional. Then you need the Unity scripts.

    I'm going to put some more information together in the next few days. I'm glad there is a lot of interest. Stay tuned.
     
  13. EvolverTim

    EvolverTim

    Joined:
    Jun 25, 2009
    Posts:
    5
    FaceFX - Great demo. We appreciate your choice of Evolver characters for the demo. www.evolver.com
     
  14. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    You can download the source files to the FaceFX-Unity Demo here:
    http://files.facefx.com.s3.amazonaws.com/unity-demo/unity-demo-facefx.zip

    This should let people:
    1) Check out the unity integration scripts.
    2) Load up the .facefx files in the FaceFX No-save version. (You can't save your work or export new XML files for playback in unity).

    With a license to FaceFX Studio Professional, getting an animation to play in unity would involve the following steps:
    1) Open one of the included .facefx files in FaceFX Studio Professional and generate one or more animations. Save the .facefx file.
    2) Run the fgcollapse command to simplify the Face Graph because it can't be computed by the Unity integration. (This step requires a license to FaceFX Studio Professional. Plugins don't have this capability, so complex Face Graphs are not an option there)
    3) Open the collapsed file, and export an XML actor file into the unity project. Make sure the FaceFX actor name matches the Unity GameObject.
    4) Make sure the FaceFXImportXMLActor.js file has successfully imported the new XML file. The animation is ready to play!

    If you don't have a license to FaceFX Studio Professional, but instead have a license to a FaceFX plugin (or you want to try out the phrase-locked evaluation version), getting an animation to play in unity would involve the following steps:
    1) Load one of the .facefx files in the plugin
    2) Generate an animation from the plugin
    3) Save the .facefx file
    4) Load the .facefx file in FaceFX Studio Free
    5) Export an XML file into the unity project.
    6) Make sure the FaceFX actor name matches the Unity GameObject. Then check that the FaceFXImportXMLActor.js file has successfully imported the new XML file. The animation is ready to play!

    For plugin users, because we haven't collapsed the Face Graph, you will notice that the body gestures, and the head and eye rotation do not work in Unity. These features are utilizing the Face Graph which is not being evaluated in Unity. There is a workaround however...when you generate the animations from the plugin, use an analysis actor that outputs curves for each bone pose. I've created an analysis actor here that will create head and eye rotations for the Evolver characters in this demo.
     
  15. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    EvolverTim, thanks for all of your help putting this demo together! I'm excited to see what the Unity community can do with FaceFX and Evolver. The sky is the limit!

    For anyone interested in using Evolver in Unity like we have done for this demo, we used the FBX export option with "Y-up", the "Darwin Default" skeleton and the facial bone rig. You can transfer bone poses from one evolver skeleton to the other using the bonepose command in FaceFX Studio Professional, so character setup is extremely easy if you are using Evolver characters!
     
  16. foundre

    foundre

    Joined:
    Sep 27, 2009
    Posts:
    36
    Looks awesome...wish I could afford it but unfortunately I am tapped out at the moment.
     
  17. pkid

    pkid

    Joined:
    Jul 10, 2009
    Posts:
    201
    I downloaded the free no-save version of facefx studio and it looks great so far. Do you have a reference model that could be sent to an animator that shows all of the poses needed?
     
  18. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    @EvolverTim @ facefx - I think a tutorial which takes us from Evolver--FaceFx in Max--Unity 3D would be awesome.
    Many people in the community, once they see the workflow, will gain a better understanding of the process and ease which is achieved via these great services/products.
     
  19. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    Inside of the FaceFX evaluation installation directory, look in the FaceFX Studio No-Save Evaluation/Samples/Src directory. Slade.FBX, and Slade-Maya.ma are in there. They have all of Slade's bone poses laid out in the timeframe, and you can use the plugins to "batch export" the bone poses with the Slade-Batch-Export.txt file. Once you have the bone poses, you can use the Slade-Maya.fxt or a Slade-Face-Graph-Setup.fxl file to set up the Face Graph.

    Another option is to use the bones-based evolver characters included with the Unity demo. You can get the source FBX files from the zip file, and you can get the bone poses from the included .facefx files.

    http://files.facefx.com.s3.amazonaws.com/unity-demo/unity-demo-facefx.zip

    Both of these options might have more bone poses and Face Graph setup than you absolutely need. For a minimalist setup, start with the 6 mouth targets (open, PBM, FV, ShCh, W, wide) then add Blink, Eyebrow Raise, and Squint. If you have a tongue, you can add the tBack, tTeeth, and tRoof poses. Finally, you can add poses to rotate the head and eyes in all directions.
     
  20. pkid

    pkid

    Joined:
    Jul 10, 2009
    Posts:
    201
    I agree with KITT, there are some good videos up on the FaceFX site but it would be great to see a start to finish video on how to bring a character through facefx into unity.
     
  21. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    I'll see what I can cook up. Might take a day or two...
     
  22. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    Here is a tutorial on youtube.


    If you want to follow along, be sure to grab the updated unity demo files.
     
  23. rahuxx

    rahuxx

    Joined:
    May 8, 2009
    Posts:
    537
    great video. explaining most of the things, but is it possible to do all this using evolution version of FaceFX?
     
  24. wilkolak

    wilkolak

    Joined:
    Aug 29, 2010
    Posts:
    1
    I can't find this information anywhere... is there (or will be) avaible Maya 2011 FaceFX plugin?
     
  25. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    Yes, I did the tutorial with FaceFX Studio Free, and the evaluation plugin. I "cheated" by using FaceFX Professional to transfer bone poses otherwise the tutorial would have been long and painful to watch.

    Yes. Supported versions listed here. Just download the evaluation installer.
     
  26. pkid

    pkid

    Joined:
    Jul 10, 2009
    Posts:
    201
    That was a good tutorial, but I was looking at facefx studio pro, is there any chance of getting a facefx studio pro to unity tutorial?
     
  27. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    Here is a tutorial demonstrating the FaceFX Studio Professional pipeline. The two commands used in the tutorial: fgcollapse and bonepose -transfer are not available in the no-save evaluation, so you need FaceFX Studio Professional to follow along to this one.
     
  28. pkid

    pkid

    Joined:
    Jul 10, 2009
    Posts:
    201
    Thanks for the video. Did you select all of the bones in 3ds max or just the face bones? It was hard to see.
     
  29. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    Thanks for both videos. Helped me understand a coupla things better.
     
  30. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    I selected all of the bones. It depends what you want to drive. Most people just want the head bone and face bones. The "Body Gestures" analysis actor creates some basic curves for posture shifting and arm movements however, so I select all of the bones. If you want to drive facial animations on top of a walk cycle or full body animation, then using FaceFX to drive only the head and face bones is what you want.
     
  31. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    I will be at Unite. I'd be interested in speaking with anyone there that is considering using FaceFX. You can reach me at (919) 727-9624 , or by sending an email to: info (at) oc3ent.com
     
  32. pitaman

    pitaman

    Joined:
    Jun 22, 2010
    Posts:
    5
    Our company licensed Face FX, but how do you get the sound to sync in Unity3d?
     
  33. IonDave

    IonDave

    Joined:
    Sep 27, 2010
    Posts:
    58
    In the tutorial video, you open Doug-Max.facefx but in the downloaded demo files, this doesn't exist. I know I can load the Doug.facefx file which solves everything, but that doesn't really help me learn. I have to be honest, the tutorial is extremely hard to follow along with and I've been using 3ds Max since R1.5. I have money burning a hole in my pocket to buy this software. I have my own evolver characters with face bones, 3ds max and all of the facefx software ready to go. Is there a maybe a clearer, step-by-step video for the complete, no shortcut workflow?
     
  34. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    In the FaceFXControllerScript.js, there is a PlayAnim function that looks like this:

    Code (csharp):
    1. function PlayAnim(animName: String, animAudio : AudioClip)
    The second variable in this function is the AudioClip to play. The FaceFX controller script will start the audio at the appropriate time in the animation (there can be some animation before the sound plays), then keep the animation in sync with the audio as it plays. When the sound is finished playing, the FaceFX Controller script will play the rest of the animation (if any) that happens after the audio ends.

    In the phraselocked.unity example, the AudioClip is the default audio attached to the Audio Source component of the Doug character. Obviously, this is not a scalable solution, but the intention was to keep things simple in that demo. In the more complete unity-demo.unity example, we download .ogg files from the web and create the AudioClip dynamically.

    The Doug-Max.facefx was created with FaceFX Studio Professional by using the bonepose -transfer command to transfer all of Jake's bone poses onto the Doug reference pose. We do this in the second video tutorial that is for FaceFX Studio Professional customers. Plugin customers do not have access to the bonepose command in FaceFX Studio Professional, so bone poses need to be created manually. In the tutorial, we create one bone pose "open". The same procedure needs to be followed for all other bone poses you want to drive. I don't have a video tutorial of this, but there is some documentation on how to create the speech target poses.

    The tutorial does move quickly and assumes some knowledge of FaceFX. For more information, definitely watch the introduction to the FaceFX plugin video tutorial too. Other than a tutorial that goes through the creation of all the bone poses, if there are any other parts that leave you scratching your head, let me know.
     
  35. pitaman

    pitaman

    Joined:
    Jun 22, 2010
    Posts:
    5
    Thank you very much for your prompt reply. I will give it a try.
     
  36. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    Dear Facefx,
    Was hoping to be in Orlando for IITSEC, and stop by the booth - wont make it so I'll ask here:

    Could you do a tutorial or briefly go over the process of adding/blending the Facefx animations in Unity with say a walk cycle/idle cycle - I am sure I can figure it out through trial and error but I barely have the time to dig in any deeper, as I'm uber busy at present.

    A video tutorial/text walkthrough would be greeeatly appreciated!
     
  37. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    Hi KITT,

    With the holiday and I/ITSEC, I won't be able to do any sort of demonstration that drives FaceFX on top of a full-body animation in the near future. I will say that the FaceFX integration is set up with standard Unity additive animations. They should play on top of whatever full body animations you want to trigger out of the box. Just make sure that you aren't driving the same bones in FaceFX and your full-body animation For example, use FaceFX to drive everything above the head bone, and your full-body animation to drive the head bone and below. If your full-body animation opens the mouth and you play a FaceFx animation on top of that, the results will not look good.

    I will look into this more as time permits.
     
  38. KITT

    KITT

    Joined:
    Jul 17, 2009
    Posts:
    221
    Cheers! I think selecting the facial bones only for my facefx animations before hand will solve the problem I was having with additive animation.

    Best wishes at the conference.
     
  39. mindengine

    mindengine

    Joined:
    Sep 8, 2008
    Posts:
    114
    anyone tried to get facefx working with a VOIP solution?
     
  40. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    FaceFX won't work with live audio, only pre-recorded. But the additive animation integration with bone poses would be a good starting point for a zero/low latency volume based mouth flapper. But you would need to write the part that drives the bone poses based on the audio tht came in over the VOIP solution.
     
  41. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    Some of the info in this post is no longer relevant, so I've moved the conversation here, where I'm showing off the new and improved pipeline.
     
  42. smiljan66

    smiljan66

    Joined:
    Jun 9, 2010
    Posts:
    610
    The company im wokring has facefx full , but i read the integration ot unity and is veyr complex we need a face fx integration and correct steps ionto unity for it to work , do some video tutorial ?

    Thanks

    Smiljan Pecjak
     
    Last edited: Dec 21, 2012
  43. abel gnangbi

    abel gnangbi

    Joined:
    Jun 21, 2013
    Posts:
    4

    Morning can someone put this video again ? cause it's not available!
    I'm in France and i bought a Professional version of FaceFX Studio. I want to integrate that in Unity 3d; i hope this tutorial help me to do that, but it's not available :(. So if someone got this tutorial or got another which might help me, it willl be very great!

    Thank you, i'm waiting for your reply ;)
     
  44. facefx

    facefx

    Joined:
    Jan 18, 2010
    Posts:
    116
    This thread is old! The Unity pipeline was made more simple, and you should refer to this thread instead.

    The video was taken down because it is no longer accurate. There is no longer a generate animation button on the Max/Maya plugin. You generate animations from FaceFX Studio, then export to FBX, or go to File->Export Collapsed XML and use the integration files to import the XML file.

    Check out the new thread (especially my last post towards the end) and the video tutorials, and post any questions there.

    Best,
    Doug
     
  45. mkgame

    mkgame

    Joined:
    Feb 24, 2014
    Posts:
    592
    FaceFX looks good, but this is just a part of the cut scene animation. About 900$ is ridicolous, because Poser and Maximo has it too for much more cheaper. We need an indie license, which means full payment at success, but before about 100$. In this state no one buy your asset, we can see it on the non available ratings. I belive that it is pitty, that we will not grow up with your tool, and so we will probably never use your tool, if we are already trained in another ones, like Maximo or Poser and other.
    But maybe you are a company, which have enough income by bigger game studios and so you don't need the small indie developers.

    The thread is old, but FaceFX is still alive and has a non acceptable price for indie developers, and i don't like the fact that Maximo and Poser are the only solutions for indie developers, where Maximo put it in an expensive bundle, which is acceptable by montly payment (if the cutscens are done after 2 month, then it is cheap) and Poser, which has lot of negative revies but can be buoght often for 200$.
     
  46. RichardKain

    RichardKain

    Joined:
    Oct 1, 2012
    Posts:
    1,258
    Well, as someone who is also working on a lip-sync related asset, I have to come to FaceFX's defense. While their offerings are certainly not cheap, what they allow you to actually do is quite impressive. FaceFX is geared toward recording full facial animation sequences, including things like blinking, raising and lowering of eyebrows, and the general deformation of the facial structure. This type of mo-cap using just a standard web-cam setup is very impressive, and can both save developers a lot of time on animating, as well as provide some incredible nuance for facial performances in-game. (an essential element to making digital characters believable and appealing)

    While FaceFX is a more premium product, you are getting quite a bit of value for your money. The real question to ask yourself is if animated character faces are worth the price. If you're creating a game where they are a visual bonus, as opposed to a central element, than you have your answer.

    As always, I am happy to pimp my own lip-sync solution.
    https://www.assetstore.unity3d.com/en/#!/content/18746
    Cheshire isn't geared toward full-facial animation, but does allow for faster automation of speech-related animation. (getting the characters to talk in a decent fashion) Even I am willing to admit that Cheshire's functionality falls far short of what FaceFX provides. But for indie developers, it is available for the low-low price of FREE, so it might be a little bit closer to what your'e looking for.
     
  47. mkgame

    mkgame

    Joined:
    Feb 24, 2014
    Posts:
    592
    No, Cheshire is not enough. I need more than lip-sync, i need facial animation too, like in FaceFX and Poser. Enhance your asset with facial emotions (then give it a price $ ), and you are the only one indie solution in the asset store.
     
  48. RichardKain

    RichardKain

    Joined:
    Oct 1, 2012
    Posts:
    1,258
    Ooooo, sorry, but that won't be happening any time soon. Cheshire is very small-scale by nature. I'm updating it right now, but the new functionality is not going to cover what you're describing. It will have better support for integrating other blend-shapes, but nothing automatic or mo-cap related. It's designed more to be a fun tool for indie developers to get some quick lip-synced animations into their games or machinima projects.

    I don't have the kind of time or know-how to create something on the scale of FaceFX. I'm sometimes amazed that I've even managed to throw together what I have.
     
  49. mkgame

    mkgame

    Joined:
    Feb 24, 2014
    Posts:
    592
    Im not sure that i want to do such high quallity cut scenes, but if yes, then the facial animation - which is by the way a small part of it - must be available.

    I found SALSA (automated facial animation) and lipsync (looks like a manually configurable facial animation tool) in the asset store, i don't know how am i missed them...
     
  50. lyc36155774

    lyc36155774

    Joined:
    Jan 28, 2016
    Posts:
    5
     
unityunity