Unity 3D + faceAPI

I became interested in incorporating the face tracking library faceAPI with Unity3D. Looking around I found tons of videos showcasing the connection but it was difficult to find any documentation, and some tutorials even required people to purchase a UDP script for Unity. I've just finished writing up my own tutorial with all the source code UDP script and all.

You can find it on my blog at: http://www.andysaia.com/radicalpropositions/?p=197

I hope somebody gets some good use out of it, any and all comments are appreciated.

 

Thank's a lot Raptor Claw!

Your tutorial is really easy and usefull!

I found only a little stability problem: when I don't move my head camera still move a little bit up-down and right-left; but I think that is a faceapi problem with my laptop webcam.

 

thanks!

I suppose this won't work in webplayer?

 

@Kenshin
Yes, there is a little bit of shake to it, as you correctly guessed the majority of the blame is given to the faceAPI streamer application, but certain factors like the lighting in the room or quality of the webcam play a role as well. Still I think there might be something that can be done inside of Unity to round the numbers in a way to prevent some jitter.

@gnoblin
No, currently it won't work in the webplayer. I have been thinking of some possible solutions though. One idea would be to have a web interface send the webcam data to a back end server that does all the calculations and sends the position and orientation information to the unity application. I think this would probably be really difficult if not impossible to do in real time but maybe somebody is smart enough to do it.

The other idea (and probably more practical) is to build a faceAPI plugin for unity using .dlls. The company that makes faceAPI (http://www.seeingmachines.com/) is currently working on porting it to C# which may make this process easier still. If anybody has any better ideas than these I'd be interested to here them.

 

Raptor Claw, thank's for your answer.

A c# porting will be very very interesting.

Do you know prices for commercial version of faceApi library?
Are you using this for any real project?

 

I'm not sure how much the commercial license is. I've been planning on making something similar to this in unity
http://www.youtube.com/watch?v=Phcdp7AjhBg

The non-commercial license doesn't support lips and eyebrow tracking so I will have to fork over the doe eventually.

 

Really interesting! ...for manage advanced tracking features I was thinking about commercial license.

 

Amazing! The future is wide open.

Would even the cheapest webcamera work with this? Or do you need one that can do a certain level of quality? I don't have one yet, but a quick search on Amazon.com shows some only cost 6 dollars! http://www.amazon.com/s/ref=nb_sb_noss?url=search-alias=aps&field-keywords=webcamera&x=0&y=0

Thanks for sharing this.

 

The most important thing to keep in mind when buying a webcam for tracking is the frame rate. A crappy webcam would probably work but If you want to have your game running at 30 fps and the webcam only captures 12fps you might have some choppy tracking. I would recommend the PS3 camera(http://www.amazon.com/PlayStation-Eye-3/dp/B000VTQ3LU) It's not too expensive and is capable of a whomping120fps, which is pretty incredible for a webcam.

You can find the driver for it here: http://thecodelabs.com/downloads/

 

All webcameras have negative reviews.

I read that the PlayStation Eye is only meant to work for PlayStation, not a personal computer, and you need a PlayStation to get the software.

The site you link to says that software is only free software for 30 days.

If I get a different webcamera, i won't need to buy that software to make this all work?

Just need the free software at http://www.faceapi.com/Download/login.php?requires_login=true&page=/Download/downloads.php&query=

If you release anything you have to give all the source code away free if its with the free version of their software. And they refuse to tell you how much it cost for a commercial license, or whether they'll let you distribute anything you make with it at all. http://www.seeingmachines.com/product/faceapi/licensing/

Any other software out there that does the same thing?

And for a webcamera, I only need it to see the whites of my eyes, which should stand out rather well from the rest of my face. Don't care about using it for any other reason.

 

I think you are wrong because only the product "CL Studio Live" require payment.

PS Eye PC driver are free and you can connect this camera to your pc without any problem.

 

My mistake. Saw the 30 day trial thing up top and got confused.

http://codelaboratories.com/products/eye/driver/ says its free for noncommercial personal use.

 

This is Marvelous now lets get it ported over to the Mac please?

 

Hmmmm.. when i try to open your project in Unity 2.6 it says that it was created with an older version of Unity.. and when i open the scene Target it is empty, so are all the prefabs etc.. any ideas?

EDIT: it seems that Unity3 opens the scene but FaceAPICamera and FaceAPIController have missing scripts.. what scripts need to be attached to what?

EDIT AGAIN: Never mind!!! Got it working in 3.1.. just needed to reatach scripts and textures.. scripts are named quite logicaly This is awesome, just need to do some anti-jittering-thingie to make the movement smoother, othervise its really amazing!!!!!!!

Thanks for this, Raptor Claw

 

I've been using the facetracking library for Processing (processing.org), and sending the tracking data using the osc .net implementation found elsewhere in the forums. There is a mac version also. Pretty straightforward and all free.

 

good ole' processing, so MANY MANY tools. I've been looking for ways to integrate it better into Unity. Would anyone have any suggestions on which OSC implementations are best for Unity?

 

Im not sure which is best, there are several different implementations for Unity that have been shared on the forums, although searching for them can be tricky as OSC is too few characters for the search engine I think.

Personally I used a modified version of OSCuMote:

http://forum.unity3d.com/threads/21273-OSCuMote-Wiimote-support-for-the-free-version

 

@elbows
Thanks a TON for sharing the OSCmodule link. I've encountered the 3 character search term limit, so this helps out a lot. Do you use VVVV too?

Time to see if the data from the Kinect can be gotten into Unity ;-)