[WIP][UMA] Master thesis player avatar creator

For my master thesis (computer science/software engineering/math) I am making an avatar creator that should be able to make an avatar for a player from a selfie or two.
The approach is to use ASM (Active Shape Models, http://www.ee.oulu.fi/research/imag/courses/Kokkinos/asm_overview.pdf) to find the place and shape of different parts of the face (nose, eyes etc) edit an UMA character face model based on those shapes and then edit the texture from image(s).

Maybe if this works well it will end up on the asset store in the future

Below you can see current result from a single selfie image, result will hopefully improve as I add the possibility to take a photo from the side as well.

 

Wow, that is pretty cool. Would be a great addition to UMA.

 

Yeah if I can get it to work properly, but this stuff is hard...

At the very least I think it could be of some help to 3D-artists for creating new face models.

 

Currently working on ability to create new shapes and train them. My earlier shape definitions were from my prototype made in matlab...

 

Here's a quick sitrep. Texturing from two views work but it requires quite a bit of manual work right now. Also there is now automatic shape editing of the UMA character face yet but that is coming!

 

The height of the nose and more, more polygons [for demo].

 

Ahh the polygons visible in the image are only for image segmentation and image-to-texture mapping, they do not make up the model itself. It is possible to define new shapes in the project if more polygons are needed for correct mapping though.

 

Here's a quick example where I created an UMA character from two photos and then let RAIN control the character while patrolling around a scene. I did do some post procesing of the textures here as I have yet to find a good solution for filling the whole texture when there is lacking image data. I need to figure out how to do a "hairmask" and a "skinmask" and then do procedural texturing of the parts of the texture that have not been filled.

Anyway here are the preliminary results:

 

And a screenshot of myself as an action hero:

 

One more example of an IRL-person turned UMA avatar. We're getting there! Right now I am working on using statistical analysis (namely finding coordinates of shapes in eigenspaces using PCA https://en.wikipedia.org/wiki/Principal_component_analysis ) to determine the UMA DNA for the face based on the photos that are segmented. It's not a trivial problem and it seems all solutions have some drawback or another.

I have been thinking about other approaches as well, maybe using a purely analytical method reducing some type of cost-function but will try the PCA-route first.

If you have experience with anything similar or have ideas or input I'd love to hear from you! Write below or send me a pm!

 

That's great! Only piece of advice I have is that it might be easier to customize the UMA DNAConverterBehavior to match the vectors of your analysis rather than trying to do some two step mapping. They are pretty straightforward.

 

Thanks that's some solid advice right there! I had not thought to dive deeper into UMA (not knowing it very well) but I have now had a look at the DNAConverters and I think you are absolutely right. Right now the UMA sliders do not correlate very well with my eigenspace coordinates leading to, just as you mention, two step swapping. As principal component analysis by definition finds the most "important" shape differences of the human face I guess it makes sense setting up a DNAConverter that correlates more closely with the eigenvectors from the PCA.

As my database of faces unfortunately is not ethnically diverse enough (consists mostly of male italians... long story...) I doubt that I will find the eigenvectors that best describe human faces but it might be a start

 

So I was intrigued by the faces of the walking dead game and decided to try and transform my own face to be similar to that. What I did was to use the face generated above that I used in a small demo project and simply draw by hand on top of the texture and then use the handdrawn version instead. Here's the result, compare to the walking dead-example. Note that I am DEFINITELY no artist, I have no ability at all and still it looks pretty good I'm going to let a friend with more artistic skill have a go at it and try to generate a few handdrawn examples from generated facial textures.

 

Here are a few more example images of completed characters.

I am currently working on the automated settings of UMA DNA values based on coordinates in the eigenspace spanned by the principal mode eigenvectors. Hitting some snags here and there, I think this may be in part due to a too small face database, but slowly getting there.

 

Here's an example of me using the system to generate myself using a webcam. Webcam works but it doesn't give the most detailed textures and it's quite hard getting good diffuse light.

The system is pretty much ready for implementing in a game or application as is but a lot refactoring, optimization and user friendliness is needed before it can be released as an asset on asset store.

 

Been tweaking a bit to get better results. Also now have a scene for fine tuning the results using sliders to set the UMA DNA

 

Putting a shirt on reduces the effect of the "skin transition" that can be seen above. In the future I guess a procedural texture generator forr the skin based on the staatistics of the face skin would be in order but one thing at a time. One thing at a time...

Getting more and more pleased with the results. Not every result is this good, I'd say that this is the best case. Also, textures can't be better than input photos of course. In this case I used two images taken with a system camera.

 

Here's an image that should give some clues to how the process works. It makes many months of work seem so easy....

 

A generated avatar in the fine-tuning scene

 

If anyone has ideas for and would like to create a demo scene/game/webcomic/movie with this system (or just the avatars, I can also provide the finished avatars to you as a package) please write here or PM me! Of course you will get this package for free in that case!

It would be really cool when presenting my thesis if I could show some stuff others have made using this system.

 

Some examples of generated people:

 

Results are fantastic, look forward to release.

 

Thank you very much, work on this has been very hard end very tedious so your appreciation means a lot! I just hope to see this in a game some day. Right now there are some user interface issues that I have just had to ignore and focus on getting the thesis done but I hope to have a version ready for broad public use in a few months or so. That is why early adopters will get it for free, they'll certainly be worth it

Still, I have no idea what the business idea should be with this. Maybe have a service where users log in to get access to an avatar they have already created in different apps or maybe sell the ful source code. I can see pros and cons with both methods...

 

Hi there,

I am following your process from the beginning and it looks really cool! I would love to use it, but I have no idea what to make with it. (And I have a larg project I am currently working on..)

Anyway, good luck with this project and keep it going

 

Your moral support goes a long way! If you come up with an idea just let me know!

 

Looking back from the beginning, its amazing how much improvement there has been. At first, it was just a neat little tool, but now it looks like something that could be used in a quality game. I would love to see this used in a social VR game. Does it only support one side photo right now? Keep up the good work, though.

 

Thanks!

Yeah in my current setup it only uses one side photo, mainly to simplify things for me but it is easily extendible to use as many photos as you want. You could add photos from top, bottom, behind etc. Or you could add photos of the whole body from different directions. The only problem is you need to obtain about 100 photos to train the algorithm if you want automatic segmentation to work. This wouldn't be a problem from the side of course, you could just switch the photo along x-axis.

This system will probably be used in a university vr-class this spring, I'll get back to you with more info if/when I can

 

Some more results.

 

Five more (can only post 5 images in a post)

 

And those are the last of the bunch...

 

Five more sample results

 

See if you can match the original photos with their corresponding avatars:
http://goo.gl/forms/qNmXa6QlUt

 

Would be really nice if you could find a way to blend with the skin. Not sure how you would do it other than in Photoshop. I have seen it done before somewhere. Nice job though!

 

Ohh I know exactly how to do it! All I need is to create a cost function which increases cost for differences in neighbourring pixels at the edge and then shift hue, sat and light to minimize that cost... Then, to top it off, smooth out the last of the distances with a filter. This will take some time to code of course so this is not in my primary task list right now

Right now I am focusing on writing the thesis paper which takes a lot more time and effort than I would have thought! Thanks you for your interest though, it makes me want to get back to coding and finishing this stuff up!

 

Great!

Good luck on your thesis. I had to write one of those once upon a time. Feels good when it is done!

 

I am getting to a point where I am starting to consider possible uses for my system. Except for the obvious - making a zombie game where you actually have to kill your entire family now turned zombies - I'm thinking this could be used for creating an awesome webbshop where you can try on clothing items on an avatar looking like yourself.

Disclaimer - I'm not a diehard fan of the Dota player Wagamama, (https://www.facebook.com/WagaGaming-453238441382927/?fref=ts), he is, through no fault of my own, my younger brother

 

I have now prepared 31 images and released as a package on asset store.

You can try out the demo here:
https://dl.dropboxusercontent.com/u/34265144/Faces for UMA/index.html

and check out the page in the asset store here:
https://www.assetstore.unity3d.com/en/#!/content/53256

Meanwhile I am working to create a web client to create avatars that can then be downloaded from a databasse into another client which would then be available as a unity package. This is a few months away though

 

...aaaand there we go!

Final version of the master thesis report is done!
Here it is in all it 60 pages of splendor: https://onedrive.live.com/redir?res...3097&authkey=!AE4h0kcLLvA4pMs&ithint=file,pdf
Unfortunately too large to attach to the post (~10MB)

For those not mathematically inclined there is actually a much simpler popular science article describing the system as well in only 2 pages attached below as pdf.

I will present my work on thursday at Lund University in Sweden, anyone who feels like coming by is welcome to do so

 

Time for an update I think. I have done some work implementing PBR-support for my system. It was actually a lot easier than I thought! I think the results are well worth it too.

Here are some images of friends and family now in glorious PBR shaders

The "Faces for UMA"-package on the asset store will soon be updated to include these of course. (https://www.assetstore.unity3d.com/en/#!/content/53256)

If you would like to try it out yourself, try this webdemo: https://dl.dropboxusercontent.com/u/34265144/Faces for UMA PBR family photo/index.html

 

Great work! I can see that being pretty fun to use in an FPS-type game. (I don't play them myself, but having faces of people you know in one would be pretty cool)

 

wow, that looks really cool!
Great update

 

Haha yeah, neither do I, but imagine the joy getting killed by the same guy over and over in some FPS and recognizing his face

 

Thanks! I am also working on a system to be able to create yourself online but that will take even more time before I get it working...

For now it needs to be a manual process where I create avatars of people

 

Man, this is fantastic stuff.

In use, it seems it would be more practical to have colors not taken as an image capture, but instead chosen from a palette like most character creation screens. Then lighting in the room would not matter, and you wouldn't get mismatched body/face skin tones, and flattened hair on the neck for the longhaired hippies out there. Then let em' pick hairstyles & eye colors, etc etc etc. That's the only thing I could think of that might improve it.. this thing is great.

 

haha damned longhaired hippies screwin' up my system!

Don't worry though I'll solve the falt hair problem as well at some point I have plans for how to do it, just not enough hours in the day Actually there's a lot of important information in the actual image if you use a decent quality camera. It will capture all the little wrinkles and stuff that makes you recognize yourself as an individual. You do have a point about skin tone though, maybe you should be able to correct the skin tone of your photo...

 

Ok I think you nailed it about keeping the image capture information... maybe you could apply it to a greyscale and layer it over the top of a base skin color, which can be chosen. Hmm, well no you'd still run into issues at the seam between the neck and chest because of it. I wonder if you could choose a color from the image capture to use as a base for the rest of the mesh? (Done for the user via a script ideally)

 

Yeah, that's actually my planned approach. If I can determine what the person's main skin tone is then I can tone the rest of the skin to that. Maybe even apply a smoothing filter at the seams.

I've been working on a "hair-detection-filter" this week using a similar approach where I determine what the average skin color is from a central part of the face and similar for the hair. It should be possible to use this to tone the rest of the body I guess... I do all this prototyping in matlab though so I have to implement it in scripts/shaders somehow

 

In case anyone is interested, here's the official link to the final master's thesis: http://lup.lub.lu.se/student-papers/record/8567678

Should anyone be interested in doing any academic work relatting to this, do not hesitate to contact me!
There is currently one guy, Robert, doing some very interesting work where he creates an eigenspace (a set of eigenfaces) from the resulting facial textures of my project. This will enable a texture to be broken down to a set of coordinates, each describing the amount of each eigenface to build a texture resembling the original. In Roberts early tests it looks like a 2kx1k face texture can be represented by only 15 eigenfaces which means that you will only need a set of 15 floats to describe a face. This will enable fast avatar downloads from servers and should also enable e.g. personal avatars in MMOs. As a positive side effect, if someone draws offending symbols in their faces, those symbols will not be able to be represented by eigenfaces so will not be included in the resulting texture. Me and Robert will cooperate to try and bring all of this into your hands via the asset store!

If you cannot wait for this to reach the asset store but really want a texture of your own face right now then you can try our very very very early alpha build of the web GL app that will be used to create the avatars. Please be aware that no work has been done AT ALL to make the process easier for you as a user so you may need some trial and error We will work on UI later, right now we have other technical issues that need solving first.

Here's a link to the Web GL app in case someone wants to give it a go:
http://vikingcrew.net/Selfie_to_Avatar/index.html
The resulting texture that you can download will work on the standard UMA2 male/female models. We will make a texture mapper to remap it to any arbitrary face model later, but, as already mentioned, right now we have other technical issues to work with

The long term goal here is to create a service where a user can create their avatar in our app, and then when playing a game made by you, fellow developer, they can log in to our service and download their avatar to play in your game. If you as developer wants it you should also be able to get friends of the player into your game so they can have their friends as allies or enemies in the games. Or zombies. I would prefer zombies. We will also use the constructed eigenspace (those eigenfaces) to be able to create an infinite amount of random faces for populating random faces in the game world.

If anyone's interested to work with us through alpha/beta please let us know! Also let us know if you'd prefer the standard texture mapping to fit the UMA2 model, DAZ, WillB's models or some other system. You can always remap but each remapping reduces quality somewhat so it would be good to use the most popular system as a base.

 

Would love something like this for mobile. Mapping the users face on to the character in our fishing sim.

 

You know, getting possible use-cases like that is just golden for us! Thanks! May I ask what avatar tech you'd prefer? Like UMA, DAZ or something else?