Search Unity

[WIP][UMA] Master thesis player avatar creator

Discussion in 'Works In Progress' started by Mikael-H, Jul 24, 2015.

  1. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    uma_editor.jpg For my master thesis (computer science/software engineering/math) I am making an avatar creator that should be able to make an avatar for a player from a selfie or two.
    The approach is to use ASM (Active Shape Models, http://www.ee.oulu.fi/research/imag/courses/Kokkinos/asm_overview.pdf) to find the place and shape of different parts of the face (nose, eyes etc) edit an UMA character face model based on those shapes and then edit the texture from image(s).

    Maybe if this works well it will end up on the asset store in the future :)

    Below you can see current result from a single selfie image, result will hopefully improve as I add the possibility to take a photo from the side as well.
     
  2. nosyrbllewe

    nosyrbllewe

    Joined:
    Oct 18, 2012
    Posts:
    180
    Wow, that is pretty cool. Would be a great addition to UMA.
     
    Arkade and Mikael-H like this.
  3. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Yeah if I can get it to work properly, but this stuff is hard... :)

    At the very least I think it could be of some help to 3D-artists for creating new face models.
     
    Arkade likes this.
  4. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Currently working on ability to create new shapes and train them. My earlier shape definitions were from my prototype made in matlab...
     
    Arkade likes this.
  5. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Here's a quick sitrep. Texturing from two views work but it requires quite a bit of manual work right now. Also there is now automatic shape editing of the UMA character face yet but that is coming!

     
  6. DxStd_IgnatPribylov

    DxStd_IgnatPribylov

    Joined:
    Sep 24, 2015
    Posts:
    30
    The height of the nose and more, more polygons [for demo].
     
    Mikael-H likes this.
  7. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Ahh the polygons visible in the image are only for image segmentation and image-to-texture mapping, they do not make up the model itself. It is possible to define new shapes in the project if more polygons are needed for correct mapping though.
     
  8. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Here's a quick example where I created an UMA character from two photos and then let RAIN control the character while patrolling around a scene. I did do some post procesing of the textures here as I have yet to find a good solution for filling the whole texture when there is lacking image data. I need to figure out how to do a "hairmask" and a "skinmask" and then do procedural texturing of the parts of the texture that have not been filled.

    Anyway here are the preliminary results:

     
    TRoNDaNeflin likes this.
  9. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    And a screenshot of myself as an action hero:

    Unity 2015-09-30 10-33-16-51.jpg
     
  10. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    One more example of an IRL-person turned UMA avatar. We're getting there! Right now I am working on using statistical analysis (namely finding coordinates of shapes in eigenspaces using PCA https://en.wikipedia.org/wiki/Principal_component_analysis ) to determine the UMA DNA for the face based on the photos that are segmented. It's not a trivial problem and it seems all solutions have some drawback or another.

    I have been thinking about other approaches as well, maybe using a purely analytical method reducing some type of cost-function but will try the PCA-route first.

    If you have experience with anything similar or have ideas or input I'd love to hear from you! Write below or send me a pm!

    david.jpg
     
    FernandoRibeiro and TRoNDaNeflin like this.
  11. ecurtz

    ecurtz

    Joined:
    May 13, 2009
    Posts:
    628
    That's great! Only piece of advice I have is that it might be easier to customize the UMA DNAConverterBehavior to match the vectors of your analysis rather than trying to do some two step mapping. They are pretty straightforward.
     
    Mikael-H likes this.
  12. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Thanks that's some solid advice right there! I had not thought to dive deeper into UMA (not knowing it very well) but I have now had a look at the DNAConverters and I think you are absolutely right. Right now the UMA sliders do not correlate very well with my eigenspace coordinates leading to, just as you mention, two step swapping. As principal component analysis by definition finds the most "important" shape differences of the human face I guess it makes sense setting up a DNAConverter that correlates more closely with the eigenvectors from the PCA.

    As my database of faces unfortunately is not ethnically diverse enough (consists mostly of male italians... long story...) I doubt that I will find the eigenvectors that best describe human faces but it might be a start :D
     
  13. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    So I was intrigued by the faces of the walking dead game and decided to try and transform my own face to be similar to that. What I did was to use the face generated above that I used in a small demo project and simply draw by hand on top of the texture and then use the handdrawn version instead. Here's the result, compare to the walking dead-example. Note that I am DEFINITELY no artist, I have no ability at all and still it looks pretty good :) I'm going to let a friend with more artistic skill have a go at it and try to generate a few handdrawn examples from generated facial textures. mike_drawn_face.jpg Unity 2015-10-08 23-04-44-49.jpg
     

    Attached Files:

    FernandoRibeiro likes this.
  14. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Here are a few more example images of completed characters.

    I am currently working on the automated settings of UMA DNA values based on coordinates in the eigenspace spanned by the principal mode eigenvectors. Hitting some snags here and there, I think this may be in part due to a too small face database, but slowly getting there.

    Unity 2015-10-20 20-47-48-46.jpg Unity 2015-10-26 13-26-38-63.jpg Unity 2015-10-27 12-59-38-63.jpg
     
  15. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Here's an example of me using the system to generate myself using a webcam. Webcam works but it doesn't give the most detailed textures and it's quite hard getting good diffuse light.

    The system is pretty much ready for implementing in a game or application as is but a lot refactoring, optimization and user friendliness is needed before it can be released as an asset on asset store.

     
  16. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Been tweaking a bit to get better results. Also now have a scene for fine tuning the results using sliders to set the UMA DNA

    finished fine tuning.jpg
     
    John-G and manpower13 like this.
  17. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Putting a shirt on reduces the effect of the "skin transition" that can be seen above. In the future I guess a procedural texture generator forr the skin based on the staatistics of the face skin would be in order but one thing at a time. One thing at a time...

    Getting more and more pleased with the results. Not every result is this good, I'd say that this is the best case. Also, textures can't be better than input photos of course. In this case I used two images taken with a system camera.
    result_lth_logo.jpg
     
    manpower13 and John-G like this.
  18. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Here's an image that should give some clues to how the process works. It makes many months of work seem so easy.... :p process overview.png
     
    John-G likes this.
  19. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    A generated avatar in the fine-tuning scene

     
  20. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    If anyone has ideas for and would like to create a demo scene/game/webcomic/movie with this system (or just the avatars, I can also provide the finished avatars to you as a package) please write here or PM me! Of course you will get this package for free in that case!

    It would be really cool when presenting my thesis if I could show some stuff others have made using this system.
     
    manpower13 likes this.
  21. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Some examples of generated people:

    results after fine tuning.jpg

    auto_segmentation_examples.png auto_segmentation_examples_side.png
     
    nosyrbllewe, manpower13 and John-G like this.
  22. John-G

    John-G

    Joined:
    Mar 21, 2013
    Posts:
    1,113
    Results are fantastic, look forward to release.
     
    Mikael-H likes this.
  23. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Thank you very much, work on this has been very hard end very tedious so your appreciation means a lot! I just hope to see this in a game some day. Right now there are some user interface issues that I have just had to ignore and focus on getting the thesis done but I hope to have a version ready for broad public use in a few months or so. That is why early adopters will get it for free, they'll certainly be worth it :)

    Still, I have no idea what the business idea should be with this. Maybe have a service where users log in to get access to an avatar they have already created in different apps or maybe sell the ful source code. I can see pros and cons with both methods...
     
  24. manpower13

    manpower13

    Joined:
    Dec 22, 2013
    Posts:
    114
    Hi there,

    I am following your process from the beginning and it looks really cool! I would love to use it, but I have no idea what to make with it. (And I have a larg project I am currently working on..)

    Anyway, good luck with this project and keep it going :)
     
    Mikael-H likes this.
  25. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Your moral support goes a long way! If you come up with an idea just let me know! :D
     
  26. nosyrbllewe

    nosyrbllewe

    Joined:
    Oct 18, 2012
    Posts:
    180
    Looking back from the beginning, its amazing how much improvement there has been. At first, it was just a neat little tool, but now it looks like something that could be used in a quality game. I would love to see this used in a social VR game. Does it only support one side photo right now? Keep up the good work, though.
     
    Mikael-H likes this.
  27. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Thanks!

    Yeah in my current setup it only uses one side photo, mainly to simplify things for me but it is easily extendible to use as many photos as you want. You could add photos from top, bottom, behind etc. Or you could add photos of the whole body from different directions. The only problem is you need to obtain about 100 photos to train the algorithm if you want automatic segmentation to work. This wouldn't be a problem from the side of course, you could just switch the photo along x-axis.

    This system will probably be used in a university vr-class this spring, I'll get back to you with more info if/when I can :)
     
  28. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Some more results.
     

    Attached Files:

  29. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Five more (can only post 5 images in a post)
     

    Attached Files:

  30. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    And those are the last of the bunch...
     

    Attached Files:

    manpower13 likes this.
  31. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Five more sample results
     

    Attached Files:

  32. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
  33. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,810
    Would be really nice if you could find a way to blend with the skin. Not sure how you would do it other than in Photoshop. I have seen it done before somewhere. Nice job though!
     
  34. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Ohh I know exactly how to do it! All I need is to create a cost function which increases cost for differences in neighbourring pixels at the edge and then shift hue, sat and light to minimize that cost... Then, to top it off, smooth out the last of the distances with a filter. This will take some time to code of course so this is not in my primary task list right now :)

    Right now I am focusing on writing the thesis paper which takes a lot more time and effort than I would have thought! Thanks you for your interest though, it makes me want to get back to coding and finishing this stuff up!
     
    manpower13 likes this.
  35. Teila

    Teila

    Joined:
    Jan 13, 2013
    Posts:
    6,810
    Great!

    Good luck on your thesis. I had to write one of those once upon a time. Feels good when it is done!
     
    Mikael-H likes this.
  36. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    I am getting to a point where I am starting to consider possible uses for my system. Except for the obvious - making a zombie game where you actually have to kill your entire family now turned zombies - I'm thinking this could be used for creating an awesome webbshop where you can try on clothing items on an avatar looking like yourself.

    Disclaimer - I'm not a diehard fan of the Dota player Wagamama, (https://www.facebook.com/WagaGaming-453238441382927/?fref=ts), he is, through no fault of my own, my younger brother :D
     

    Attached Files:

  37. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Teila likes this.
  38. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    ...aaaand there we go!

    Final version of the master thesis report is done!
    Here it is in all it 60 pages of splendor: https://onedrive.live.com/redir?res...3097&authkey=!AE4h0kcLLvA4pMs&ithint=file,pdf
    Unfortunately too large to attach to the post (~10MB)

    For those not mathematically inclined there is actually a much simpler popular science article describing the system as well in only 2 pages attached below as pdf.

    I will present my work on thursday at Lund University in Sweden, anyone who feels like coming by is welcome to do so :)
     

    Attached Files:

    manpower13 and ecurtz like this.
  39. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Time for an update I think. I have done some work implementing PBR-support for my system. It was actually a lot easier than I thought! I think the results are well worth it too.

    Here are some images of friends and family now in glorious PBR shaders


    The "Faces for UMA"-package on the asset store will soon be updated to include these of course. (https://www.assetstore.unity3d.com/en/#!/content/53256)

    If you would like to try it out yourself, try this webdemo: https://dl.dropboxusercontent.com/u/34265144/Faces for UMA PBR family photo/index.html
     

    Attached Files:

    Last edited: Feb 25, 2016
    LevonRavel and Venryx like this.
  40. Venryx

    Venryx

    Joined:
    Sep 25, 2012
    Posts:
    444
    Great work! I can see that being pretty fun to use in an FPS-type game. (I don't play them myself, but having faces of people you know in one would be pretty cool)
     
    Mikael-H likes this.
  41. manpower13

    manpower13

    Joined:
    Dec 22, 2013
    Posts:
    114
    wow, that looks really cool!
    Great update :)
     
    Mikael-H likes this.
  42. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Haha yeah, neither do I, but imagine the joy getting killed by the same guy over and over in some FPS and recognizing his face :)
     
  43. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Thanks! I am also working on a system to be able to create yourself online but that will take even more time before I get it working...

    For now it needs to be a manual process where I create avatars of people :)
     
  44. CrisisSystem

    CrisisSystem

    Joined:
    Oct 7, 2015
    Posts:
    124
    Man, this is fantastic stuff.

    In use, it seems it would be more practical to have colors not taken as an image capture, but instead chosen from a palette like most character creation screens. Then lighting in the room would not matter, and you wouldn't get mismatched body/face skin tones, and flattened hair on the neck for the longhaired hippies out there. Then let em' pick hairstyles & eye colors, etc etc etc. That's the only thing I could think of that might improve it.. this thing is great.
     
    Last edited: Feb 25, 2016
    Mikael-H likes this.
  45. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    haha damned longhaired hippies screwin' up my system!

    Don't worry though I'll solve the falt hair problem as well at some point :) I have plans for how to do it, just not enough hours in the day :p Actually there's a lot of important information in the actual image if you use a decent quality camera. It will capture all the little wrinkles and stuff that makes you recognize yourself as an individual. You do have a point about skin tone though, maybe you should be able to correct the skin tone of your photo...
     
    CrisisSystem likes this.
  46. CrisisSystem

    CrisisSystem

    Joined:
    Oct 7, 2015
    Posts:
    124
    Ok I think you nailed it about keeping the image capture information... maybe you could apply it to a greyscale and layer it over the top of a base skin color, which can be chosen. Hmm, well no you'd still run into issues at the seam between the neck and chest because of it. I wonder if you could choose a color from the image capture to use as a base for the rest of the mesh? (Done for the user via a script ideally)
     
  47. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    Yeah, that's actually my planned approach. If I can determine what the person's main skin tone is then I can tone the rest of the skin to that. Maybe even apply a smoothing filter at the seams.

    I've been working on a "hair-detection-filter" this week using a similar approach where I determine what the average skin color is from a central part of the face and similar for the hair. It should be possible to use this to tone the rest of the body I guess... I do all this prototyping in matlab though so I have to implement it in scripts/shaders somehow :)
     
  48. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    In case anyone is interested, here's the official link to the final master's thesis: http://lup.lub.lu.se/student-papers/record/8567678

    Should anyone be interested in doing any academic work relatting to this, do not hesitate to contact me!
    There is currently one guy, Robert, doing some very interesting work where he creates an eigenspace (a set of eigenfaces) from the resulting facial textures of my project. This will enable a texture to be broken down to a set of coordinates, each describing the amount of each eigenface to build a texture resembling the original. In Roberts early tests it looks like a 2kx1k face texture can be represented by only 15 eigenfaces which means that you will only need a set of 15 floats to describe a face. This will enable fast avatar downloads from servers and should also enable e.g. personal avatars in MMOs. As a positive side effect, if someone draws offending symbols in their faces, those symbols will not be able to be represented by eigenfaces so will not be included in the resulting texture. Me and Robert will cooperate to try and bring all of this into your hands via the asset store!

    If you cannot wait for this to reach the asset store but really want a texture of your own face right now then you can try our very very very early alpha build of the web GL app that will be used to create the avatars. Please be aware that no work has been done AT ALL to make the process easier for you as a user so you may need some trial and error :) We will work on UI later, right now we have other technical issues that need solving first.

    Here's a link to the Web GL app in case someone wants to give it a go:
    http://vikingcrew.net/Selfie_to_Avatar/index.html
    The resulting texture that you can download will work on the standard UMA2 male/female models. We will make a texture mapper to remap it to any arbitrary face model later, but, as already mentioned, right now we have other technical issues to work with :)

    The long term goal here is to create a service where a user can create their avatar in our app, and then when playing a game made by you, fellow developer, they can log in to our service and download their avatar to play in your game. If you as developer wants it you should also be able to get friends of the player into your game so they can have their friends as allies or enemies in the games. Or zombies. I would prefer zombies. We will also use the constructed eigenspace (those eigenfaces) to be able to create an infinite amount of random faces for populating random faces in the game world.

    If anyone's interested to work with us through alpha/beta please let us know! Also let us know if you'd prefer the standard texture mapping to fit the UMA2 model, DAZ, WillB's models or some other system. You can always remap but each remapping reduces quality somewhat so it would be good to use the most popular system as a base.
     
    chelnok likes this.
  49. derkoi

    derkoi

    Joined:
    Jul 3, 2012
    Posts:
    2,014
    Would love something like this for mobile. Mapping the users face on to the character in our fishing sim.
     
    Mikael-H likes this.
  50. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    281
    You know, getting possible use-cases like that is just golden for us! Thanks! May I ask what avatar tech you'd prefer? Like UMA, DAZ or something else?
     
unityunity