Discussion Empathetic Humans in a Mobile app

I have a project that I may be working on that requires 3D models to be extremely believable and inspire empathy from the viewer. Basically, as close to the Enemies as possible on a mobile device. The reason being it's a sensitive topic and bringing a 3D character to life in the app that feels stiff or cartoonish would go against the whole point of the project, which is to humanize these lost people.

At first glance, I realize this sounds impossible and as this is for a non-profit, the budget is going to be limited. My first thought was of course MetaHumans, which would allow fairly easy human creation, but I am unsure about animation and re-targeting like I can do with Mecamin. Also, this is an AR app for mobile, and I am not sure how smooth of a process that would be with Unreal (not to mention the learning curve). I know about UMA, but have not seen examples at the quality I would be looking for.

I also realize that to do this well could make for a gigantic app in terms of file sizes, textures, etc. that make the app too bloated for people to want to download. Unity seems to shine in the mobile space and makes it somewhat easier to manage application bundle sizes. However, I have not seen examples for mobile that come anywhere near Enemies in terms of fidelity.

So, my question is, do any of you have examples of mobile games with humans that generate empathy? If not through fidelity, are there techniques using voice, acting or something else that might make up for lower quality models and shaders? Is using Unreal and Metahumans feasible for a mobile app?

I've considered suggesting video, but I'm not crazy about the file sizes that would generate and AR apps are often used away from Wifi, so streaming is not a great option. I've also thought about suggesting a Ken Burns approach, using still images and high quality audio. Any other suggestions?

 

you can relatively easily do mocap with your iphone for metahumans. And they come with plenty of LODs so the grunt work to make it at least theoretically viable for a phone is largely taken care of.

i dont know anything about actually building unreal games for mobile devices though.

I think you'd be better off going for stylized pixar type thing with big eyes and exaggerated expressions. Even the top of the line realistic characters remain in uncanny valley. Not to say they can't inspire empathy. I think empathy comes more from the writing and journey over time more so than animations.

 

There is plenty technique to bring down digital character to an acceptable level. I'm supposed to work on it but my computer died, and I currently need a HD.

Some quick observation based on my experience:
- Digital actor are hard, but the hard part is mostly the animation, especially around eyes and mouth area.
- Techniques for empathy is known forever in animation, some animation books can help you.
- There is way to retarget performance and blendshape to any face topology, but I didn't got that far yet, so I don't know.
- Acquiring said (quality) blendshape is all the works
- Performance capture can be done on the cheap with apple smartphone with sufficient quality, but you'll have to avoid extreme poses.
- Wrinkle map for animation can probably be achieve on mobile, you need a single map and compute stretch in shader (difference between raw vertex position and displaced position).
- UMA isn't a mesh solution, it's a code solution with a free placeholder mesh coming with it. The quality depend on what data you put and how you use it.
- Texture is non issue, you use techniques similar to terrain see this video from square enix, with low res map(=<512), you van have similar details than a 16k map.

I use to know a reference on mobile using unity, from before the pandemics, on twitter, but with all the rebranding and shenanigan, I'm sure if I can retrieve it.

There has been discussion about license issues, but Iclones and character creator has a panel of tools to simplify creating such character.

EDIT:
Not exactly mobile, but check the twitter feed of sakura rabbit

 

I found the reference back, but it's in korean!

https://unitysquare.co.kr/growwith/unityblog/webinarView?id=268

 

These are all great leads, thank you! I did find what looks like a viable option, using Epic's Live Link to capture audio and facial animation, and then applying that to a MetaHuman character. I found a video of a demo where that can be done in a matter of minutes. That Korean video seems to indicate something like that is possible using Unity, but I don't speak korean!

 

I shared the video because there is hint about how to make it work on mobile (notably mesh segmentation) with a workflow, they use arkits from unity, it's from 2019 too, so we probably can do better.

BUT this is a unity forum, I don't think the license of epic allow you to use their tool on unity!

 

I believe that is correct too according to the following entry in their content EULA.

https://www.unrealengine.com/en-US/eula/content

 

I do realize licensing would not permit me to use MetaHumans in Unity, so it's an either/or situation. I know the ease of cross-platform development would make the app design part much easier (for me) in Unity, but I have to decide if the work-flow model quality of MetaHumans outweighs that for this project. I don't think I'm up for making up a workflow from scratch, as I said it's not a high-budget project and that would eat up a lot of my time.

Still keeping an eye out for a Unity solution that could work. Hopefully Unity is considering bringing some of the Weta technology into a product that's easily accessible to the masses!

 

Few cents on the subject.

1. AR development in unreal is less polished, even setting it up can be difficult.
2. Cartoonishness shouldn't be a problem as cartoons are meant to be likeable. It could be animator's job.
3. Keep in mind that you can trigger empathy through means other than appearance/animation. See visual novels. A 2d character cutout with a few frames, but a well written text AND voiced by a good actor can trigger sympathy and empathetic response.

#2 and #3 depends on goal and nature of your project. "Ken Burns approach" as you described it seems to be the easiest solution.

 

You should probably consider Character creator:
https://www.youtube.com/playlist?list=PLNV5zSFadPdkRZZ7WIkctjGd3HD1q8wgE
https://www.youtube.com/playlist?list=PLNV5zSFadPdlGb3RdfizSW5WjFUnJziI0

Here is the work of a member of the unity forum @wetcircuit :

Her video are made using character creator

They have workflow for animating face

and importing into unity

 

Thanks for all of the suggestions. So far, I haven't been able to find any good examples of realistic humans in an AR app, the one I did find takes the uncanny valley to a new level! The model is fairly decent, but in AR with mobile-quality shaders in a real environment it's just super creepy.

This project can't really use cartoons, because the people that show up in AR are real people from the past, that have been overlooked or forgotten, which is the point of the app. To bring these people to life and make the audience care about them, thus the focus on empathy.

I think what will work is to recreate them in a hologram/flickering in and out style. They will look human, but the quality is flickering as if struggling to bring these into existence. That will disquise any problems with the models and animation, so I think the Character Creator models animated with iClone will be good enough. I also am confident the shaders I need could be done in Unity, my preferred platform.

Hopefully, this is also a simple enough workflow that additional animations could be added later to add more people and stories. If that's too sci-fi I think my other option is either video or a ken burns, documentary photography with a good voice over.