Recommendation on Emotion Recognition with Unity

Hello community!

I am currently working on a project that is heavily reliant on the emotion recognition technology. It is an interactive app that changes what’s happening on the screen depending on the user’s emotion. We have decided to go with Unity Engine because we want to supplement our app with graphics produced by Unity (3D meshes + shaders, particles effects, etc.) and because our development team has the best expertise with it.

I am looking for advice in terms of what we can reliably use to suggest the user emotion based on the webcam camera feed. Current prototype we have interfaces with Python Keras model via Web API, and this proves to be working somehow okay (considering that the server is on the same local machine), and you can also see how this can maybe be replaced with the inter-process communication. Please note that technically the same result can be achieved using native C# libraries which we also attempted but there seems to be the issues with compatibility and single-threaded nature of Unity.

The only issue, however, that is not dependent on the proposed way (local server, IPC or native libraries) is the quality of the model we are using in Python layer. It is an open-source model that can detect some of the emotions but is not very practical in reality. We don’t have enough time (and expertise) to train our own model that will work very well in the desired production setting.

We have also considered using some existing Web APIs which have two disadvantages: price and delay. We are also currently looking into https://www.mood-me.com/products/unity-3d-face-sdk/ but it is hard to say how effective / usable it is at the moment, we’ve just begun the conversation.

The questions I have are the following: is there something we're missing? Is there an “in-house” solution like a Unity plugin or a robust library for emotion recognition that can be used in Unity and reliably map emotions? Is there a paid solution that can work reliably well within Unity? Finally, if all other options fail, is there a clever technique to tie together Web API or some Python model with Unity that can produce the result reliably and quickly enough? The emphasis is on the responsiveness (it’s an interactive app) and reliability (extremely good at emotion detection).

Thank you for you help!

 

IronPython and ZMQ are pretty easy ways to communicate between Python and C#.

No comment on the rest.

 

Thanks, EternalAmbiguity. Our team indeed investigated this topic and IronPython came across. However, the biggest issue at hand is the quality of the emotion detection.

 

You know what they say about getting something done right.

 

You can use NamedPipe to communicate between C# and Python.
https://docs.microsoft.com/en-us/dotnet/standard/io/pipe-operations

 

Hello @sourtess.

I think I know the solution you might still be looking for. It's Banuba's Unity Face Tracking Plugin. This is an extension which supports emotion recognition and can be used to build cross-platform Face AR apps running on Android, iOS, Mac and Unity. It also helps to engage users of the app with the help of face filters, facial animation, 3D masks and live emojis.

 

Please did you find a suitable solution to the problem?