Question Running TF/PyTorch Model in AR Foundation

Hey,

I am trying to build an AR Foundation App for Android and IOS. For our app, we need to estimate a few things on camera-frames. Our idea is to use a TFLite model in order to do these estimations.

Are there any example projects using TFLite for camera frame processing tasks? Or is there an alternative framework with better unity integration (e.g. PyTorch mobile)?

As I understand, ML-agents is not quite what we are looking for, since it is about training game-agents. Is ML-Agents still suitable for inference-only on mobile devices? Is it possible to run external models (ONNX, TFlite, etc) within unity?

 

You can run onnx models inside unity using mlagents. The model needs to have a specific input and output format, that might not be clearly documented, and you might need to dig around in the code and/or experiment a little, to find out what that format is. mlagents is built around the concept of taking an action given an observation, but there seems no obvious reason why your camera output image couldn't be the 'observation', and the 'few things you need to estimate' could be your actions. I believe that actions can be discrete or continuous, which covers many things.

You'll need to create an 'Agent' script in a game object, which creates a 'Behavior' component in the inspector. You can then modify the geometry of your observations and actions there.

If it was me, I'd probably set up an agent with the geometry I want, then use mlagents enough to create a randomly initialized network and save it out, then see if I can find a way to inspect the geometry of that saved onnx file. Then use that for my own network.

I haven't tried any of the above, but I see no obvious reason why it wouldn't work.

That said, having written all of that, you probably might want to just skip straight to Barracuda: https://docs.unity3d.com/Packages/com.unity.barracuda@1.0/manual/index.html

This page looks good: https://docs.unity3d.com/Packages/com.unity.barracuda@1.0/manual/GettingStarted.html