Question Mixing Barracuda with AR Fondation

Hi,

We are trying to run an object detector or classifier (SSD MobileNet V2 or Yolo) at the same time as being inside AR Foundation. We are basically trying to detect in what room the user is. And we would have a few questions.

From what we saw, Barracuda supports Yolo v2, but the object we are trying to detect doesn't seem to be inside the tiny-yolo v2 model we tried.

• Would it be possible to get tiny-Yolo v3 to work with Unity ? (We would prefer a tiny version because the end platform is mobile).

• We know that Barracuda is in preview, do you think we could still use it in production ? If yes, any special things to keep in mind ?

From the test we did (currently using mobilenet-v2), we are able to get a decent 30 / 25 fps on different devices (ios and android). We also tried Yolov2 but the results were quite disappointing even though the frame rate was very good. Our end goal (if possible) would be to run an Object detection model while doing AR with Ar Fondation on Android and Ios.

We used these exemples : https://classifai.net/blog/tensorflow-onnx-unity/

• We ended up not being able to build for android with both barracuda and AR foundation, it seems like AR Core doesn't support Vulkan, is there any way around that ? Could barracuda be ran at the same time as AR Core ?

• Do you think it is possible to achieve (if we train a custom tiny-yolo model with less classes) to get AR foundation and some object detection going at the same time with decent performance?

• What model would you advise us to use if we are attempting to train a custom solution (fastest detection with a small list of classes) ? (We currently considering YoloV3)

• Would we even get a better performance using a custom tiny-yolo v3 model vs a coco trained tiny-yolo v3?

• Would sharing the camera feed be an issue with Barracuda + AR Foundation ?

• If we end up trying to train a custom model, would you advise a specific tool to train in order to have the smoothest conversion to .nn or .onnx (Darknet,pytorch,...) ?

Thanks a lot, its a lot of questions but it would help us to have clarifications on all of this.

 

Our plan is to get out of preview soon, but focus of this release will be ML-Agents use cases in later updates we will be expanding on use cases outside of ML-Agents. AR and object detection is definitely one of areas many people are asking about, so we are planning to invest more time in them.

You are correct, Yolo v2 is supported already and Yolo v3 is on our radar to be investigated next.
I think in general you should expect to train these networks specifically for your game, this way getting more accurate results and and somewhat saving on network capacity if you need to detect only few classes.

Our primary interchange format is .onnx, typically people get good results when doing training in PyTorch/Tensorflow and then exporting model to .onnx. Best practice for getting your model into Barracuda is to stick to the basics in the neural net itself and move post-processing/decoding code to the Python / C#.

AR Foundation (AR Core) + Barracuda is tough one, I think it should work fine with iOS and we will investigate how to achieve their interoperability on Android.

 

I'm in the same boat. I have struggled a few hours with ARFoundation + Barracuda on Android, but it seems Barracuda doesn't work on Android with OpenGL:

In fact it gives weird inference results and then crashes on my device.
Using Vulkan is no alternative because Vulkan isn't compatible with ARCore...

@louistbdpt I think the best solution right now is to use TFLite with ARFoundation. I could successfully run object detection (SSDMobileNet V3 as well as AutoML Edgr TPU models) together with AR on a Pixel 2 XL with about 100ms inference latency. Using more cores got me to about 80ms but then the AR experience starts to lag.

 

Did you happen to get a custom model to work? Or just the already working models provided by google?

 

Custom models trained with Google Auto ML Vision

 

Just to have feature parity with object detection in ARFoundation would give me the confidence to take it up and use it in a live project starting soon. Otherwise I will have to figure out whether Google is working on Object detection on their end or if I should put together my own solution.

 

Here is a Barracuda + ARFoundation example that uses self-trained Yolo version 2 (tiny). I just noticed that Barracuda is currently product ready and thus wrote my own solution for ios and android. Unfortunately, Barracuda currently doesn't support certain operators in Yolo version > 2. Hope it helps.

 

Nice work and thank you for sharing!
How did you get it running on Android? Does Barracuda now run on OpenGL? Or ARCore with Vulkan?

 

I'm not a Unity expert, but yes we can use ARCore with Vulkan for android! For the github repo I shared, I think once we select build on android, Unity will handle these things itself.

We can also use cpu to inference as well. The latency is ok for apps if we do inference asynchronously.

 

Nice work on the project! You mentioned you got ARCore working with Android, if so, did you change anything within Project/Player Settings?

I cloned the repo and tried building for Android on my side (followed the instructions) and still get a warning from ARCore that it is incompatible with Vulkan:


Edit: to add, in Project Settings -> XR Plug-in Management, I enabled Initialize XR on Startup, as well as ARCore in Plug-in Providers under Android tab

 

Is there any update on this? AR with object detection is a really powerful use case, I already have a workaround solution with ARCore and ARKit from the links in this thread but having more official support for such a workflow in Unity would be extremely helpful!

 

Good question! This is a common issue for android and I believe changing some Unity settings can do a fix (Switching Vulkan to Auto Graphics API). I'll update the Github README for android developers. Thanks!

 

This has been super useful, I have been having trouble with implementing the Yolo v3 model into it due to the output format of Yolo v3-tiny being quite different to the output of Yolo v2-tiny.

Has anyone here had any luck implementing Yolo v3 tiny into Unity Barracuda? Ideally here where there is a nice working example from Deren https://github.com/derenlei/Unity_Detection2AR

 

I am having this error "multiples Google.protobuf.dll" when intall ARCore Extensions plugin and MLAgents.
There is any solution for this?

 

I don't know if you still need help but you can convert your YOLO tiny (2/3) model to onnx model and then imported to unity.
check this link:
https://gist.github.com/Kanyade/c8ee3b7eca6f6e470e418b018ea02e83