Search Unity

  1. Looking for a job or to hire someone for a project? Check out the re-opened job forums.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  3. Read here for Unity's latest plans on OpenXR.
    Dismiss Notice

Feature Request AR Foundation Support for 3D Object Detection with Google MediaPipe??

Discussion in 'AR' started by ROBYER1, Dec 3, 2020.

  1. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,037
    Are there any plans to incorporate Google MediaPipe Objectron support into AR Foundation for 3d Object Detection?

    It works on Android/IOS and I even found someone has made a Unity plugin version of it for Android/Desktop with IOS support coming soon - https://github.com/homuler/MediapipeUnityPlugin

    objectron_cup_android_gpu.gif

    I was looking around for object detection solutions that were cross platform on Android/IOS due to ARCore/ARKit not having the same features for 3d object detection, mediapipe would be a fantastic useful addition to the ARFoundation tool set or even just as a standalone plugin.
     
    Last edited: Dec 3, 2020
  2. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    71
    Hi Robyer1,

    Tried your plugin and I would like to know how I can get the distance over 2 object detected ? or object detected and an 3d object in scene ? Any method to get the bounding box face ?

    Best,
     
  3. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,037
    Hi, the plugin here is not my own, you will have to contact the owner of the repository on git (I'm just the messenger)
    https://github.com/homuler/MediapipeUnityPlugin

    What I do know is that it currently only works on android, the object detection samples are limited, box tracking is in the works (you can see all of this on the main readme screen in the link above if you scroll and read).

    I have offered to help the repository owner with testing and development of the plugin but unfortunately they are not very active.

    If you need ML object detection running in AR, check out this repository from the talented @derenlei , it works with Darknet Yolo v2 tiny models converted to ONNX used in Unity Barracuda. I am hoping to get Yolo-v3-tiny models working in the example at some point also as Barracuda 1.2.0 preview now supports that also. It works on IOS and Android which is very good!
     
  4. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    71
    Thanks Robyer1 for link and reply, I try this.
     
  5. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    71
    Hey Robyer1, Tried Derenlie project with ONNX model:

    mobilenet_v2
    mobilenet_v2_labels

    but what is it exacltly INPUT_NAME and OUTPUT_NAME in Detector.cs and how to change for mobilenet ?:

    private const string INPUT_NAME = "yolov2-tiny/net1";
    private const string OUTPUT_NAME = "yolov2-tiny/convolutional9/BiasAdd";

    Best,
     
  6. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,037
    Sorry I cannot help with mobilenet, the example only works with Yolo-v2-tiny, you will have to figure out how to adapt it for your model.

    Let me know if you find a way to or what you did as I am investigating adapting Detector.cs for Yolo-v3-tiny.

    As you can see, a lot of the ground work has been laid at least for running Barracuda in AR, sticking labels on detected objects in AR, passing the AR camera image to an .onnx model.
     
  7. Protapsereseses

    Protapsereseses

    Joined:
    Nov 25, 2020
    Posts:
    1
    Maybe you know how to do surface tracking using mediapipe, I need an object to appear conditionally on the table and be fixed in space, I've been trying to find a solution for a very long time
     
  8. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,037
    Not at this time with Mediapipe in Unity. Look into using Unity AR Foundation, it is very easy to do that with AR Foundation?
     
  9. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,037
    We have now added Yolo v3 tiny support in that example repo!
     
unityunity