Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on March 30, 2023, between 5 am & 1 pm EST, in the Performance Profiling Dev Blitz Day 2023 - Q&A forum and Discord where you can connect with our teams behind the Memory and CPU Profilers.
    Dismiss Notice

AI OAK For Unity - Spatial AI meets the power of Unity

Discussion in 'Tools In Progress' started by gerard-luxonis, Dec 1, 2021.

  1. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Welcome to Spatial CV and Edge AI Revolution !


    What is OAK For Unity?
    OAK For Unity is native plugin for Windows, Linux and MacOS (Android support ongoing) to enable OAK devices and capabilities inside Unity.
    Main goal is bring the power of OAK devices, CV and Edge AI to Unity community to build the next generation of Spatial AI / Edge AI applications.
    • OAK For Unity Creators: Build interactive videogames and experiences with pretrained models and high-level API

    Captura de pantalla 2021-12-01 105246.png

    • OAK For Unity Developers: Productivity tools for Unity developers: VR workflows, MoCap tools (body pose and face mesh)
    • OAK For CV/AI/Robotics Developers:
      • Unity Virtual OAK camera
      • Define custom pipelines (as much as possible inside Unity with visual tools)
      • Robotics Simulation and Digital Twins
      • Synthetic dataset generation with automatic labelling
      • Deep RL with ML-Agents
    What is OAK?
    OAK is real swiss army knife for CV/AI.
    OAK devices are Depth CV (Computer Vision) + AI (Deep Learning) cameras from Luxonis powered by Intel MyriadX VPU (Vision Processing Unit) at unbeatable price tag (<200$)
    Check out OAK devices, documentation and specs on Luxonis website.

    Application examples:
    Are you building interactive experiences/installation/games where users can interact with their face, head, body, eyes-gaze and/or hands?
    Are you building health/sports applications monitoring users body pose?
    Are you building multiplayer VR experiences? Are you looking to speedup VR development?
    Are you looking/building an affordable body / face MoCap solution?
    Are you building next generation of streaming application using advanced background segmentation? Or depth/3D streaming to LookingGlass holographic display?
    Are you building AR filters thanks to face detection and face mesh?
    Are you building monitoring applications needs object detection?
    Are you building application needs OCR (Optical Character Recognition)
    Are you building robotic digital twin using CV/AI?
    …. Endless application list here ...

    Features and Roadmap:
    v1.0.0 Initial Release to AssetStore (free):
    • No code approach. Just drag and drop some prefabs.
    • Access to OAK device:
      • RGB, mono images and depth (point cloud viz currently ongoing dev).
      • About point cloud: Add support for external libraries: PCL, ...
      • Access to IMU if available and retrieve system information
      • Record and Replay capability
    • OAK Device Manager and multiple OAK devices support
    • OAK For Unity Creators: High-Level API - Unlock “Myriad” applications with predefined and ready-to-use pretrained models (with depth support if you run OAK-D family)
      • Face Detector
      • 3D Head Pose Estimation
      • Face emotion
      • Body Pose (MoveNet)
      • Hand Tracking and Gesture recognition (Blaze Hands)
      • Object detector (Tiny Yolo v3/4/5 and MobileNet SSD)
      • DeepLabv3 segmentation. Background removal.
      • Depth (only OAK-D family) - point cloud visualization
      • Example how to integrate your own pipeline
      • Integration with some paid assets (demos)
    • OAK For CV/AI/Robotics: Unity Virtual OAK Camera
    Next versions:
    • Improve Hand track
    • Face mesh and Face animation tool
    • Humanoid support with Body pose
    • Eye-gaze
    • OCR
    • More integration with paid assets
    • OpenCV and PCL support inside Unity
    • More advanced demos: combining different pretrained models
    Do you miss some models / use cases / integration with some specific paid asset? Please let us know.
    • Android support. Integration with AR.
    • End-to-end workflows for synthetic dataset generation, training and deploy
    • Integration with Unity Robotics Hub (ROS) / SystemGraph / SensorSDK
    • Integration with Unity Simulation and Unity Simulation Pro
    • Intergration with Unity ML-Agents
    • Define custom pipelines inside Unity (visual tool)

    2022-01-22: New demo menu scene added. Now easier navigate through the demos


    Some examples:
    Old video showing some possibilites with ML pretrained models:


    Body Pose:


    Hand Tracking:


    Synthetic Datasets:



    Documentation | Video Tutorials | Github | Discord
     
    Last edited: Jan 20, 2022
    DragonCoder, VinceGunday and mgear like this.
  2. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Some updates about ongoing dev

    Example of face detector and device manager:


    One of the last experiments with hand tracking:
     
  3. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Some WIP on point cloud visualization

     
  4. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Time for weekly update on OAK For Unity:

    ✔ Record and replay feature for demo scenes (replay working even without OAK device - automatic fallback when there is no device)
    ✔ Xcode project to build .bundle for MacOS (in case you want to build the unity lib)
    ✔ New demo "photobooth" using face emotion + depth to show potential uses of OAK inside Unity. Small improve pending: Load pipeline outside main ui thread and combine with background removal and blending with photobooth

     
  5. Parsec3d

    Parsec3d

    Joined:
    Apr 2, 2015
    Posts:
    26
    Hi! any news on the release in the store? or is it possible to join as a Beta tester? I already have an OAK-D
     
  6. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi ! Working on it but still bit early to confirm dates for assetstore. Before we plan to open beta on github repo (we target this month, so should start happening soon).

    To join as beta tester we'd love to get some insights from you (platform, use case, ..) to make beta testing the best for everyone. Would you mind to answer following questions?:

    - Main platform are you using? (MacOS, Win, Linux)
    - Could you explain bit more about your use case? What are you looking to build with the Unity plugin?
    - What part of the Unity plugin you find most interesting? (OAK for Creators, OAK for Developers, OAK for CV/AI)
    - Do you have experience with OAK API/SDK? What platform are you using the most? (C/C++ , Python)

    Thanks in advance
     
    VinceGunday likes this.
  7. Parsec3d

    Parsec3d

    Joined:
    Apr 2, 2015
    Posts:
    26
    Mainly looking to detect ppl and report the position back to parent stuff on top in the editor, later would be great to differentiate recognized subjects so I can parent their respective pre determined "thing"

    Platform Windows
     
  8. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Think that could fit very well on the beta.

    Are you using any specific model right now?

    As explained in the first post, the beta and first version to release on the assetstore will be focused on some pretrained models. This one could be interesting for you:
    • Object detector (Tiny Yolo v3/4/5 and MobileNet SSD)
    I will post here as soon as beta testing is available but for more agile discussion please also join our discord https://discord.gg/4hGT3AFPMZ (#unity and @Luxonis-Gerard)
     
    Last edited: Dec 27, 2021
  9. Parsec3d

    Parsec3d

    Joined:
    Apr 2, 2015
    Posts:
    26
     
  10. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    That's perfect ! Btw I meant if you're using any specific ML model for ppl detection
     
  11. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    New update. This week working on basic streams with support for point cloud and VFX thanks to the amazing work of Keijiro Takahashi

     
  12. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    And here some updates:
    ✔ Device Manager
    ✔ Basic streams demo scene (color, mono, depth and disparity)
    ✔ Point Cloud VFX demo scenes (point cloud, "matrix" effects)
    ✔ All demo scenes has support for record and replay data beside live mode
    Next:
    Working on "Start here" scene: kind of main demo menu to navigate through all demos
    Rest of pretrained demos: object detector scene in prep

    So stay tuned please follow / star repository: https://github.com/luxonis/depthai-unity and join discussion about the roadmap

    Captura de pantalla (33).png Captura de pantalla (54).png Captura de pantalla (55).png Captura de pantalla (56).png Captura de pantalla (59).png
     
    Head-Trip likes this.
  13. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    And some images of OAK-D Lite device :)

    IMG_8120.jpg IMG_8121.jpg IMG_8122.jpg
     
    Head-Trip likes this.
  14. Head-Trip

    Head-Trip

    Joined:
    Oct 21, 2016
    Posts:
    3
    Looks great! I'm very keen to try out the basic streams demo scene :)
     
  15. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi Head-Trip ! Stay tuned on the repository. What platform are you using? (MacOS, Win, Linux)
     
  16. Head-Trip

    Head-Trip

    Joined:
    Oct 21, 2016
    Posts:
    3
    Will do! I'm on Windows 10.
     
    gerard-luxonis likes this.
  17. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
  18. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hey ! new demo menu scene available.

     
    keviny likes this.
  19. matmat_35

    matmat_35

    Joined:
    Apr 20, 2018
    Posts:
    9
    Hello,

    So great! Can't wait to test the Bodypose with my OAK-D lite!
     
  20. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi ! Happy to hear that ! Please follow/star the repo https://github.com/luxonis/depthai-unity and stay tuned !
     
  21. tototek

    tototek

    Joined:
    Feb 4, 2022
    Posts:
    1
    Hola Gerard,
    Estuve viendo tu presentacion de OpenCV. Felicitaciones.Deseo hacerte algunas preguntas,puedes darme tu email ?
    Saludos Cordiales
    Marcelo

    I was watching your OpenCV presentation. Congratulations. I want to ask you some questions, can you give me your email?
     
  22. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi @tototek , you can contact me on gerard at luxonis.com or for more easy discussion on discord
     
  23. VinceGunday

    VinceGunday

    Joined:
    Mar 9, 2021
    Posts:
    5
    Thank you so much for sharing this and helping other devs.
     
  24. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi @VinceGunday ! you're welcome !
     
  25. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi ! Last week did OpenCV webinar talking about the status of Unity plugin ... now on YT
     
  26. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hey ! just quick update here. Face detector was released on the repo and object detector coming soon !


    objectdetector.png
     
    Head-Trip likes this.
  27. Parsec3d

    Parsec3d

    Joined:
    Apr 2, 2015
    Posts:
    26
    On Github??
     
  28. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
  29. unity_JlDp9BLcQQoZBw

    unity_JlDp9BLcQQoZBw

    Joined:
    Sep 14, 2022
    Posts:
    3
  30. unity_JlDp9BLcQQoZBw

    unity_JlDp9BLcQQoZBw

    Joined:
    Sep 14, 2022
    Posts:
    3
    Hi ! Could you teach me how to use my own cutomed object detection model on unity with oak !
     
  31. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi @unity_JiDUroFFBmZ0Gg ! On the repo you have an example of object detector, think should be good starting point for you.

    You need to update the depthai pipeline here:
    https://github.com/luxonis/depthai-unity/blob/main/src/predefined/ObjectDetector.cpp

    And then in Unity probably you don't need to change anything .. you can reuse this example:
    https://github.com/luxonis/depthai-...Plugins/OAKForUnity/Example Scenes/Predefined

    Regards,
     
  32. patrickMNM

    patrickMNM

    Joined:
    Nov 13, 2017
    Posts:
    2
    Hello Gerard:

    I am working on an application that requires a high quality depth camera stream to track user interaction (not body tracking). In the past I have used Azure Kinect for this method of interaction with success, and I am trying to determine whether Oak-D-Pro or RealSense is the best hardware substitute.

    A few questions:

    1) Is there a timeline for the official release of the Unity asset?

    2) In the file DaiStreams.cs it appears that stream display is set at 640 x 480. Is it possible to access the higher-resolution 1280x800 depth data from the Oak-D-Pro within Unity?

    Thank you,
    Patrick
     
  33. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi Patrick,

    Thanks for sharing details about your project.

    1) Do you mean to release to unity assetstore p.eg? If you work on Windows platform, you have pretty much initial version available on github repo. Including unity package under "Releases"

    2) I checked quickly the code of the plugin (you can find under https://github.com/luxonis/depthai-unity/blob/main/src/device/Streams.cpp and if not wrong everything is parametrized so I think it's possible to change params and get higher resolution.

    Regards,
    Gerard Espona
     
  34. patrickMNM

    patrickMNM

    Joined:
    Nov 13, 2017
    Posts:
    2
    Thanks, Gerard. Will give it a try.



     
  35. unity_JlDp9BLcQQoZBw

    unity_JlDp9BLcQQoZBw

    Joined:
    Sep 14, 2022
    Posts:
    3
    Hello Gerard:

    When I opened the scene from Oak for unity plugin the window of camera in game window became gray and the Unity Editor collapsed. Do you have any suggestion help me out the issue? Thanks!
     
  36. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    2
    Hi Gerard,

    when will your work be available for Linux?

    For our product coming mid of this year (based on Ubuntu 20.04) I have to decide between OAK-D Lite + DepthAI and Orbbec Astra + Nuitrack (where a Unity plugin for Linux is already available).

    Kind regards,
    Michael
     
  37. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi, Just now I'm revisiting the issues we had in the past with Linux. Some users did recent progress on Linux, even there are some issues present.

    I'd suggest you to check the repo and stay tuned. I will try to come back you soon here anyway
     
  38. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    23
    Hi ! Apologize for my very late reply here. Somehow I missed this message.

    Is this happening under Windows ?

    Please join our discord to have more direct support.
     
  39. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    2
    Ok, thanks Gerard.
     
  40. Sojkamat

    Sojkamat

    Joined:
    Mar 22, 2023
    Posts:
    1
    Hello,
    I would like to ask if you already have an approximate release date of the hand tracking feature?
    It would be great for my school project.
    Thank you in advance.

    Matyas
    CTU in Prague