Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

AI OAK For Unity - Spatial AI meets the power of Unity

Discussion in 'Tools In Progress' started by gerard-luxonis, Dec 1, 2021.

  1. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Welcome to Spatial CV and Edge AI Revolution !


    What is OAK For Unity?
    OAK For Unity is native plugin for Windows, Linux and MacOS (Android support ongoing) to enable OAK devices and capabilities inside Unity.
    Main goal is bring the power of OAK devices, CV and Edge AI to Unity community to build the next generation of Spatial AI / Edge AI applications.
    • OAK For Unity Creators: Build interactive videogames and experiences with pretrained models and high-level API

    Captura de pantalla 2021-12-01 105246.png

    • OAK For Unity Developers: Productivity tools for Unity developers: VR workflows, MoCap tools (body pose and face mesh)
    • OAK For CV/AI/Robotics Developers:
      • Unity Virtual OAK camera
      • Define custom pipelines (as much as possible inside Unity with visual tools)
      • Robotics Simulation and Digital Twins
      • Synthetic dataset generation with automatic labelling
      • Deep RL with ML-Agents
    What is OAK?
    OAK is real swiss army knife for CV/AI.
    OAK devices are Depth CV (Computer Vision) + AI (Deep Learning) cameras from Luxonis powered by Intel MyriadX VPU (Vision Processing Unit) at unbeatable price tag (<200$)
    Check out OAK devices, documentation and specs on Luxonis website.

    Application examples:
    Are you building interactive experiences/installation/games where users can interact with their face, head, body, eyes-gaze and/or hands?
    Are you building health/sports applications monitoring users body pose?
    Are you building multiplayer VR experiences? Are you looking to speedup VR development?
    Are you looking/building an affordable body / face MoCap solution?
    Are you building next generation of streaming application using advanced background segmentation? Or depth/3D streaming to LookingGlass holographic display?
    Are you building AR filters thanks to face detection and face mesh?
    Are you building monitoring applications needs object detection?
    Are you building application needs OCR (Optical Character Recognition)
    Are you building robotic digital twin using CV/AI?
    …. Endless application list here ...

    Features and Roadmap:
    v1.0.0 Initial Release to AssetStore (free):
    • No code approach. Just drag and drop some prefabs.
    • Access to OAK device:
      • RGB, mono images and depth (point cloud viz currently ongoing dev).
      • About point cloud: Add support for external libraries: PCL, ...
      • Access to IMU if available and retrieve system information
      • Record and Replay capability
    • OAK Device Manager and multiple OAK devices support
    • OAK For Unity Creators: High-Level API - Unlock “Myriad” applications with predefined and ready-to-use pretrained models (with depth support if you run OAK-D family)
      • Face Detector
      • 3D Head Pose Estimation
      • Face emotion
      • Body Pose (MoveNet)
      • Hand Tracking and Gesture recognition (Blaze Hands)
      • Object detector (Tiny Yolo v3/4/5 and MobileNet SSD)
      • DeepLabv3 segmentation. Background removal.
      • Depth (only OAK-D family) - point cloud visualization
      • Example how to integrate your own pipeline
      • Integration with some paid assets (demos)
    • OAK For CV/AI/Robotics: Unity Virtual OAK Camera
    Next versions:
    • Improve Hand track
    • Face mesh and Face animation tool
    • Humanoid support with Body pose
    • Eye-gaze
    • OCR
    • More integration with paid assets
    • OpenCV and PCL support inside Unity
    • More advanced demos: combining different pretrained models
    Do you miss some models / use cases / integration with some specific paid asset? Please let us know.
    • Android support. Integration with AR.
    • End-to-end workflows for synthetic dataset generation, training and deploy
    • Integration with Unity Robotics Hub (ROS) / SystemGraph / SensorSDK
    • Integration with Unity Simulation and Unity Simulation Pro
    • Intergration with Unity ML-Agents
    • Define custom pipelines inside Unity (visual tool)

    2022-01-22: New demo menu scene added. Now easier navigate through the demos


    Some examples:
    Old video showing some possibilites with ML pretrained models:


    Body Pose:


    Hand Tracking:


    Synthetic Datasets:



    Documentation | Video Tutorials | Github | Discord
     
    Last edited: Jan 20, 2022
    DragonCoder, VinceGunday and mgear like this.
  2. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Some updates about ongoing dev

    Example of face detector and device manager:


    One of the last experiments with hand tracking:
     
  3. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Some WIP on point cloud visualization

     
  4. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Time for weekly update on OAK For Unity:

    ✔ Record and replay feature for demo scenes (replay working even without OAK device - automatic fallback when there is no device)
    ✔ Xcode project to build .bundle for MacOS (in case you want to build the unity lib)
    ✔ New demo "photobooth" using face emotion + depth to show potential uses of OAK inside Unity. Small improve pending: Load pipeline outside main ui thread and combine with background removal and blending with photobooth

     
  5. Parsec3d

    Parsec3d

    Joined:
    Apr 2, 2015
    Posts:
    27
    Hi! any news on the release in the store? or is it possible to join as a Beta tester? I already have an OAK-D
     
  6. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi ! Working on it but still bit early to confirm dates for assetstore. Before we plan to open beta on github repo (we target this month, so should start happening soon).

    To join as beta tester we'd love to get some insights from you (platform, use case, ..) to make beta testing the best for everyone. Would you mind to answer following questions?:

    - Main platform are you using? (MacOS, Win, Linux)
    - Could you explain bit more about your use case? What are you looking to build with the Unity plugin?
    - What part of the Unity plugin you find most interesting? (OAK for Creators, OAK for Developers, OAK for CV/AI)
    - Do you have experience with OAK API/SDK? What platform are you using the most? (C/C++ , Python)

    Thanks in advance
     
    VinceGunday likes this.
  7. Parsec3d

    Parsec3d

    Joined:
    Apr 2, 2015
    Posts:
    27
    Mainly looking to detect ppl and report the position back to parent stuff on top in the editor, later would be great to differentiate recognized subjects so I can parent their respective pre determined "thing"

    Platform Windows
     
  8. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Think that could fit very well on the beta.

    Are you using any specific model right now?

    As explained in the first post, the beta and first version to release on the assetstore will be focused on some pretrained models. This one could be interesting for you:
    • Object detector (Tiny Yolo v3/4/5 and MobileNet SSD)
    I will post here as soon as beta testing is available but for more agile discussion please also join our discord https://discord.gg/4hGT3AFPMZ (#unity and @Luxonis-Gerard)
     
    Last edited: Dec 27, 2021
  9. Parsec3d

    Parsec3d

    Joined:
    Apr 2, 2015
    Posts:
    27
     
  10. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    That's perfect ! Btw I meant if you're using any specific ML model for ppl detection
     
  11. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    New update. This week working on basic streams with support for point cloud and VFX thanks to the amazing work of Keijiro Takahashi

     
  12. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    And here some updates:
    ✔ Device Manager
    ✔ Basic streams demo scene (color, mono, depth and disparity)
    ✔ Point Cloud VFX demo scenes (point cloud, "matrix" effects)
    ✔ All demo scenes has support for record and replay data beside live mode
    Next:
    Working on "Start here" scene: kind of main demo menu to navigate through all demos
    Rest of pretrained demos: object detector scene in prep

    So stay tuned please follow / star repository: https://github.com/luxonis/depthai-unity and join discussion about the roadmap

    Captura de pantalla (33).png Captura de pantalla (54).png Captura de pantalla (55).png Captura de pantalla (56).png Captura de pantalla (59).png
     
    Head-Trip likes this.
  13. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    And some images of OAK-D Lite device :)

    IMG_8120.jpg IMG_8121.jpg IMG_8122.jpg
     
    Head-Trip likes this.
  14. Head-Trip

    Head-Trip

    Joined:
    Oct 21, 2016
    Posts:
    3
    Looks great! I'm very keen to try out the basic streams demo scene :)
     
  15. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi Head-Trip ! Stay tuned on the repository. What platform are you using? (MacOS, Win, Linux)
     
  16. Head-Trip

    Head-Trip

    Joined:
    Oct 21, 2016
    Posts:
    3
    Will do! I'm on Windows 10.
     
    gerard-luxonis likes this.
  17. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
  18. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hey ! new demo menu scene available.

     
    keviny likes this.
  19. matmat_35

    matmat_35

    Joined:
    Apr 20, 2018
    Posts:
    9
    Hello,

    So great! Can't wait to test the Bodypose with my OAK-D lite!
     
  20. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi ! Happy to hear that ! Please follow/star the repo https://github.com/luxonis/depthai-unity and stay tuned !
     
  21. tototek

    tototek

    Joined:
    Feb 4, 2022
    Posts:
    1
    Hola Gerard,
    Estuve viendo tu presentacion de OpenCV. Felicitaciones.Deseo hacerte algunas preguntas,puedes darme tu email ?
    Saludos Cordiales
    Marcelo

    I was watching your OpenCV presentation. Congratulations. I want to ask you some questions, can you give me your email?
     
  22. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi @tototek , you can contact me on gerard at luxonis.com or for more easy discussion on discord
     
  23. VinceGunday

    VinceGunday

    Joined:
    Mar 9, 2021
    Posts:
    5
    Thank you so much for sharing this and helping other devs.
     
  24. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi @VinceGunday ! you're welcome !
     
  25. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi ! Last week did OpenCV webinar talking about the status of Unity plugin ... now on YT
     
  26. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hey ! just quick update here. Face detector was released on the repo and object detector coming soon !


    objectdetector.png
     
    Head-Trip likes this.
  27. Parsec3d

    Parsec3d

    Joined:
    Apr 2, 2015
    Posts:
    27
    On Github??
     
  28. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
  29. unity_JlDp9BLcQQoZBw

    unity_JlDp9BLcQQoZBw

    Joined:
    Sep 14, 2022
    Posts:
    3
  30. unity_JlDp9BLcQQoZBw

    unity_JlDp9BLcQQoZBw

    Joined:
    Sep 14, 2022
    Posts:
    3
    Hi ! Could you teach me how to use my own cutomed object detection model on unity with oak !
     
  31. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi @unity_JiDUroFFBmZ0Gg ! On the repo you have an example of object detector, think should be good starting point for you.

    You need to update the depthai pipeline here:
    https://github.com/luxonis/depthai-unity/blob/main/src/predefined/ObjectDetector.cpp

    And then in Unity probably you don't need to change anything .. you can reuse this example:
    https://github.com/luxonis/depthai-...Plugins/OAKForUnity/Example Scenes/Predefined

    Regards,
     
  32. patrickMNM

    patrickMNM

    Joined:
    Nov 13, 2017
    Posts:
    2
    Hello Gerard:

    I am working on an application that requires a high quality depth camera stream to track user interaction (not body tracking). In the past I have used Azure Kinect for this method of interaction with success, and I am trying to determine whether Oak-D-Pro or RealSense is the best hardware substitute.

    A few questions:

    1) Is there a timeline for the official release of the Unity asset?

    2) In the file DaiStreams.cs it appears that stream display is set at 640 x 480. Is it possible to access the higher-resolution 1280x800 depth data from the Oak-D-Pro within Unity?

    Thank you,
    Patrick
     
  33. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi Patrick,

    Thanks for sharing details about your project.

    1) Do you mean to release to unity assetstore p.eg? If you work on Windows platform, you have pretty much initial version available on github repo. Including unity package under "Releases"

    2) I checked quickly the code of the plugin (you can find under https://github.com/luxonis/depthai-unity/blob/main/src/device/Streams.cpp and if not wrong everything is parametrized so I think it's possible to change params and get higher resolution.

    Regards,
    Gerard Espona
     
  34. patrickMNM

    patrickMNM

    Joined:
    Nov 13, 2017
    Posts:
    2
    Thanks, Gerard. Will give it a try.



     
  35. unity_JlDp9BLcQQoZBw

    unity_JlDp9BLcQQoZBw

    Joined:
    Sep 14, 2022
    Posts:
    3
    Hello Gerard:

    When I opened the scene from Oak for unity plugin the window of camera in game window became gray and the Unity Editor collapsed. Do you have any suggestion help me out the issue? Thanks!
     
  36. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    23
    Hi Gerard,

    when will your work be available for Linux?

    For our product coming mid of this year (based on Ubuntu 20.04) I have to decide between OAK-D Lite + DepthAI and Orbbec Astra + Nuitrack (where a Unity plugin for Linux is already available).

    Kind regards,
    Michael
     
  37. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi, Just now I'm revisiting the issues we had in the past with Linux. Some users did recent progress on Linux, even there are some issues present.

    I'd suggest you to check the repo and stay tuned. I will try to come back you soon here anyway
     
  38. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi ! Apologize for my very late reply here. Somehow I missed this message.

    Is this happening under Windows ?

    Please join our discord to have more direct support.
     
  39. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    23
    Ok, thanks Gerard.
     
  40. Sojkamat

    Sojkamat

    Joined:
    Mar 22, 2023
    Posts:
    1
    Hello,
    I would like to ask if you already have an approximate release date of the hand tracking feature?
    It would be great for my school project.
    Thank you in advance.

    Matyas
    CTU in Prague
     
  41. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    23
    Hi Gerard, is there any realistic date for providing the linux release? I have to decide to use OAKs (or not) until end of this month.

    Cheers,
    Michael
     
  42. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    I just tested the current version of the plugin in Linux and happy to say that there is no major issues. All the examples are working fine, also the binary export.

    The only known issue (annoying issue) we're investigating is that inside editor the second time you press play there is crash because issue detecting the previous close device. But this is only inside editor, no affecting binary export.

    Regarding the release, I will update repo explaining how to compile plugin library and even provide pre-compile library.

    Regarding your project, is related to any example of unity plugin? could you explain bit more about your product?
     
  43. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi ! currently hand tracking require bit of work for the release, even it's next example, we don't have yet ETA . Please follow the repo and will come back here when we have estimation. We plan to add some python support via tcp so potentially we can run hand tracking project from geagx.
     
  44. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    23
    Hi Gerard,
    sounds very good.

    Our product is an interactive home device (Ubuntu 20.04, AMD mainboard, OAK-D lite camera) running a Unity application which needs body position, head pose and hand tracking (just to know the hand position would be enough for the first release). It's planned to be released in late summer.
     
  45. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44
    Hi ! happy to announce that repository is updated with linux support. Pre-compiled ubuntu library and instructions how to build in case you need.

    Also bumped to latest depthai-core release fixed issue generating crash between scenes.

    Feel free to check out on the repo: https://github.com/luxonis/depthai-unity
     
    stormvision likes this.
  46. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    23
    Hi Gerard,
    thanks for your great job. I just tested it on my Linux device and everything seems to work.

    If hand tracking is added, I'll be completely happy – and with head pose it's perfect for me.

    Michael
     
  47. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    23
    Hi Gerard,
    yesterday (in my memory) the builded standalone OAK Playground application worked correctly (Ubuntu 20.04). Today it crashes, when I'm trying to start the Face Detection, Face Emotion, Body Pose and Object Detector tools. Only Basic Streams and the 3 Point Cloud tools run without problems – like all tools in the editor play mode.

    Attached you find the contents of my player.log after I tried to start the Face Detection.

    Code (CSharp):
    1. Mono path[0] = '/home/stormvision/stormvision/Unity/depthai-unity/OAKForUnity/URP/OAKapp_Data/Managed'
    2. Mono config path = '/home/stormvision/stormvision/Unity/depthai-unity/OAKForUnity/URP/OAKapp_Data/MonoBleedingEdge/etc'
    3. Preloaded 'lib_burst_generated.so'
    4. Preloaded 'libdepthai-core.so'
    5. Preloaded 'libdepthai-opencv.so'
    6. Preloaded 'libdepthai-unity.so'
    7. Display 0 'EV2730Q 27"': 1920x1920 (primary device).
    8. Display 1 '1''std::runtime_error'
    9.   what():  Cannot load blob, file at path /home/stormvision/stormvision/Unity/depthai-unity/OAKForUnity/URP/OAKapp_Data/Plugins/OAKForUnity/Models/face-detection-retail-0004_openvino_2021.2_4shave.blob doesn't exist.
    10. Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80003a5c5
    11. Obtained 23 stack frames.
    12. #0  0x007f4440d4e420 in funlockfile
    13. #1  0x007f4440b8b00b in gsignal
    14. #2  0x007f4440b6a859 in abort
    15. #3  0x007f4405437911 in __cxa_throw_bad_array_new_length
    16. #4  0x007f440544338c in std::rethrow_exception(std::__exception_ptr::exception_ptr)
    17. #5  0x007f44054433f7 in std::terminate()
    18. #6  0x007f44054436a9 in __cxa_throw
    19. #7  0x007f440600560c in dai::OpenVINO::Blob::Blob(dai::Path const&)
    20. #8  0x007f4405f087f4 in dai::node::NeuralNetwork::setBlobPath(dai::Path const&)
    21. #9  0x007f44049246f1 in createFaceDetectorPipeline(PipelineConfig*)
    22. #10 0x007f4404925195 in InitFaceDetector
    23. #11 0x000000403d73f0 in (wrapper managed-to-native) OAKForUnity.DaiFaceDetector:InitFaceDetector (OAKForUnity.PredefinedBase/PipelineConfig&)
    24. #12 0x000000403d6cd4 in OAKForUnity.DaiFaceDetector:InitDevice ()
    25. #13 0x000000403cb9b4 in System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
    26. #14 0x007f44075fd228 in mono_get_runtime_build_info
    27. #15 0x007f440779bfae in mono_runtime_invoke
    28. #16 0x007f440779d864 in mono_runtime_delegate_invoke
    29. #17 0x007f44077bb209 in mono_thread_has_sufficient_execution_stack
    30. #18 0x007f44077bb052 in mono_thread_has_sufficient_execution_stack
    31. #19 0x007f44078345c4 in GC_inner_start_routine
    32. #20 0x007f4407834257 in GC_start_routine
    33. #21 0x007f4440d42609 in start_thread
    34. #22 0x007f4440c67133 in clone
     
    Last edited: Apr 20, 2023
  48. gerard-luxonis

    gerard-luxonis

    Joined:
    Dec 1, 2021
    Posts:
    44



    Hi, as you can see in line 9
    what(): Cannot load blob, file at path /home/stormvision/stormvision/Unity/depthai-unity/OAKForUnity/URP/OAKapp_Data/Plugins/OAKForUnity/Models/face-detection-retail-0004_openvino_2021.2_4shave.blob doesn't exist


    You need to copy the folder Assets/Plugins/OAKForUnity/Models/ to /OAKapp_Data/Plugins/OAKForUnity/Models/

    It's explained on the repo README.md on the section "Export Binary"
     
  49. stormvision

    stormvision

    Joined:
    Aug 30, 2014
    Posts:
    23
    Ah, I'm sorry about that. Should have read the readme.
    Thanks
     
  50. OddTD

    OddTD

    Joined:
    May 2, 2023
    Posts:
    2
    I'm trying to get an application running using the demo code for the body pose, any idea how I can modify it to capture multiple people's poses?