AI OAK For Unity - Spatial AI meets the power of Unity

Welcome to Spatial CV and Edge AI Revolution !​

Documentation | Video Tutorials | Github | Discord​

What is OAK For Unity?
OAK For Unity is native plugin for Windows, Linux and MacOS (Android support ongoing) to enable OAK devices and capabilities inside Unity.
Main goal is bring the power of OAK devices, CV and Edge AI to Unity community to build the next generation of Spatial AI / Edge AI applications.

• OAK For Unity Creators: Build interactive videogames and experiences with pretrained models and high-level API

• OAK For Unity Developers: Productivity tools for Unity developers: VR workflows, MoCap tools (body pose and face mesh)

• OAK For CV/AI/Robotics Developers:
  • Unity Virtual OAK camera
  • Define custom pipelines (as much as possible inside Unity with visual tools)
  • Robotics Simulation and Digital Twins
  • Synthetic dataset generation with automatic labelling
  • Deep RL with ML-Agents

What is OAK?
OAK is real swiss army knife for CV/AI.
OAK devices are Depth CV (Computer Vision) + AI (Deep Learning) cameras from Luxonis powered by Intel MyriadX VPU (Vision Processing Unit) at unbeatable price tag (<200$)
Check out OAK devices, documentation and specs on Luxonis website.

Application examples:
Are you building interactive experiences/installation/games where users can interact with their face, head, body, eyes-gaze and/or hands?
Are you building health/sports applications monitoring users body pose?
Are you building multiplayer VR experiences? Are you looking to speedup VR development?
Are you looking/building an affordable body / face MoCap solution?
Are you building next generation of streaming application using advanced background segmentation? Or depth/3D streaming to LookingGlass holographic display?
Are you building AR filters thanks to face detection and face mesh?
Are you building monitoring applications needs object detection?
Are you building application needs OCR (Optical Character Recognition)
Are you building robotic digital twin using CV/AI?
…. Endless application list here ...

Features and Roadmap:
v1.0.0 Initial Release to AssetStore (free):

• No code approach. Just drag and drop some prefabs.
• Access to OAK device:
  • RGB, mono images and depth (point cloud viz currently ongoing dev).
  • About point cloud: Add support for external libraries: PCL, ...
  • Access to IMU if available and retrieve system information
  • Record and Replay capability
• OAK Device Manager and multiple OAK devices support

• OAK For Unity Creators: High-Level API - Unlock “Myriad” applications with predefined and ready-to-use pretrained models (with depth support if you run OAK-D family)
  • Face Detector
  • 3D Head Pose Estimation
  • Face emotion
  • Body Pose (MoveNet)
  • Hand Tracking and Gesture recognition (Blaze Hands)
  • Object detector (Tiny Yolo v3/4/5 and MobileNet SSD)
  • DeepLabv3 segmentation. Background removal.
  • Depth (only OAK-D family) - point cloud visualization
  • Example how to integrate your own pipeline
  • Integration with some paid assets (demos)

• OAK For CV/AI/Robotics: Unity Virtual OAK Camera

Next versions:

• Improve Hand track
• Face mesh and Face animation tool
• Humanoid support with Body pose
• Eye-gaze
• OCR
• More integration with paid assets
• OpenCV and PCL support inside Unity
• More advanced demos: combining different pretrained models

Do you miss some models / use cases / integration with some specific paid asset? Please let us know.

• Android support. Integration with AR.
• End-to-end workflows for synthetic dataset generation, training and deploy
• Integration with Unity Robotics Hub (ROS) / SystemGraph / SensorSDK
• Integration with Unity Simulation and Unity Simulation Pro
• Intergration with Unity ML-Agents
• Define custom pipelines inside Unity (visual tool)

2022-01-22: New demo menu scene added. Now easier navigate through the demos

Some examples:
Old video showing some possibilites with ML pretrained models:

Body Pose:

Hand Tracking:

Synthetic Datasets:

Documentation | Video Tutorials | Github | Discord

 

Some updates about ongoing dev

Example of face detector and device manager:

One of the last experiments with hand tracking:

 

Some WIP on point cloud visualization

 

Time for weekly update on OAK For Unity:

✔ Record and replay feature for demo scenes (replay working even without OAK device - automatic fallback when there is no device)
✔ Xcode project to build .bundle for MacOS (in case you want to build the unity lib)
✔ New demo "photobooth" using face emotion + depth to show potential uses of OAK inside Unity. Small improve pending: Load pipeline outside main ui thread and combine with background removal and blending with photobooth

 

Hi! any news on the release in the store? or is it possible to join as a Beta tester? I already have an OAK-D

 

Hi ! Working on it but still bit early to confirm dates for assetstore. Before we plan to open beta on github repo (we target this month, so should start happening soon).

To join as beta tester we'd love to get some insights from you (platform, use case, ..) to make beta testing the best for everyone. Would you mind to answer following questions?:

- Main platform are you using? (MacOS, Win, Linux)
- Could you explain bit more about your use case? What are you looking to build with the Unity plugin?
- What part of the Unity plugin you find most interesting? (OAK for Creators, OAK for Developers, OAK for CV/AI)
- Do you have experience with OAK API/SDK? What platform are you using the most? (C/C++ , Python)

Thanks in advance

 

Mainly looking to detect ppl and report the position back to parent stuff on top in the editor, later would be great to differentiate recognized subjects so I can parent their respective pre determined "thing"

Platform Windows

 

Think that could fit very well on the beta.

Are you using any specific model right now?

As explained in the first post, the beta and first version to release on the assetstore will be focused on some pretrained models. This one could be interesting for you:

• Object detector (Tiny Yolo v3/4/5 and MobileNet SSD)

I will post here as soon as beta testing is available but for more agile discussion please also join our discord https://discord.gg/4hGT3AFPMZ (#unity and @Luxonis-Gerard)

 

That's perfect ! Btw I meant if you're using any specific ML model for ppl detection

 

New update. This week working on basic streams with support for point cloud and VFX thanks to the amazing work of Keijiro Takahashi

 

And here some updates:
✔ Device Manager
✔ Basic streams demo scene (color, mono, depth and disparity)
✔ Point Cloud VFX demo scenes (point cloud, "matrix" effects)
✔ All demo scenes has support for record and replay data beside live mode
Next:
Working on "Start here" scene: kind of main demo menu to navigate through all demos
Rest of pretrained demos: object detector scene in prep

So stay tuned please follow / star repository: https://github.com/luxonis/depthai-unity and join discussion about the roadmap

 

And some images of OAK-D Lite device

 

Looks great! I'm very keen to try out the basic streams demo scene

 

Hi Head-Trip ! Stay tuned on the repository. What platform are you using? (MacOS, Win, Linux)

 

Will do! I'm on Windows 10.

 

Time for quick update. Basic streams and point cloud vfx available on repository: https://github.com/luxonis/depthai-unity
for Windows. Working on MacOS and Linux support

 

Hey ! new demo menu scene available.

 

Hello,

So great! Can't wait to test the Bodypose with my OAK-D lite!

 

Hi ! Happy to hear that ! Please follow/star the repo https://github.com/luxonis/depthai-unity and stay tuned !

 

Hola Gerard,
Estuve viendo tu presentacion de OpenCV. Felicitaciones.Deseo hacerte algunas preguntas,puedes darme tu email ?
Saludos Cordiales
Marcelo

I was watching your OpenCV presentation. Congratulations. I want to ask you some questions, can you give me your email?

 

Hi @tototek , you can contact me on gerard at luxonis.com or for more easy discussion on discord

 

Thank you so much for sharing this and helping other devs.

 

Hi @VinceGunday ! you're welcome !

 

Hi ! Last week did OpenCV webinar talking about the status of Unity plugin ... now on YT

 

Hey ! just quick update here. Face detector was released on the repo and object detector coming soon !

 

On Github??

 

Hi @Parsec3d Yes you can find on GitHub (https://github.com/luxonis/depthai-unity/tree/beta)

 

Hi !

 

Hi ! Could you teach me how to use my own cutomed object detection model on unity with oak ！

 

Hi @unity_JiDUroFFBmZ0Gg ! On the repo you have an example of object detector, think should be good starting point for you.

You need to update the depthai pipeline here:
https://github.com/luxonis/depthai-unity/blob/main/src/predefined/ObjectDetector.cpp

And then in Unity probably you don't need to change anything .. you can reuse this example:
https://github.com/luxonis/depthai-...Plugins/OAKForUnity/Example Scenes/Predefined

Regards,

 

Hello Gerard:

I am working on an application that requires a high quality depth camera stream to track user interaction (not body tracking). In the past I have used Azure Kinect for this method of interaction with success, and I am trying to determine whether Oak-D-Pro or RealSense is the best hardware substitute.

A few questions:

1) Is there a timeline for the official release of the Unity asset?

2) In the file DaiStreams.cs it appears that stream display is set at 640 x 480. Is it possible to access the higher-resolution 1280x800 depth data from the Oak-D-Pro within Unity?

Thank you,
Patrick

 

Hi Patrick,

Thanks for sharing details about your project.

1) Do you mean to release to unity assetstore p.eg? If you work on Windows platform, you have pretty much initial version available on github repo. Including unity package under "Releases"

2) I checked quickly the code of the plugin (you can find under https://github.com/luxonis/depthai-unity/blob/main/src/device/Streams.cpp and if not wrong everything is parametrized so I think it's possible to change params and get higher resolution.

Regards,
Gerard Espona

 

Thanks, Gerard. Will give it a try.

 

Hello Gerard:

When I opened the scene from Oak for unity plugin the window of camera in game window became gray and the Unity Editor collapsed. Do you have any suggestion help me out the issue? Thanks!

 

Hi Gerard,

when will your work be available for Linux?

For our product coming mid of this year (based on Ubuntu 20.04) I have to decide between OAK-D Lite + DepthAI and Orbbec Astra + Nuitrack (where a Unity plugin for Linux is already available).

Kind regards,
Michael

 

Hi, Just now I'm revisiting the issues we had in the past with Linux. Some users did recent progress on Linux, even there are some issues present.

I'd suggest you to check the repo and stay tuned. I will try to come back you soon here anyway

 

Hi ! Apologize for my very late reply here. Somehow I missed this message.

Is this happening under Windows ?

Please join our discord to have more direct support.

 

Ok, thanks Gerard.

 

Hello,
I would like to ask if you already have an approximate release date of the hand tracking feature?
It would be great for my school project.
Thank you in advance.

Matyas
CTU in Prague

 

Hi Gerard, is there any realistic date for providing the linux release? I have to decide to use OAKs (or not) until end of this month.

Cheers,
Michael

 

I just tested the current version of the plugin in Linux and happy to say that there is no major issues. All the examples are working fine, also the binary export.

The only known issue (annoying issue) we're investigating is that inside editor the second time you press play there is crash because issue detecting the previous close device. But this is only inside editor, no affecting binary export.

Regarding the release, I will update repo explaining how to compile plugin library and even provide pre-compile library.

Regarding your project, is related to any example of unity plugin? could you explain bit more about your product?

 

Hi ! currently hand tracking require bit of work for the release, even it's next example, we don't have yet ETA . Please follow the repo and will come back here when we have estimation. We plan to add some python support via tcp so potentially we can run hand tracking project from geagx.

 

Hi Gerard,
sounds very good.

Our product is an interactive home device (Ubuntu 20.04, AMD mainboard, OAK-D lite camera) running a Unity application which needs body position, head pose and hand tracking (just to know the hand position would be enough for the first release). It's planned to be released in late summer.

 

Hi ! happy to announce that repository is updated with linux support. Pre-compiled ubuntu library and instructions how to build in case you need.

Also bumped to latest depthai-core release fixed issue generating crash between scenes.

Feel free to check out on the repo: https://github.com/luxonis/depthai-unity

 

Hi Gerard,
thanks for your great job. I just tested it on my Linux device and everything seems to work.

If hand tracking is added, I'll be completely happy – and with head pose it's perfect for me.

Michael

 

Hi Gerard,
yesterday (in my memory) the builded standalone OAK Playground application worked correctly (Ubuntu 20.04). Today it crashes, when I'm trying to start the Face Detection, Face Emotion, Body Pose and Object Detector tools. Only Basic Streams and the 3 Point Cloud tools run without problems – like all tools in the editor play mode.

Attached you find the contents of my player.log after I tried to start the Face Detection.

Code (CSharp):

1. Mono path[0] = '/home/stormvision/stormvision/Unity/depthai-unity/OAKForUnity/URP/OAKapp_Data/Managed'
2. Mono config path = '/home/stormvision/stormvision/Unity/depthai-unity/OAKForUnity/URP/OAKapp_Data/MonoBleedingEdge/etc'
3. Preloaded 'lib_burst_generated.so'
4. Preloaded 'libdepthai-core.so'
5. Preloaded 'libdepthai-opencv.so'
6. Preloaded 'libdepthai-unity.so'
7. Display 0 'EV2730Q 27"': 1920x1920 (primary device).
8. Display 1 '1''std::runtime_error'
9.   what():  Cannot load blob, file at path /home/stormvision/stormvision/Unity/depthai-unity/OAKForUnity/URP/OAKapp_Data/Plugins/OAKForUnity/Models/face-detection-retail-0004_openvino_2021.2_4shave.blob doesn't exist.
10. Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80003a5c5
11. Obtained 23 stack frames.
12. #0  0x007f4440d4e420 in funlockfile
13. #1  0x007f4440b8b00b in gsignal
14. #2  0x007f4440b6a859 in abort
15. #3  0x007f4405437911 in __cxa_throw_bad_array_new_length
16. #4  0x007f440544338c in std::rethrow_exception(std::__exception_ptr::exception_ptr)
17. #5  0x007f44054433f7 in std::terminate()
18. #6  0x007f44054436a9 in __cxa_throw
19. #7  0x007f440600560c in dai::OpenVINO::Blob::Blob(dai::Path const&)
20. #8  0x007f4405f087f4 in dai::node::NeuralNetwork::setBlobPath(dai::Path const&)
21. #9  0x007f44049246f1 in createFaceDetectorPipeline(PipelineConfig*)
22. #10 0x007f4404925195 in InitFaceDetector
23. #11 0x000000403d73f0 in (wrapper managed-to-native) OAKForUnity.DaiFaceDetector:InitFaceDetector (OAKForUnity.PredefinedBase/PipelineConfig&)
24. #12 0x000000403d6cd4 in OAKForUnity.DaiFaceDetector:InitDevice ()
25. #13 0x000000403cb9b4 in System.Threading.ExecutionContext:Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool)
26. #14 0x007f44075fd228 in mono_get_runtime_build_info
27. #15 0x007f440779bfae in mono_runtime_invoke
28. #16 0x007f440779d864 in mono_runtime_delegate_invoke
29. #17 0x007f44077bb209 in mono_thread_has_sufficient_execution_stack
30. #18 0x007f44077bb052 in mono_thread_has_sufficient_execution_stack
31. #19 0x007f44078345c4 in GC_inner_start_routine
32. #20 0x007f4407834257 in GC_start_routine
33. #21 0x007f4440d42609 in start_thread
34. #22 0x007f4440c67133 in clone

 

Hi, as you can see in line 9
what(): Cannot load blob, file at path /home/stormvision/stormvision/Unity/depthai-unity/OAKForUnity/URP/OAKapp_Data/Plugins/OAKForUnity/Models/face-detection-retail-0004_openvino_2021.2_4shave.blob doesn't exist


You need to copy the folder Assets/Plugins/OAKForUnity/Models/ to /OAKapp_Data/Plugins/OAKForUnity/Models/

It's explained on the repo README.md on the section "Export Binary"

 

Ah, I'm sorry about that. Should have read the readme.
Thanks

 

I'm trying to get an application running using the demo code for the body pose, any idea how I can modify it to capture multiple people's poses?