[Released] Awesome ML Kit — Optimized real-time ML Models centered around computer vision

Awesome ML Kit gives the Unity developers easy way to employ optimized real-time ML models centered around computer vision.




The current set of models for the Unity Barracuda inference library includes
1. High-quality background removal/background matting based on RobustVideoMatting (optimized for Unity)
2. Depth estimation by single image
And more is planned!

Features
1. Sample scene for each model
2. Specially made visualizations with different post effects
3. Select solid color / image / video as background replacement
4. Different model versions for Quality / Speed tradeoff
5. WebCamTexture support
6. Unity VideoPlayer integration
7. AVPro Live Camera integration (recommended)

Ideal for

• Media installations
• Apps with video streaming (e.g. can be used with Agora.io)
• Kiosks

Performance
NVidia RTX3070

• Background removal: 300 FPS
• Depth estimation: 400 FPS

NVidia GTX1050Ti (mobile)

• Background removal: 50 FPS
• Depth estimation: 80 FPS

To show what's possible to do with such asset, we've created a remix of the Disclosure - Hourglass ft. LION BABE music video where all the male dance masking is done using this asset (we just discard the color information from the webcam)

Also Demolition Media Hap was used in the creation process of this experience (to sync with original video)

Note: only desktop platforms currently supported (tested on Windows DX11 and Linux Vulkan)
Unity versions: 2021/2022 and above

Get it on the Asset Store: https://assetstore.unity.com/packages/tools/ai-ml-integration/demolition-ml-kit-249251

 

A small update for all existing and new users: in most cases the Gamma color space should be selected in the project settings in order for the colors to be correct on the video. You can do that with Edit->Project Settings->Player->Other Settings->Color Space set to Gamma