YOLO in-game object detection (deep learning)

https://github.com/mbaske/yolo-unity
Not exactly sure what this is useful for, but it's a fun proof of concept: Connecting Unity as a client to a Yolo Service via gRPC. The pretrained model seems to be better suited for photos & videos though, compared to recognizing game content. I guess my demo props and lighting are a little too artificial looking.

 

awesome idea

 

Does it also works for streaming webcam video feed?

 

I haven't tried that, but it should work. Unity is sending a texture to the yolo service. You'd have to replace the cam texture provided in https://github.com/mbaske/yolo-unity/blob/master/YoloUnity/Assets/Scripts/Yolo/Cam.cs with a texture from the webcam.

 

Hi, your work is very intersting. I would like to ask if i have the opportunity for detection my own objects. If yes, do you know the way?
Thanks in advance

 

I haven't look on git rep, but I love this concept.
Nice horse at T 2:29

 

Please see https://github.com/mbaske/yolo-unity/issues/1

 

Ha, are you sure it's not a horse? Seriously though, if state-of-the-art image classifiers are proven to be on par with humans in extensive benchmarks, then one might think the notion of "ground truth" becomes somewhat shaky.

 

I am sure I have seen galloping horse there. It was fast, jumping over the bench

Nothing new, but just wanted to say, deep learning challenges interesting problems. Humans fall indeed in similar traps as deep learning. Confusing sometimes one animal, with other, or think there is something which is not. Basically magicians and illusionists job

Giving us enough time and examples to learn, we getting good at recognition. I love seeing AI following the same foot steps. Sometimes a bit scary, but we shouldn't stop.


How many maximum objects your system is able to recognize at runtime?

 

I'm actually not sure how many objects can be detected simultaneously. The image classifier code sits in a DLL, not sure if it has a hard limit on the number of objects, but so far I haven't had any issues with that. The image classes which can be recognized are from a pretrained yolo model "v3-tiny" https://github.com/mbaske/yolo-unity/blob/master/YoloPretrained/YOLOv3-tiny/coco.names

 

I would expect more limitation based on CPU, rather scripts itself.
I have no intention to give a play at given moment, even I would love to, since I have other priorities atm. Problem is, I would never stop playing then
But if you find out at some point, how many things your rig is capable to run at real time, I think it will be nice info for benchmarking.

 

Because i am new to all this stuff, can you please help me
I cloned the Repo and the bulid the solution
After that it says publish the YoloServer and the run the excecutable file
How i can publish the Yoloserver, with which tool i publish it and where is the Yoloserver??

 

You can publish from Visual Studio
https://docs.microsoft.com/en-us/vi...uickstart-deploy-to-local-folder?view=vs-2019

 

Really cool. Any idea, how to get it worked on the gpu?

 

Please see https://github.com/AlturosDestinations/Alturos.Yolo#gpu-requirements-optional

 

Thank u for posting the link. But I tried to follow the steps on this link, before asking my question here . But nevertheless i didn't manage to get it worked on gpu. So i was wondering, if you or someone else get this project run on the gpu.

 

Sorry, no, I haven't run this on a GPU yet.

 

What a pity. You did a really really great job on this project. Thanks u

 

https://github.com/wojciechp6/YOLO-UnityBarracuda

 

I got the above running on a smartphone btw

 

I'm intersted by that project. With how much FPS did it run on smartphone ? And do you think this is possible to do with more recent yolo version in Unity with images from webcam ?

 

Check out NatML, it's currently in open beta and there's an open-source object detection package built on it. NatML takes advantage of ML hardware on devices, so it has much better performance than Barracuda.

 

@Lanre thanks for posting this, i'll check it out