Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

CoreML examples.

Discussion in 'iOS and tvOS' started by SAOTA, Jun 23, 2017.

  1. SAOTA

    SAOTA

    Joined:
    Feb 9, 2015
    Posts:
    220
    Hello All.

    iOS introduces some nice new tech for mobile.

    https://developer.apple.com/machine-learning/

    I have made a suggestion on Unity Feedback to get simple CoreML integration like the recent ARKit. If you've used these you'll know how simple it is to get going.

    These libraries allow many more people to access these technologies.

    Thoughts?
     
    Blarp likes this.
  2. HonorableDaniel

    HonorableDaniel

    Joined:
    Feb 28, 2007
    Posts:
    2,829
    I can't think of it's use in games off the top of my head. Got any in mind?
     
    Blarp likes this.
  3. Blarp

    Blarp

    Joined:
    May 13, 2014
    Posts:
    193
  4. L4Z3RC47

    L4Z3RC47

    Joined:
    Jun 21, 2015
    Posts:
    28
    Core ML enables things like native language processing, image recognition, facial tracking, etc... These could all be amazing tools for games (and non game applications) built in Unity. For example, what if you could issue commands just by speaking in VR. Or what if your game/app could react dynamically to the facial expressions of your user. The possibilities abound.
     
    ina and Blarp like this.
  5. steffanPL

    steffanPL

    Joined:
    Oct 9, 2014
    Posts:
    36
    The most impressive opportunity is in combining CoreML with ARKit itself- just imagine being able to place all important game elements based on actual physical settings of the user's environment. You can even create the game around mechanisms of finding something to progress in the game, i.e. find book, to read next clue -> find door -> find keys etc.
     
    Blarp likes this.
  6. mavisakal

    mavisakal

    Joined:
    Aug 26, 2017
    Posts:
    15
    Hi I've voted for core-ml integration. How many votes needed for implementing this feature in unity? The vote count is 140 now.
     
    steffanPL and Blarp like this.
  7. sticklezz

    sticklezz

    Joined:
    Oct 27, 2015
    Posts:
    33
    This is one of the best reasons why this could be useful for games

    You could make an AR shooter, sports, etc w/ this

     
    Blarp likes this.
  8. MaryamKamel

    MaryamKamel

    Joined:
    Feb 5, 2016
    Posts:
    11
    any update bout this topic?
     
    Blarp likes this.
  9. Blarp

    Blarp

    Joined:
    May 13, 2014
    Posts:
    193
    cdrakep likes this.
  10. Kevin__h

    Kevin__h

    Joined:
    Nov 16, 2015
    Posts:
    38
    I managed to implement CoreML in Unity, the only problem I have right now is that you can't use it with Unity ARkit because both use an "AVCaptureSession" (this is an objective C / swift function). And I can't find a way to access the ARKit avcapturesession so I can add my CoreML output to it. I tried accessing it through the CaptureCameraController implementation from Unity but for some reason it won't link, Symbols not found for architecture (This is a very common error with a lot of different solutions, I tried them all, none seem to work)
     
    RosieReality and Blarp like this.
  11. Kevin__h

    Kevin__h

    Joined:
    Nov 16, 2015
    Posts:
    38
    For those of you who just want CoreML without ARKit, here is a tutorial I wrote: https://medium.com/@kevinhuyskens/implementing-coreml-in-unity-e91bcf80a3c5
    It's just a quick draft and I will make it better once I have some free time, but for now it should help you get started.

    UPDATE: The tutorial now includes a Unity package that allows you to use both Unity ARKit and CoreML. I suggest using a Mobilenet model for the smoothest experience (still not very smooth on my iPhone 6S tho, getting around 11fps when analyzing the pixelbuffer with CoreML). Resnet50 only gives 3fps on my iPhone 6S. So now I'm looking for a way to multithread the analyzing, never used multithreading before so it should be fun!
     
    Last edited: Apr 11, 2018
    Blarp likes this.
  12. OREI

    OREI

    Joined:
    Mar 12, 2018
    Posts:
    2
    World needs someone like you. Thanks for the tutorial.
     
    Blarp likes this.
  13. Kevin__h

    Kevin__h

    Joined:
    Nov 16, 2015
    Posts:
    38
    I suggest keeping an eye on it since I am still updating it whenever I get some spare time. I'm also planning on adding a github project. But I can't just upload my project I'm working on right now because it's for a company. Didn't sign any NDA since all I'm doing is research but I'm hoping to get a job there so it wouldn't be smart just sharing their projects I guess ... :')

    Also I hope it's of any help to you and if you have any suggestions feel free to send them my way. I'm trying (still very far from it tho) to make a tutorial comprehensible for everyone. Always looking to improve it.
     
    Blarp likes this.
  14. adamhegedues

    adamhegedues

    Joined:
    Mar 22, 2018
    Posts:
    5
    Blarp likes this.
  15. 3d_Artist1987

    3d_Artist1987

    Joined:
    Jul 9, 2012
    Posts:
    716
    Hey,
    I download this project But when I Build in Xcode it's giving an error.
     
  16. Blarp

    Blarp

    Joined:
    May 13, 2014
    Posts:
    193
    Did you do this part:

    For image classification, the plugin uses the InceptionV3 machine learning model. This model is provided in this repository inside the MLModel folder. Add this model to your Xcode project (generated by Unity after building), by dragging the model into the project navigator. Make sure that the model is added to the Unity-iPhone build target.
     
  17. 3d_Artist1987

    3d_Artist1987

    Joined:
    Jul 9, 2012
    Posts:
    716
    Hey,
    Thanks for the reply. It's working now.
    How do I create an Ml model or train model?