Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

NatML - Machine Learning Runtime

Discussion in 'Assets and Asset Store' started by Lanre, May 13, 2021.

  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    natml.png

    Join the NatML Beta Waitlist!

    Hello all! I'm very excited to announce NatML, the machine learning runtime for Unity Engine. NatML is a high performance, cross-platform machine learning runtime for Unity. With NatML, you can run your ONNX machine learning models across iOS, Android, macOS, and Windows. With NatML, you are guaranteed to have bare metal performance, as NatML takes advantage of hardware accelerators like the Apple Neural Engine.

    NatML is very (very) easy to use. It has the best runtime performance for machine learning, often multiple times faster than Unity's Barracuda. It is has full support for ONNX, and doesn't require writing custom code to convert models for compatibility (much unlike Barracuda). And it has tight integration with other NatSuite Framework API's like NatDevice and NatCorder, giving you the ability to create highly interactive ML-powered apps in very few lines of code.

    NatML Features:
    • High Performance Machine Learning. NatML takes advantage of the Apple Neural Engine on iOS and macOS (CoreML); the Neural Processing Unit on Android (NNAPI); and the GPU on Windows (DirectML). See the NatML benchmark against Unity Barracuda.
    • Easy Machine Learning. You will typically use NatML in just three steps: create a model; create a predictor; and make a prediction!
    • Cross Platform Machine Learning. You can run your ML models on device, in the Editor, and anywhere in between because NatML supports iOS, Android, macOS, and Windows.
    • Realtime Computer Vision. Using NatML with NatDevice, you can create apps that do realtime object classification (see the classifier example using MobileNet); realtime object detection (see the detection example using YOLO); and much more. Face detection and pose detection examples coming soon!
    • Full Layer Coverage. NatML supports the full ONNX specification, which means you can use ONNX models from any ML framework (PyTorch, Tensorflow, others) or from anywhere. No need to write conversion code.
    • Community Tools. We are developing an open-source extension library which will provide out-of-the-box support for making easy predictions with more models, all from our developer community.

    We also partner with studios to quickly and affordably create custom models that can then be used with NatML. For this and any other questions:

    Join the NatSuite Community on Discord

    Read More about NatML

    Email Us
     
    ROBYER1 and jiraphatK like this.
  2. poyapaya12

    poyapaya12

    Joined:
    Apr 20, 2020
    Posts:
    5
    it dosen't work
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Yo! What errors are you facing? Thanks for letting me know.
     
  4. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    If you were facing errors when importing the package from the Asset Store on Windows, this has been fixed.
     
  5. unity_4DD17FAC8C84B533500D

    unity_4DD17FAC8C84B533500D

    Joined:
    Jun 29, 2021
    Posts:
    1
    Iam using this asset to develop an educational sound based game , the sample scene has one dummy audio clip to test which return its equivalent generated text however when I try to record my own audio clip through microphone using Unity's Api and send that clip for prediction , it returns an empty string every time. Please help me how can I fix it.
     
  6. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hey there, I presume you are using the Silero STT predictor package? If so, then the model requires a specific audio format for the audio data. As a result, I recommend using NatDevice to stream the microphone with the required format. See the example code that uses NatDevice.
     
  7. artmole

    artmole

    Joined:
    Nov 9, 2016
    Posts:
    2
    Hello there. I am trying to use the MobileNet v2. But i am having some issues. After having imported the MobileNet v2 unity package and the NatML - Machine Learning Runtime unity package, I am met by a wall of errors, all variants of:

    Code (CSharp):
    1. Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(10,23): error CS0234: The type or namespace name 'Extensions' does not exist in the namespace 'NatSuite.ML' (are you missing an assembly reference?)
    Can you tell me what I am doing wrong?
    Have a great one.
     
  8. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Heyo, you might be using an older version of NatML. Try importing NatML 1.0.5 by modifying your `manifest.json` file in the 'Packages' folder:
    Code (CSharp):
    1. {
    2.   "scopedRegistries": [
    3.     {
    4.       "name": "NatSuite Framework",
    5.       "url": "https://registry.npmjs.com",
    6.       "scopes": ["api.natsuite"]
    7.     }
    8.   ],
    9.   "dependencies": {
    10.     "api.natsuite.natml": "1.0.5"
    11.   }
    12. }
    Let me know if this issue still exists.
     
  9. artmole

    artmole

    Joined:
    Nov 9, 2016
    Posts:
    2
    Thank! Worked like a charm. I had missed the whole JSON file setup. Sorry for the inconvinience.
     
  10. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    No worries! I'm glad to hear it worked out. Make sure to join us on Discord for quicker support.
     
  11. nola_dev

    nola_dev

    Joined:
    Oct 18, 2015
    Posts:
    3
    I saw an example where the ML model is pulled from the NatML hub in code. If we purchase a model such as MoveNet from the hub can the model get embedded in Unity application vs downloading from the hub at run-time.
     
  12. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    NatML Hub is designed to only deliver models to users at runtime. There are two reasons for this:
    1. NatML doesn't actually require running a specific ML model format (CoreML vs. TFLite vs. ONNX) and NatML Hub takes advantage of this to provide an ML model optimized specifically for the device that requests the model. You can read more here.
    2. In addition to the model graph, Hub also delivers supplementary data for the model including classification labels (for classification and detection models), normalization coefficients, and more.
    If you have your own ONNX model, you can load it from file or from StreamingAssets. But all predictors that are on NatML Hub require fetching model data from Hub.
     
  13. AmmarSalim

    AmmarSalim

    Joined:
    Sep 2, 2016
    Posts:
    25
    Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(10,23): error CS0234: The type or namespace name 'Extensions' does not exist in the namespace 'NatSuite.ML' (are you missing an assembly reference?)
    Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(11,23): error CS0234: The type or namespace name 'Features' does not exist in the namespace 'NatSuite.ML' (are you missing an assembly reference?)
    Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(12,23): error CS0234: The type or namespace name 'Internal' does not exist in the namespace 'NatSuite.ML' (are you missing an assembly reference?)
    Assets\ML\mobilenet-v2\Runtime\MobileNetv2Predictor.cs(18,48): error CS0246: The type or namespace name 'IMLPredictor<>' could not be found (are you missing a using directive or an assembly reference?)
    ......ext


    I just installed it with the latest version and the following problems appeared. I installed it on a project I previously installed opencv .
     
    Last edited: Oct 15, 2021
  14. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Make sure you are using the latest version of NatML. See the README instructions for how to install it with the package manager.
     
  15. AmmarSalim

    AmmarSalim

    Joined:
    Sep 2, 2016
    Posts:
    25
    Thank you, i have installed it successfully.

    We need a lot of explanations and examples.
    Thanks again
     
    Jamda and Lanre like this.
  16. BigToe

    BigToe

    Joined:
    Nov 1, 2010
    Posts:
    208
    Can NatML and Barracuda exist in the same project? After importing NatML, the NNModels that I had were no longer recognized as NNModels by Unity. Once I deleted NatML, they worked again.
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    NatML and Barracuda both define importers for `onnx` models, so Unity rejects them both. You'll need to use one or the other.
     
  18. ALberto_Saul

    ALberto_Saul

    Joined:
    Jul 23, 2017
    Posts:
    5
    Hi! I am using unity 2020.3.24 and when using the sample tiny-yolov3 sample it crashes when playing in editor when it gets to this line:
    IntPtr[] IMLModel.Predict (params IntPtr[] inputs) => (this as MLEdgeModel).Predict(inputs); in the MLModel.cs script.
    Any idea what could it be?
    Its just the dog image of sample project.
     
  19. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hey there, thanks for bringing this to our attention. I recommend using other object detection models, like yolox, ssd-lite, or nanodet. We've archived our current tiny-yolo-v3 model, and will be replacing it with yolo-v5.
     
  20. byfen

    byfen

    Joined:
    Jul 11, 2018
    Posts:
    5

    Attached Files:

    yanhuiru likes this.
  21. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
  22. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    We've fixed the issue! You'll need to clear your editor's predictor cache so that NatML can re-download the updated graph. To do so, print the value of `Application.persistentDataPath` in the console, then navigate there in Windows Explorer and delete the `natml` folder.

    We will add a menu option to automate this process in the next update.
     
  23. byfen

    byfen

    Joined:
    Jul 11, 2018
    Posts:
    5
    very thankful!
    But I found that no matter how big the computer memory is, it will be full.
    upload_2022-3-16_14-43-14.png upload_2022-3-16_14-43-44.png
     
  24. mchangxe

    mchangxe

    Joined:
    Jun 16, 2019
    Posts:
    69
    great stuff, thx alot. One question, im using the robust video matting package, how do i show the foreground composite image with green background instead of the black and white alpha output?

    if this is not possible in the natML hub robust video matting model,I can use the onnx model export from the robust video matting model. But, since theres no documentation on how to use the model in unity, im kinda lost what to do here:
    Screenshot 2022-03-23 193907.png
     
    Last edited: Mar 23, 2022
  25. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    You can use a custom shader for this. Have a simple shader that uses the mask to do alpha blending of the camera image on top of a green background.
    If you are using your own ML model file, you have to implement a predictor class for it. This class must implement the `IMLPredictor` interface, and must perform all the heavy lifting for passing raw tensors into the model, and extracting the result tensors from the model. We've just updated the docs on this.
     
  27. mchangxe

    mchangxe

    Joined:
    Jun 16, 2019
    Posts:
    69
    Thx, i got it to work with the first method, however, i find that the performance of the matte algorithm is really slow. I have a i9-12900H cpu and a 3070ti gpu, with webcamtexture resolution of 1280-720 im only getting a fps of around 10. The Robust Video Matting github states that the algorithm can run 4k resolution webcam texture to around 100fps. So what am i missing?
     
  28. mchangxe

    mchangxe

    Joined:
    Jun 16, 2019
    Posts:
    69
    also found that like byfen said, the memory always fills up leading to a crash on windows
     
    Lanre likes this.
  29. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hey there, I've followed up on this on Discord. But to recap, we found a bug with DirectML where initialization of the model hangs the application for a few seconds. As such, we've disabled using DirectML on Windows until we find a fix. Once it's fixed, we'll re-enable DirectML.
     
  30. robottomo

    robottomo

    Joined:
    Jan 14, 2022
    Posts:
    1
    Hello, is it possible to run your models on Hololens2?
     
  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hey there, NatML currently doesn't support the HoloLens 2. It's on our roadmap, but we don't have an ETA.
     
    robottomo likes this.
  32. byfen

    byfen

    Joined:
    Jul 11, 2018
    Posts:
    5
    upload_2022-4-14_15-43-25.png
    In NalML1.0.10 RobustVideoMattingSample Error
    meet-segmentation can work .
    I've cleaned editor's predictor cache.
     
  33. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Can you share the editor logs?
     
  34. sshimji

    sshimji

    Joined:
    Jan 12, 2022
    Posts:
    3
    Hello, I tried to run "movenet" but I got some error such as attached image.
    Should I paid for the cloud plan??

    Unity console error is like below.

    InvalidOperationException: Cannot create session because you are not on the Cloud plan. Upgrade to the Cloud plan at https://hub.natml.ai/billing
    NatSuite.ML.Hub.NatMLHub+<Request>d__7`2[TRequest,TResponse].MoveNext () (at Library/PackageCache/api.natsuite.natml@1.0.9/Runtime/Hub/NatMLHub.cs:143)
    --- End of stack trace from previous location where exception was thrown ---
    System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <695d1cc93cca45069c528c15c9fdd749>:0)
    System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) (at <695d1cc93cca45069c528c15c9fdd749>:0)
    System.Runtime.CompilerServices.TaskAwaiter`1[TResult].GetResult () (at <695d1cc93cca45069c528c15c9fdd749>:0)
    NatSuite.ML.Hub.NatMLHub+<CreateSession>d__1.MoveNext () (at Library/PackageCache/api.natsuite.natml@1.0.9/Runtime/Hub/NatMLHub.cs:46)
    --- End of stack trace from previous location where exception was thrown ---
    System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <695d1cc93
     

    Attached Files:

  35. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hey there, yes fetching any model from NatML Hub requires the Cloud plan. The alternative is to manually find, import, and implement the model yourself in Unity, which is possible but usually difficult and time-consuming.
     
  36. ooo_earth

    ooo_earth

    Joined:
    May 26, 2017
    Posts:
    4

    Hi. I already followed the installation advice including in the manifest.json the above quoted NatSuit FrameWork lines but it's not working. Here are my errors. This is a clean Unity Version 2020.3.26f1, so I don'tunderstand what is the problem. Thanks in advance.
    Erros.png
     
  37. ooo_earth

    ooo_earth

    Joined:
    May 26, 2017
    Posts:
    4
    It's not working in fact.
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Can you share your `manifest.json` in an attachment? The last error indicates that the JSON is invalid, so no packages have been imported.
     
  39. ooo_earth

    ooo_earth

    Joined:
    May 26, 2017
    Posts:
    4
    Hi. Now I have a clean error.
    I have my NatML account, all is installed in the package manager, and I pasted my key, but it says is invalid.
     

    Attached Files:

  40. ooo_earth

    ooo_earth

    Joined:
    May 26, 2017
    Posts:
    4
    Now it was correctly installed. First I installed this time the
    NatML - Machine Learning Runtime / after that I was trying with the Segmentation. So, then it updates correctly the manifest.json but now the problem it says my key is invalid. As I explained in the following question.

    ---InvalidOperationException: Cannot create session because access key is invalid. Get your access key at https://hub.natml.ai/profile -- And I'm copying it precisely from there
     

    Attached Files:

    Last edited: May 31, 2022
  41. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Hey there, I think it's best if we make this a private conversation, because we will likely need private information (your username, email) to be able to pinpoint the cause of this issue. I'll shoot you a PM.
     
  42. creat327

    creat327

    Joined:
    Mar 19, 2009
    Posts:
    1,756
    Guys, just so you know, the included examples in Unity assets store none work by default. None of them. Also they are all with a static picture. Nothing in there is using real time video or camera input. If you try to do it yourself, the programs keep crashing.
    To top it all, now you have to pay $29 monthly to use this junk. For that you just do a one time payment and use a tested library with tons of good examples.
     
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Next time, I recommend simply asking for some assistance. From what you've said, it looks like you're dissatisfied that the examples use a static image instead of a camera preview. We're going to be updating all the samples to use the NatDevice camera preview.
     
  44. creat327

    creat327

    Joined:
    Mar 19, 2009
    Posts:
    1,756
    It's not only the static image. Let me list the points:
    1) Your asset description makes it seem like you have a working real-time example included. Not a static picture of a cat.
    2) Your asset makes it seem it is free, only when you download it, and spend time figuring out how to set it up and stuff, you will find out you have to pay $29 a month to use it.
    3) Your asset will not work when you download it. You have to download additional libraries and the documentation is very bad on the steps to follow.
    4) Your asset will crash and not work properly depending on the rotation of your phone. The canvas is off the screen and you won't see anything.
    5) If you try to use a webcam and pass that image to your package, unity will crash. Since all your documentation says is "pass a webcam texture" to your image input and when I do that, the app crashes, it's very difficult to figure out what's going on.

    In short, the whole thing is BAD. And for $29 a month I would not expect something that is not even in beta stage ready. You need to improve this a lot before releasing it as a Unity asset and asking for $29 monthly.
     
    Last edited: Jul 14, 2022
    EthanFischer likes this.
  45. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    Switching between prediction on an image vs. the camera preview is trivial (see the docs). We have to improve the docs to surface topics like this, but everything works.
    The NatML API for Unity is in fact open-source; you can use it without paying a dime. What isn't free is access to our managed catalog of models. You can implement your own models and not pay anything (again, see the docs).
    The only dependency is the NatML machine learning runtime for Unity, and this dependency is explicitly stated on the Asset Store listing.
    This sounds 100% unrelated to NatML.
    Without crash logs, it's impossible to know exactly what is crashing. We don't have any open reports of crashes when making predictions with WebCamTexture.

    I appreciate the feedback. Machine learning is incredibly complex, and our entire reason for existing is making it as un-complicated as possible.
     
  46. creat327

    creat327

    Joined:
    Mar 19, 2009
    Posts:
    1,756
    You say it is open source, but the link contains only the client code. The "plugin" folder is with dlls that I see no source code for:
    NatML/Plugins/Windows at main · natmlx/NatML · GitHub
    Where is the source code for all the libraries?

    The doc says you can use your own models, but there is no documentation on how to do such a thing and all code examples are calling your server API that people need to pay for. Basically, you say "you can use it free", and provide no examples, no info, nothing. Heck, there is almost not even documentation on how to use it paying! I had to figure out all the stuff by myself because the documentation would say things like: pass an image or the camera texture.
    No examples, so I had to figure out how to convert the camera texture to a format the library would understand and then test and crash many times until I figured the way threading was handled. Again, maybe the library is OK, but with no documentation, no test cases and no real-time demos, I don't understand how you are asking $29 a month for it!
     
  47. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,970
    We don't publish the native source code for any of our plugins. We only publish the managed (C#) sources, instead of keeping them in DLL's. So if you want to be pedantic, you could call it "partially open-source".
    But there is, and I shared this link with you on Discord. See the docs on implementing custom predictors.
    This has *nothing* to do with NatML. There is no threading model, so I have no clue what you're referring to.
    Oh, I might have failed to mention this, but we do have lots of demos on GitHub. We're going to make an update to our website to add a link to each of these demos from the predictor pages on NatML Hub. Everything is either on NatML Hub, GitHub, or our docs.
     
  48. creat327

    creat327

    Joined:
    Mar 19, 2009
    Posts:
    1,756
    Well, i wouldn't call been pedantic if the main library and functions of the library are a compiled dll with no open source... and you say "you can download the code, it's open sourced". I would call it a plain lie. Following your logic Windows is open-sourced too. Since the interface for the libraries are all over the internet and well documented.

    Anyway, I just hope I clarified the points. This package is not free, it crashes by default and to make any use of it would cost $29 monthly. There are no real-time working examples either. If you made all these clear in the asset store, I would have no complaints.
     
  49. kyuhyoung

    kyuhyoung

    Joined:
    Jul 16, 2012
    Posts:
    3
  50. design888

    design888

    Joined:
    Dec 29, 2018
    Posts:
    1
    need money!!!!!!!!!!!!!!!!!!!1111 Free Free forever Cloud $29/month or $299/year Enterprise Custom pricing Current Plan Upgrade Contact Sales