Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[RELEASED] OpenCV for Unity

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Oct 30, 2014.

  1. kirbyderby2000

    kirbyderby2000

    Joined:
    Apr 28, 2017
    Posts:
    35
    I can tell that this package is excellent so far just by playing with the demo scenes, but my only gripe is that there is no thorough documentation or walk-through to using the API. I see that this package refers us to the Java OpenCV documentation (since this is essentially a clone of it), but even that documentation is not very thorough. For example, I'm trying to figure out what the heck the MatOfPoint2f class does but the Java OpenCV documentation doesn't even try to explain what the purpose of it is: https://docs.opencv.org/3.4/javadoc/org/opencv/core/MatOfPoint2f.html

    The Python and C++ API documentation of OpenCV seems to be much more thorough but there are some things that aren't inclusive to the Java and this C# API (like the MatOfPoint2f class example I gave), so it kind of bothers me that I have to look at the code that I wanted abstracted away in order to understand how to use it.

    I'm following code in the example scenes to the best of my ability, but there are some things I just don't understand. I'm breaking it all down on my own but I'd suggest providing more comments in the demo sample code explaining what's going on to help us understand how to use the API.
     
    Last edited: Jul 23, 2020
  2. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Unfortunately, the trial version only works with UnityEditor.
     
  3. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Thank you for your input.
    I plan to add explanatory comments to the Example code in the future.
     
  4. Leanhope

    Leanhope

    Joined:
    Jul 4, 2020
    Posts:
    2
    Hello. For a project, I placed ArUco markers on cubes, placed in front of a camera. In Unity, I recreated the setup, with the location of the Cubes and the ARCamera being the same as in real life, the camera being the origin. I used the OpenCVForUnity package with the MarkerBased AR Example to attach the respective ArUco markers to the virtual cubes in the scene. In my expecation, since the distances in the scene are the same as in real life, the cubes should stay in exactly the same position, unless they are moved in real life, in which case the virtual cubes move the same distance.

    Now my expectations were not met. While the virtual cubes move if the real cubes move, the markers are detected at entirely different locations as where the cubes are currently, so the virtual cubes are at "wrong" locations in the scene. This is not a small error, the results are more than 10 units off in Unity.

    It seems to me, like there is some conversion from one coordinate system to the other that I'm missing here, but I'm not sure which one. If you need any more information I will try to provide anything necessary.

    Thank you in advance!

    EDIT: Here some screenshots to help visualize the problem. The smaller cube should stay at the same position in play mode.

    sc1.png sc2.png
     
    Last edited: Jul 29, 2020
  5. fm64

    fm64

    Joined:
    Oct 30, 2018
    Posts:
    3
    Hi, I have a question about the Mask R-CNN implementation. My goal this time is to take body measurements from pictures (similar to what Bodygram does https://bodygram.com/en/ ) so I need an accurate mask for a single person (no other classes needed), however since mask RCNN has other classes and I assume is being run on CPU from Unity, the accuracy is very poor (see picture). Although I recognize comparing it to google colab is pretty unfair lol.

    So my question are:
    - what backbone is MarkRCNN using from the plugin? (resnet, mobilenet, etc) is it possible to adjust that?
    - if I manage to create my own model based on mask_rcnn_coco.h5 or a custom .h5, convert it to .pb and .pbtxt, will the accurary increase? (I'm still studying about this so I have an inkling on how to do it but it will take a lot of time and my boss wants this for like next week)
    - Is there any other way to enhance accuracy for a particular class in the model from the code? I'm open to suggestions.

    I also tried with image operations (canny, grabcut, contours) but since I can't control the background the user will take the photo in, results weren't great either.

    Thank you.
     

    Attached Files:

    • test.png
      test.png
      File size:
      946.7 KB
      Views:
      374
  6. masterchop

    masterchop

    Joined:
    Oct 9, 2015
    Posts:
    39
  7. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Unfortunately I don't have expert knowledge of AR, so I don't know if the location of the cube on Unity matches that of the real cube.
     
  8. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    In order to load matterport's MarkRCNN model with OpenCV's Dnn module, you need to convert 'mask_rcnn_coco.h5' to a format such as .pb/.pbtxt or .onnx.
    A lot of people are working on the same problem, but I don't think we've found a solution to it.
    https://answers.opencv.org/question/206589/readnetfromtensorflow-errors-from-mask_rcnn-model/
    https://answers.opencv.org/question/224764/run-converted-pb-model-by-using-opencvs-dnn-module/
     
  9. fahruz

    fahruz

    Joined:
    Mar 5, 2020
    Posts:
    57
    This is quite complicated and cumbersome, not to mention the article of the second link is in Japanese.
    It would be really great if you could include precompiled CUDA/cuDNN-enabled DLLs (for instance in your Extra package) and provide simple instructions. That would make your customers' life much easier.
     
  10. fahruz

    fahruz

    Joined:
    Mar 5, 2020
    Posts:
    57
    So I compiled OpenCV with CUDA support, added all the generated DLLs to the "Plugins>Windows>x86_64" folder of OpenCVForUnity, swapped the original opencvforunity.dll with the one from dll_version in the Extra package, but when I run the project, I get a DllNotFoundException:
    DllNotFoundException: opencvforunity
    OpenCVForUnity.CoreModule.Mat..ctor (System.Int32 rows, System.Int32 cols, System.Int32 type) (at Assets/OpenCVForUnity/org/opencv/core/Mat.cs:512)​
    My platform target for all the DLLs is set to x86_64 and "Any Platform", i.e. Editor + Standalone, as with the original opencvforunity.dll. Am I missing something?
     
  11. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    In order to display the native opencv's error code, please enclose the code in Utils.setDebugMode(true) and Utils.setDebugMode(false). Errors are displayed in the console?

    Utils.setDebugMode(true);

    ------------

    Utils.setDebugMode(false);
     
  12. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    This issue seems to occur in Unity 2019.3 and higher versions when CUDA backend is enabled. Unfortunately, the cause is under investigation.
     
  13. fahruz

    fahruz

    Joined:
    Mar 5, 2020
    Posts:
    57
    Thanks for the quick reply. The error also occurs in Unity 2018.

    I checked if there were any missing dependencies with the "light" opencvforunity.dll, but that wasn't the case. Everything that's needed seems to be in the x86_64 folder.
     
  14. kirbyderby2000

    kirbyderby2000

    Joined:
    Apr 28, 2017
    Posts:
    35
    Okay. After a couple days of following OpenCV tutorials, I've come to the conclusion that the trackers in this Unity package are not very good. It's pretty bizarre because I followed some tutorials in Python using OpenCV and got pretty decent tracking results; however when I try to recreate the same tutorial using the same video / webcam environment in C# using this Unity OpenCV package, the results are completely different. Many of the trackers in this package don't even work on my end. Here's a list of the tracker modules that have not successfully followed / found a target, not even once for a single frame:
    • CSRT, KCF, and MOSSE.
    It's pretty unfortunate because KCF was my favorite tracker of all the ones I've played with in the native package. What do I need to do to get this tracker to work in this package?

    The only trackers that did work for me were the following:
    • Boosting, MedianFlow, MIL, and TLD.
    However, all of these trackers didn't work very well for me at all. MedianFlow's tracking worked the worst for me (lot's of false positive tracking) however it had the best performance on my computer (got 95 FPS while using it). Boosting, MIL, and TLD worked okay but they all dropped my FPS between 5 - 15. This is particularly unusual because some of these trackers weren't performance heavy when I used them in Python with the native package.


    For some perspective on my hardware:

    CPU: Intel i5-8600K
    GPU: GeForce GTX 1060 6GB
    RAM: 16 GB DDR4 (8 GB Dual Channeled)
     
    DoAugMenRo likes this.
  15. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Could you tell me the environment you tested?
    Unity version :
    OpenCVforUnity version :
     
  16. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    An example of the KCF tracker is included with OpenCVForUnity. Have you tried it already?
    https://github.com/EnoxSoftware/Ope...s/tracking/TrackingExample/TrackingExample.cs
     
  17. dinosb

    dinosb

    Joined:
    May 24, 2017
    Posts:
    1
    Hello. It takes too long to create a custom dictionary of aruco markers so the app freezes while the dictionary is being created. Is there a way to load a custom dictionary of pre-created aruco markers?
     
  18. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Unfortunately, there doesn't seem to be a wayto load a custom dictionary of pre-created aruco markers.
     
  19. sidbhise

    sidbhise

    Joined:
    May 13, 2020
    Posts:
    7
    What is best way to try out the trial version in OpenCV for Unity plugin for editor before purchasing it?
     
  20. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    You can download the trial version from the page below and try the included example code.
    https://enoxsoftware.com/opencvforunity/get_asset/
     
  21. sidbhise

    sidbhise

    Joined:
    May 13, 2020
    Posts:
    7
    I did download the trial version. It has several directories with the with just 3 (asset, pathname)files in it. Is there a documentation to refer to get started. Not sure how do I use the downloaded files. Thanks for the support.
     
  22. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Could you unzip OpenCVForUnity2.4.0TrialVersion.zip, then import OpenCVForUnity2.4.0TrialVersion.unitypackage into your project ?
    https://github.com/EnoxSoftware/OpenCVForUnity/blob/master/Assets/OpenCVForUnity/ReadMe.pdf
     
  23. WashyRules

    WashyRules

    Joined:
    Apr 12, 2015
    Posts:
    3
    Hi, i am having trouble reading an IP camera mjpg stream. I get the error: WindowsVideoMedia error 0xc00d36c4 while reading http://IP_ADDRESS/mjpg/video.mjpg, wich i think i can solve with the ffmpeg.dll as @EnoxSoftware said 2 years ago in this same thread. So, since i have the latest OpenCVForUnity, wich has opencv 4.4.0, i downloaded the appropiate dll (opencv_videoio_ffmpeg440_64.dll) and pasted it in the root directory of the project and also tried in OpenCVForUnity/Plugins/Windows/x86_64 but i had no luck.
    The url works with the browser and VLC.

    The code is just:
    Code (CSharp):
    1. VideoCapture capture = new VideoCapture("http://IP_ADDRESS/mjpg/video.mjpg");
    2.  
    3. if (capture.isOpened())
    4. {
    5.     Debug.Log("capture.isOpened() true");
    6. }
    7. else
    8. {
    9.     Debug.Log("capture.isOpened() false");
    10. }
    and i get "capture.isOpened() false"
     
  24. fahruz

    fahruz

    Joined:
    Mar 5, 2020
    Posts:
    57
    Hello,

    I have a Mat that is continuously being refreshed and converted to a Texture2D using the fastMatToTexture2D method in Utils. I update the texture of a UI object with the result, but it creates a memory leak because new textures are constantly being created. It seems like I should be using a RenderTexture for this, but I'm not sure how I can get a RenderTexture with the methods available in Utils. Do you have some examples showing how to achieve what I'm describing without causing memory leaks?
     
  25. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    To play streaming file, ffmpeg.dll is required.
    1)Download "OpenCV for Windows Version 4.4.0"(https://sourceforge.net/projects/opencvlibrary/).
    2)Copy "opencv_videoio_ffmpeg440_64.dll" to the root directory of the Project.

    I succeeded in playing this file.
    capture.open ("http://archive.org/download/SampleMpeg4_201307/sample_mpeg4.mp4");
    Did you succeed in playing this file?
     
  26. kirbyderby2000

    kirbyderby2000

    Joined:
    Apr 28, 2017
    Posts:
    35
  27. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Does your code generate a new Texture2D on every frame?
     
  28. YUUKI_KOJIN

    YUUKI_KOJIN

    Joined:
    Aug 6, 2017
    Posts:
    2
    Hello!
    Is there a sample that cuts out the red areas of the four corners?
    *For clarity, the parts you want to cut out are shown in red.
    It seems like we need to mess with the "DocumentScannerExample", but I'll ask you a question.
    IMG_1313.jpeg
     
  29. fahruz

    fahruz

    Joined:
    Mar 5, 2020
    Posts:
    57
    Pretty much, yes.
     
  30. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    As you say, the file seems to have been excluded from the latest repository.
    Could you please get the MobileNetSSD_deploy.prototxt from the following link
    https://raw.githubusercontent.com/c...05da4bf1000b2836/MobileNetSSD_deploy.prototxt
     
  31. WashyRules

    WashyRules

    Joined:
    Apr 12, 2015
    Posts:
    3
    Yes, i succeeded in playing that file, and as i said in my previous message, i already did what you say about copying the "opencv_videoio_ffmpeg440_64.dll" to the root directory of the project, but i still get the same error about codecs when i try with my ip camera video stream. I think its beacuse of the .mjpg format, could you confirm me that you can also open urls with video streams of the .mjpg extension? You can try with this one for example: https://filesamples.com/samples/video/mjpeg/sample_960x540.mjpeg
     
  32. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    This video played fine in my environment.
    https://filesamples.com/samples/video/mjpeg/sample_960x540.mjpeg
    videocapture_mjpeg.PNG
     
  33. pzo

    pzo

    Joined:
    Aug 27, 2020
    Posts:
    3
    Hi I'm testing few technologies (Unity, UE4, Qt) for my project and wondering if Unity + NatDevice + OpenCV4Unity would be a good fit. I have few questions.

    1) I have few private opencv modules. Ideally I would prefer to avoid moving them outside opencv_contrib and neither duplicating depending headers and nor embedding 2 different versions of opencv. Does this OpenCV4Unity asset provide source code or only binary blobs? If not is there any example that show how to write basic opencv module that would link to OpenCV4Unity headers and libraries?
    2) I'm thinking of trying integrating MediaPipe to unity. Since MediaPipe uses opencv as 3rd party (but modified without buildin opencv dnn module and protobuf module - because it embeds different version of tensorflow and protofbuf). Again I'm just wondering if there is source code of OpenCV4Unity included or if there is maybe plans for seperate MediaPipe plugin. Otherwise again I would have include 2 versions of opencv in unity projects which is far from ideal. Seems like there is demand for mediapipe for unity - just an idea ;)
    https://github.com/google/mediapipe/issues/36
    3) Does Opencv4Unity integration with PlayMaker only provides WebcamTexture template or is there also template for camera capture via NatDevice?
     
  34. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    It is recommended to reuse the previously initialized Texture2D class every frame.
     
  35. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    I think this could be accomplished by changing these lines to code that only detects red color.
    https://github.com/EnoxSoftware/Ope...erExample/DocumentScannerExample.cs#L175-L179

    https://stackoverflow.com/questions/32522989/opencv-better-detection-of-red-color
     
    YUUKI_KOJIN likes this.
  36. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Thank you for your inquiry.
    1) OpenCVForUnity asset provide only binary blobs.
    2) There are currently no plans for a MediaPipe plugin.
    3) OpenCVForUnity integration with PlayMaker only provides WebcamTexture template.
     
  37. nguyennh2

    nguyennh2

    Joined:
    Apr 24, 2020
    Posts:
    1
    Hi @EnoxSoftware,

    I'm using Ubuntu 20.04 with unity 2020.1.4f.

    Testing LibFaceDetectionV2Example, opencvforunity verion 2.4.0 ( with opencv 4.4.0) performs very poor, only 3fps. But with the previous version of 2.3.0 (with opencv 4.3.0 ), I got 129fps. Suprised!

    I think something wrong with dnn module of new version. Please check it.
     
  38. runtodev

    runtodev

    Joined:
    Nov 30, 2012
    Posts:
    4
    Hi, EnoxSoftware

    I have a plan to use matchTemplate function of opencv.

    1. Can I use this function to find match images regardless of image colors? (I found it's ok in a specific method of matching template)

    2. how to crop matched area of the image and save it to image file in unity3d?

    Thanks in advance.
     
  39. aguroshou4620502

    aguroshou4620502

    Joined:
    Jun 28, 2020
    Posts:
    3
    Dear EnoxSoftware.
    I want to make a WebGL application "VirtualBackgroundMovieMaker for ZOOM".
    I could build this WebGL application.
    https://aguroshou.github.io/OpenCvForUnityWebGlMovieDownload/
    Press the record button to make a movie, and press the space key to download to your local storage.
    I want to load the movie in the ZOOM application's virtual background movie. But, the movie's codec is ".avi". So, I can't load in the ZOOM application.
    In desktop build, I could change the movie's codec to ".mp4". this function needs extension data. I don't know how to load extension data in WebGL build. Is this possible and how to set?

    I'm a Japanese student and maybe my English is strange. If that, please teach me.

    Thank you for reading.
     
  40. ludos

    ludos

    Joined:
    Nov 14, 2011
    Posts:
    55
    I have problems with all Camera examples on OSX Cataline. It crashes unity completely when starting. Anybody else can confirm this?
     
  41. runtodev

    runtodev

    Joined:
    Nov 30, 2012
    Posts:
    4

    I have additional question about above question.

    Do I have to use matchShapes function?

    because, I want to make a program that match and crop image in the scanned image that user drawed with colorful crayons inside thin black sketched lines such as fish or car on the paper.

    How to get the right part of image for mapping to 3D object in Unity3d?
     
  42. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Thank you very much for reporting.
    I'll check.
     
  43. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    1. I'm not familiar with matchTemplate. Please take a look at this page.
    https://docs.opencv.org/master/de/da9/tutorial_template_matching.html

    2.
    Could you add the following code to your MatchTemplateExample?
    https://github.com/EnoxSoftware/Ope...chTemplateExample/MatchTemplateExample.cs#L43
    Code (CSharp):
    1.                         Mat submat = imgMat.submat(new OpenCVForUnity.CoreModule.Rect(j, i, tempMat.cols(), tempMat.rows()));
    2.                         Imgcodecs.imwrite("C:/Users/aaaaaaa/Desktop/matchTemplateResult.jpg", submat);
    https://enoxsoftware.github.io/Open..._1_mat.html#a1ac9038fdbdc543453a7a2a825100484
    https://enoxsoftware.github.io/Open...codecs.html#a3272eb63413b15d49c3a48ad84332f36
     
  44. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Unfortunately, I don't have such an example, but I think an example of scanning paper with a camera would help you.
    https://github.com/EnoxSoftware/Ope...umentScannerExample/DocumentScannerExample.cs
     
  45. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    You can refer to the following page about arguments that can be set to fourcc in VideoWriter class.
    https://qiita.com/MuAuan/items/5d968d41e993848a4332
     
    aguroshou4620502 likes this.
  46. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
  47. ctedin187

    ctedin187

    Joined:
    Aug 20, 2018
    Posts:
    11
    I hope I'm posting correctly by replying to this post. I'm guessing it's an "uber post". I tried switching cameras with some of the example files. Wondering if there is a quick and dirty way to switch to an attached camera. I'm using the Zed2 camera from Stereolabs, and every other application (OpenCV and others) seems to be able to access it. I'm guessing there is a hook into those controls from within C# api? Thanks.
     
  48. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,557
    Could you set the device name to "Requested Device Name"?
    WebCamTextureHelper_DeviceName.PNG
     
  49. ctedin187

    ctedin187

    Joined:
    Aug 20, 2018
    Posts:
    11
    I'll check it out. Thanks.
     
  50. ctedin187

    ctedin187

    Joined:
    Aug 20, 2018
    Posts:
    11
    I did see that, but it didn't work. I typed the name in exactly as "ZED 2" with a space, and it worked perfectly. Thanks again!