Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[RELEASED] OpenCV for Unity

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Oct 30, 2014.

  1. FaberVi

    FaberVi

    Joined:
    Nov 11, 2014
    Posts:
    145
    Enox can reply to my previous question?
     
  2. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    I plan to rewrite this sample to the "OpenCV for Unity" version.

    Please wait for a while.
     
    ValrikRobot and FaberVi like this.
  3. Awesumo

    Awesumo

    Joined:
    Nov 6, 2011
    Posts:
    57
    Thanks!! I didn't see anything like this in the documentation.
     
  4. LAFI

    LAFI

    Joined:
    Sep 5, 2014
    Posts:
    47
    Hi,
    can Read-Extract-Text-from- Texture using open_cv??
     
  5. FaberVi

    FaberVi

    Joined:
    Nov 11, 2014
    Posts:
    145
    NICE! Oh thanks! And one with (many) color recognition? It would be really useful!
     
  6. ValrikRobot

    ValrikRobot

    Joined:
    Jun 26, 2013
    Posts:
    206
    Hi

    This looks great and perfect for me. Im sending you an email
     
  7. inkipinki

    inkipinki

    Joined:
    Jun 19, 2013
    Posts:
    22
    Hello,

    I'm trying to do the following:

    Code (CSharp):
    1. Mat img = new Mat(_screenShot.height, _screenShot.width, CvType.CV_32FC4);
    2.         Utils.texture2DToMat(_screenShot, img);
    3.         Mat destimg = new Mat(300, 300, CvType.CV_32FC4);
    4.  
    5.         Mat srcM = new Mat(2, 2, CvType.CV_32FC2);
    6.         srcM.put(0, 0, topleft.x, topleft.y, topright.x, topright.y, botleft.x, botleft.y, botright.x, botright.y);
    7.         Mat destM = new Mat(2, 2, CvType.CV_32FC2);
    8.         destM.put(0, 0, 0, 0, 300, 0, 0, 300, 300, 300);
    9.         Mat M = new Mat(3, 3, CvType.CV_32FC1);
    10.         M = Imgproc.getPerspectiveTransform(srcM, destM);
    11.  
    12.         Imgproc.warpPerspective(img, destimg, M, new Size(300, 300));
    13.         _screenShot = new Texture2D(300, 300, TextureFormat.RGB24, false);
    14.         Utils.matToTexture2D(destimg, _screenShot);
    15.         _screenShot.Apply();
    16.         rawimg.texture = _screenShot;
    And the "M = Imgproc.getPerspectiveTransform(srcM, destM);" lines throws a "native object address is null" error.

    Could you help me with this? (i don't quite understand this Mat object yet :/ )

    Thanks!
     
  8. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Since ”OpenCV for Unity” is a clone of ”OpenCV Java”, please refer to the documentation of "OpenCV Java".
     
  9. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Currently, "opencv_contrib text module" is not implemented.
     
  10. ValrikRobot

    ValrikRobot

    Joined:
    Jun 26, 2013
    Posts:
    206
  11. jococo

    jococo

    Joined:
    Dec 15, 2012
    Posts:
    232
    Hello,

    I have OpenCV up and running and able to detect a face from a loaded user pic. When it finds a face it draws boxes around eyes, nose and mouth. Looks good!

    However, is there a way to get the locations on the pic of the eyes, nose, and mouth detected?

    I think I can grab it from the IDictionary/List created but so far I can't seem to figure it out.

    Thanks!
     
    Last edited: Nov 4, 2015
  12. ValrikRobot

    ValrikRobot

    Joined:
    Jun 26, 2013
    Posts:
    206
    hi. Just started using OpenCV for Unity and i wondered if anyone was familiar with face identification using the FisherFaceRecognizer and how to go about it. So in the basic face detection sample, once we have found a face, how can i extract the face image to save it?
     
  13. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Is it the example such as this video?
     
    FaberVi likes this.
  14. FaberVi

    FaberVi

    Joined:
    Nov 11, 2014
    Posts:
    145
    Yes! :D
     
  15. FaberVi

    FaberVi

    Joined:
    Nov 11, 2014
    Posts:
    145
    A question, in addition to the purely visual recognition is possible to understand for each object marked its "forward" (z-axis) where is pointing?
     
  16. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    It is as follows.
    Code (CSharp):
    1.  
    2.               if (cascade != null)
    3.                          cascade.detectMultiScale (grayMat, faces, 1.1, 2, 2,
    4.                                            new Size (20, 20), new Size ());
    5.  
    6.               OpenCVForUnity.Rect[] rects = faces.toArray ();
    7.  
    8.               for (int i = 0; i < rects.Length; i++) {
    9.                          Debug.Log ("detect faces " + rects [i]);
    10.  
    11.                          Core.rectangle (imgMat, new Point (rects [i].x, rects [i].y), new Point (rects [i].x + rects [i].width, rects [i].y + rects [i].height), new Scalar (255, 0, 0, 255), 2);          
    12.  
    13. ///////////////////////////////////////////////////////////////////////////////////////////////////////
    14.                           Mat faceMat = new Mat(imgMat, rects[i]);
    15.                           Highgui.imwrite("C:/savefolder/face_" + i + ".jpg", faceMat);
    16. ///////////////////////////////////////////////////////////////////////////////////////////////////////
    17.               }
    18.  
     
  17. ikazrima

    ikazrima

    Joined:
    Feb 11, 2014
    Posts:
    320
    Hi,

    Just bought your plugin and wanted to test some samples, specifically the webcam related ones. I followed your readme/video of setting up, open up the demo scene and it doesn't seem to detect my webcam since the scene is all black.
    Is there's any other setup that I need to do for it to detect my webcam? When I tried Vuforia before, it seems to access my webcam just fine.

    Using logitech hd c310, tested in the editor and standalone build.

    Thanks.
     
  18. ikazrima

    ikazrima

    Joined:
    Feb 11, 2014
    Posts:
    320
    Oops, nevermind. My default camera was set to FaceRig Virtual camera, just need to find a way to disable that.
     
  19. jabuka

    jabuka

    Joined:
    Mar 24, 2012
    Posts:
    4
    Currently, "opencv_contrib text module" is not implemented.
    when we can use OCR? and what is implemented in beta?
    I can't use MOG cause function is there but not variables inside. please explain to all of us. THANK YOU
     
  20. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Unfortunately, because "opencv_contrib text module" has not been implemented, OCR is not available.
    Does MOG mean BackgroundSubtractorMOG?
     
  21. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    OpenCV for Unity
    Released Version 1.2.3

    Version changes
    1.2.3
    [Common] Add Beta7Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.
    [Common]Add WrapPerspectiveSample,HandPoseEstimationSample,MultiObjectTrackingBasedOnColorSample.
     
    FaberVi likes this.
  22. FaberVi

    FaberVi

    Joined:
    Nov 11, 2014
    Posts:
    145
    GREAT!!
     
  23. MacyK

    MacyK

    Joined:
    May 10, 2013
    Posts:
    15
    The scene WebCamTextureToMatSample crashes on Mac. Is there a way I can fix it?
     
  24. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Please tell me your development environment.
    OS version :
    Unity version :
    OpenCV for Unity version :

    Does the other samples scene work normally?
     
  25. Nihil688

    Nihil688

    Joined:
    Mar 12, 2013
    Posts:
    503
    Hi, are there any benchmarks for simple image to image recognition. I want to compare two images and see whether one is included in the other and the positions if more than once.
     
  26. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
  27. Nihil688

    Nihil688

    Joined:
    Mar 12, 2013
    Posts:
    503
    This algorithm only finds the best occurrence, what about finding all of them? I have read a very backwards way to trick the algorithm into doing it but is there any method or anything more straightforward than that? I mean if it finds the best there should be a list of potential matches
     
  28. FaberVi

    FaberVi

    Joined:
    Nov 11, 2014
    Posts:
    145
    @EnoxSoftware A question, in addition to the purely visual recognition is possible to understand for each object marked its "forward" (z-axis) where is pointing (and all x, y, z)?
     
  29. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
  30. Nihil688

    Nihil688

    Joined:
    Mar 12, 2013
    Posts:
    503
  31. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Sorry for my late reply.
    I am afraid I cannot answer, because I do not know much about it.
     
  32. Nihil688

    Nihil688

    Joined:
    Mar 12, 2013
    Posts:
    503
    Are there any settings to be aware of? Imgproc.TM_CCOEFF for example or a resource I can check?

    When I tried it, it found the correct first result but it continued finding bad ones saying they still matched 100% ( Core.MinMaxLocResult maxVal ). My images are black and white so I thought it'd be easier to search those but in the
    end I kept having issues. Any ideas?
     
  33. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    http://stackoverflow.com/questions/17001083/opencv-template-matching-example-in-android
    Correct results are displayed by using the lena image on this page?
     
  34. Nihil688

    Nihil688

    Joined:
    Mar 12, 2013
    Posts:
    503
    Yes, that's finding the first best result. If you want to keep on using it to find the next best it will find something wrong but still say it's the best answer. Core.MinMaxLocResult maxVal supposedly gives you the threshold of correctness but for some reason in this plugin it doesn't work
     
  35. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Please tell me the code that caused the problem.

    In addition, I tested another example.

    Code (CSharp):
    1.  
    2.                         Mat img = Highgui.imread (Utils.getFilePath ("mario.jpg"));
    3.                         Mat templ = Highgui.imread (Utils.getFilePath ("mario_template.jpg"));
    4.            
    5.                         // / Create the result matrix
    6.                         int result_cols = img.cols () - templ.cols () + 1;
    7.                         int result_rows = img.rows () - templ.rows () + 1;
    8.                         Mat result = new Mat (result_rows, result_cols, CvType.CV_32FC1);
    9.  
    10.                         int match_method = Imgproc.TM_CCOEFF_NORMED;
    11.            
    12.                         // / Do the Matching and Normalize
    13.                         Imgproc.matchTemplate (img, templ, result, match_method);
    14.                        
    15.  
    16.                         Imgproc.threshold (result, result, 0.8, 1.0, Imgproc.THRESH_TOZERO); //=0.8
    17.            
    18.                         for (int i=0; i<result.rows(); i++) {
    19.                                 for (int j=0; j<result.cols(); j++) {
    20.                                         if (result.get (i, j) [0] > 0) {
    21.                        
    22.                                                 Core.rectangle (img, new Point (j, i), new Point (j + templ.cols (), i + templ.rows ()), new Scalar (0, 0, 255));
    23.                                                 Debug.Log ("value" + result.get (i, j) [0]);
    24.                                         }
    25.                                 }
    26.                         }
    27.  
    28.                         Imgproc.cvtColor (img, img, Imgproc.COLOR_BGR2RGB);
    29.  
    30.  
    31.                         Texture2D texture = new Texture2D (img.cols (), img.rows (), TextureFormat.RGBA32, false);
    32.  
    33.        
    34.                         Utils.matToTexture2D (img, texture);
    35.  
    36.                         gameObject.GetComponent<Renderer> ().material.mainTexture = texture;
    mario.jpg
    mario.jpg

    mario_template.jpg
    mario_template.jpg

    result
    matchTempate_result.PNG
     
  36. Nihil688

    Nihil688

    Joined:
    Mar 12, 2013
    Posts:
    503
    Not sure how you're getting that result with the code you posted, you have forgotten to normalize:
    Code (CSharp):
    1. Core.normalize( result, result, 0, 1, Core.NORM_MINMAX, -1, new Mat() );
    Also any reason you used: Imgproc.TM_CCOEFF_NORMED and not Imgproc.TM_SQDIFF_NORMED ?

    What does the following do exactly? This is the only part I wasn't doing the same and also I couldn't find in the documentation that Mat.get will return the threshold as in the C++ version it's the maxVal I think

    Code (CSharp):
    1. Imgproc.threshold( result, result, 0.8, 1.0, Imgproc.THRESH_TOZERO ); //=0.8
    But cool, with this I am seeing some ok results, it currently finds the correct answer more than one time but I am sure I can fix that =)
     
  37. bysreg

    bysreg

    Joined:
    Jul 28, 2013
    Posts:
    8
    Hi Enox,

    I have a question. Is machine learning, specifically neural network (CvANN_MLP) is implemented ?
     
  38. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Sorry, I do not understand matchTemplate parameter.
    http://docs.opencv.org/2.4/doc/tutorials/imgproc/histograms/template_matching/template_matching.html

    I created a sample referring to following pages.
    http://opencv-python-tutroals.readt...y_template_matching/py_template_matching.html
    http://workpiles.com/2015/05/opencv-matchtemplate-java/

    Code (CSharp):
    1. Imgproc.threshold( result, result, 0.8, 1.0, Imgproc.THRESH_TOZERO ); //=0.8
    Probably, this code is processing to change the values less than the threshold(0.8) to 0.
     
  39. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
  40. BrianND

    BrianND

    Joined:
    May 14, 2015
    Posts:
    79
    I think I've speed the algorithm up a bit using the gpu to blit the web texture to a downscaled version then use RenderTexture GetNativeTexturePtr () + readpixels to get 60fps on 4s for a single face detection. I can probably use it to transpose the sprites and gray scale it also.
     
  41. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Thank you for the information.
    Could you explain that to me in a little more detail?
     
  42. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    OpenCV for Unity
    Released Version 1.2.4


    Version changes
    1.2.4
    [Common]Fix FaceRecognizerSample.
    [Common] Delete the method using Default parameter specifiers.
     
  43. BrianND

    BrianND

    Joined:
    May 14, 2015
    Posts:
    79
    In the Start function I made a render texture

    Code (CSharp):
    1. _downScaled = new RenderTexture (_webCamTexture.width / 4, _webCamTexture.height / 4, 0);
    Then in Update I copy the web cam to the downscaled rendertexture. Here I can also probably greyscale and transpose using the GPU

    Code (CSharp):
    1.  
    2. Graphics.Blit (_webCamTexture, _downScaled);
    3.  
    4. or
    5.  
    6. Graphics.Blit (_webCamTexture, _downScaled, someMaterialToGrayscaleAndTranspose);
    7.  
    I made a new function that does the same as webCamTextureToMat but with a render texture.

    for non iOS platforms I just use readpixels

    Code (CSharp):
    1.  
    2.  
    3. OpenCVForUnity_LowLevelTextureToMat (renderTex.GetNativeTexturePtr (), webCamTexture.width, webCamTexture.height, mat.nativeObj);
    4.  
    5. .....
    6.  
    7. RenderTexture.active = webCamTexture;
    8. _tempTexture.ReadPixels(newUnityEngine.Rect(0, 0, renderTex.width, renderTex.height), 0, 0);
    9. _tempTexture.Apply();
    10. RenderTexture.active = null;
    11.  
    12. Color32[] colors = _tempTexture.GetPixels32();
    13. GCHandlecolorsHandle = GCHandle.Alloc (colors, GCHandleType.Pinned);
    14.  

    It seems to track faster. Not sure how much downscaling effects the actual accuracy.
     
  44. ikazrima

    ikazrima

    Joined:
    Feb 11, 2014
    Posts:
    320
    How do I make interaction between multiple markers and objects?

    Studying the example, there are unique AR cameras just to render certain layers. I can separate the objects into different layers (lets say AR1, AR2, etc.), and change the culling mask of the cameras respectively. But how do I move the objects in game based on the markers position? I want to make the objects interact based on their proximity between each other. With the example scene method, this is impossible since the objects are in the same position.

    Appreciate if someone can give me pointers.
     
  45. Frankster

    Frankster

    Joined:
    May 6, 2013
    Posts:
    12
    Hello Enox Team. I got successfully implemented a stereo-calibration-tool in unity with opencv. Which is great (since i was starting to learn c# 4 months ago)! Now i'd like to optimize the code and performance. Is it theoretically possible to create several multi threads within unity+opencv for image processing and undistorting the cam-textures. (I have 2 cams with 60fps 1280x720 pixels each). I know from other blogs that it is possible with c++. Thank you very much for all your efforts!
     
  46. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    Your post is very useful.
    Use of GPU may be useful for the simple image processing.
     
  47. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    "WebCamTextureAsyncDetectFaceSample" is the example that improve speed of "WebCamTextureDetectFaceSample" by using multi threads.I used this code(https://github.com/Itseez/opencv/blob/master/modules/objdetect/src/detection_based_tracker.cpp) as reference.
    There have been no problems so far.
     
    Frankster likes this.
  48. michaelstv

    michaelstv

    Joined:
    Nov 11, 2015
    Posts:
    5
    So, I have a Unity project using OpenCVForUnity.VideoCapture feature.
    I am failing to run the Windows standalone player.

    I saw the comment in VideoCapture.cs:
    * (Support for Win,Mac,iOS)
    * Win Setup
    * 1)Download "OpenCV for Windows Version 2.4.11"(http://opencv.org/downloads.html).
    * 2)Set Path to "opencv_ffmpeg2411.dll"
    * if 32bit, "C:\opencv\build\x86\vc10\bin\".
    * if 64bit, "C:\opencv\build\x64\vc10\bin\".


    (Yes I am using OpenCVForUnity 1.2.4 and not the new version based on “OpenCV3.0.0)

    Should I really create a file under C:/opencv\build\x64\vc10\bin\ ?
    The thing is that it still doesn't work for me: the player fails to open the AVI file on another PC (without OpenCV or any dev environment installed), although it succeeds on my development PC. I wish I had some exception or log messages but the VideoCapture.open functions just returns false. I am assuming this is because of the opencvforunity.dll failing to open opencv_ffmpeg2411.dll. Can somebody confirm that ?

    A side question: can't I just copy opencv_ffmpeg2411.dll under my Assets/Plugins/x86_64 folder, so I get it under my deployed Windows build's app_Data/Plugins folder ?
     
  49. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,554
    You do not need to create opencv_ffmpeg2411.dll under C:/opencv\build\x64\vc10\bin\.
    Please add folder path that contains "opencv_ffmpeg2411.dll" to PATH environment variables.
    or
    Please copy the "opencv_ffmpeg2411.dll" in the same folder as the .exe file which was output by Unity.
     
  50. EdilsonJunior

    EdilsonJunior

    Joined:
    Mar 17, 2015
    Posts:
    29
    I wonder how to add audio on a code excerpt?