Search Unity

  1. The 2022.1 beta is now available for testing. To find out what's new, have a look at our 2022.1 beta blog post.
    Dismiss Notice

[RELEASED] OpenCV for Unity

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Oct 30, 2014.

  1. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    For CalibrateCamera, the following functions have been implemented.
    http://docs.opencv.org/java/org/opencv/calib3d/package-frame.html
     
  2. sreedhar3d

    sreedhar3d

    Joined:
    Aug 13, 2012
    Posts:
    6
    I am working on creating object tracking using webcam texture face tracking but the the tracking is not stable. too noise how we can reduce the movement tracking point any help thanks in adv.
     
  3. Visual_Lies

    Visual_Lies

    Joined:
    Nov 3, 2014
    Posts:
    17
    Hello,
    I am hoping someone here can assist me. I am trying to create a script to take the standard unity C# vector2 and vector3 and put them into Mats for open CV.

    Code (CSharp):
    1. static public Mat vec2toMatV(Vector2 vec2)
    2. {
    3. Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC3);
    4.  
    5. tempMatofFloat.put (0, 0, vec2.x);
    6. tempMatofFloat.put (0, 1, vec2.y);
    7. returntempMatofFloat;
    8. }
    The error I am receiving is:

    I am not sure if I am having the wrong the Type or what. I am trying to set up functions to create Mats out of the Screen position Vector2s and Word Position Vector3s to use with the calibrate camera function:

    public static double calibrateCamera(java.util.List<Mat> objectPoints,
    java.util.List<Mat> imagePoints,
    Size imageSize,
    Mat cameraMatrix,
    Mat distCoeffs,
    java.util.List<Mat> rvecs,
    java.util.List<Mat> tvecs,
    int flags,
    TermCriteria criteria)

    I am not the most advanced coder and some of this is still a bit beyond my grasp at the moment.
     
    Last edited: Feb 2, 2015
  4. sloopidoopi

    sloopidoopi

    Joined:
    Jan 2, 2010
    Posts:
    244
    newMat is misspelling. Try: (whitespace between new & Mat)
    Code (CSharp):
    1.  
    2. Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC3);
    3.  
     
  5. Visual_Lies

    Visual_Lies

    Joined:
    Nov 3, 2014
    Posts:
    17
    That was an error in copy and posting. The whitespace exists and still get error.
     
  6. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408

    Where it was tested in my environment, the following code is working well.
    Code (CSharp):
    1. static public Mat vec2toMatV(Vector2 vec2)
    2. {
    3.        //Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC3);
    4.    Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC1);
    5.        
    6.    tempMatofFloat.put (0, 0, vec2.x);
    7.    tempMatofFloat.put (0, 1, vec2.y);
    8.  
    9.    return tempMatofFloat;
    10. }
     
  7. Visual_Lies

    Visual_Lies

    Joined:
    Nov 3, 2014
    Posts:
    17
    Are you testing on a Mac or Windows? I am working on Mac OSX on a Mac Pro. What about when you try using the function?
    <Fixed error had plugins in wrong folder - must be asset level>
     
    Last edited: Feb 4, 2015
  8. Visual_Lies

    Visual_Lies

    Joined:
    Nov 3, 2014
    Posts:
    17
    Fixed error. I didn't move the plugins folder to the proper place. I am an idiot. Thanks for the assistance!

    However now I need to still figure out the proper way to pass float values into the proper type of Mat as needed.

    I've created a function to debug my Mats and I can tell that I am creating them of at least appropriate size:

    Code (CSharp):
    1.     static public void debugMat(Mat incMat)
    2.     {
    3.         int cols = incMat.cols ();
    4.         int rows = incMat.rows ();
    5.  
    6.         for (int i = 0; i < cols; i++)
    7.         {
    8.             for(int j = 0; j < rows; j++)
    9.             {
    10.                 Debug.Log("Row " + i + "Column " + j + "Value " + incMat.get(i,j));
    11.             }
    12.         }
    13.  
    14.     }
    I have tried it with various operators of each CvType but in this manner:
    Code (csharp):
    1.  
    2.   A.create (2, 1, CvType.CV_16S);
    3. debugMat(A);
    4. A.create (2, 1, CvType.CV_16U);
    5. debugMat(A);
    6. A.create (2, 1, CvType.CV_32F);
    7. debugMat(A);
    8. A.create (2, 1, CvType.CV_32S);
    9. debugMat(A);
    10. A.create (2, 1, CvType.CV_8S);
    11. debugMat(A);
    12. A.create (2, 1, CvType.CV_8U);
    13. debugMat(A);
    14. A.create (2, 1, CvType.CV_USRTYPE1);
    15. debugMat(A);
    However the debug result is the same each time:

    I've tried many different ways to set float values into the Mats using .put(). I have tried adding double[] and recalling them at an index and still not receiving a value. Any ideas?
     
    Last edited: Feb 4, 2015
  9. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Code (CSharp):
    1. Vector2 vec2 = new Vector2 (100.5f, 200.2f);
    2. Mat matV = vec2toMatV (vec2);
    3. Debug.Log ("matV " + matV.ToString ());
    4. Debug.Log ("matVdump " + matV.dump ());
    5.  
    6. float f = (float)matV.get (0, 0) [0];
    7. Debug.Log ("f " + f);
    8.  
    9. float[] f_Array = new float[matV.channels ()];
    10. matV.get (0, 0, f_Array);
    11. Debug.Log ("f_Array " + f_Array [0]);
    12.  
    13.  
    14. static public Mat vec2toMatV (Vector2 vec2)
    15. {
    16.     Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC1);
    17.          
    18.     tempMatofFloat.put (0, 0, vec2.x);
    19.     tempMatofFloat.put (0, 1, vec2.y);
    20.  
    21.     return tempMatofFloat;
    22. }
     
  10. Visual_Lies

    Visual_Lies

    Joined:
    Nov 3, 2014
    Posts:
    17
    Oh thank you so very much sir that is awesome!
     
  11. hkha

    hkha

    Joined:
    Jul 21, 2012
    Posts:
    6
    Can this be used to do edge detection ?
     
  12. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
  13. nDman

    nDman

    Joined:
    Nov 1, 2013
    Posts:
    8
    Can i use this for Text Recognition (OCR)?
     
  14. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Unfortunately the OCR method is not implemented.
     
  15. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Released Version 1.0.8

    Version changes
    1.0.8
    [Common]Update to OpenCV2.4.10
     
  16. CheeryDach_Bryan

    CheeryDach_Bryan

    Joined:
    Jul 10, 2013
    Posts:
    7
    Hi,

    I'm wondering if the face detection works in the latest version?
    It seems nothing happened when I used camera to detect my face.
     
  17. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Does DetectFaceSample work fine?
    If the red frame is not displayed, you might have failed to read the "haarcascade_frontalface_alt.xml".
    Please confirm whether there is ”StreamingAssets”folder at the right position.
     
  18. CheeryDach_Bryan

    CheeryDach_Bryan

    Joined:
    Jul 10, 2013
    Posts:
    7
    Thanks. It works like a charm.
    One more question, it seems that the frame rate is pretty low.
    I'm wondering how I can improve the performance?
     
  19. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Can improve speed by adjusting the parameter of the sample scene.
    • Adjusting the parameters of detectMultiScale (). (minSize etc)
    • Change Mat to handle in detectMultiScale() to the smaller size.
     
  20. crazycircuit

    crazycircuit

    Joined:
    Sep 28, 2013
    Posts:
    4
    Hi!
    Thanks for this useful plug in.
    I bought and installed the plug in today.
    Moved the sample file that I wanted to try in 'streaming assets' folder and also moved the opencv bundle to assets folder too.
    I constantly keep on getting this error in unity editor - "ArgumentException: The output Mat object has to be of same size"
    I am using a Mac system OS 10.9.4.

    I tried changing the width and height variable in webcamTexture but it doesn't help. I also tried changing the unity game player size through edit>project setting>player>resolution and presentation.
    What am I doing wrong?

    Thank you!
    -Priya
     
  21. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Thank you for an error report.

    Sample worked without a problem when I tested it in my environment.(OSX 10.9.4 ,Unity 4.5.5)
    I'm sorry, a setup method of ReadMe.pdf was wrong. Please move a file in reference to a capture image.

    Mac Standalone Setup
    × Copy from “OpenCVForUnity/Plugins/opencvforunity.bundle” to “Assets/Plugins/opencvforunity.bundle” folder.
    ○ Copy from “OpenCVForUnity/Plugins/opencvforunity.bundle” to “Assets/Plugins/” folder.


    スクリーンショット 2015-02-24 15.05.18.png
     
  22. ShoaibSadaqat

    ShoaibSadaqat

    Joined:
    Mar 2, 2015
    Posts:
    4
    Can we do Gestures recognitions? like Hand, mouth, and teeth etc.
     
  23. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    In ”OpenCV for Unity”, ”Gestures recognitions” is not implemented as API.
    Because "OpenCV for Unity" is a clone of "OpenCV Java ",I think that it is feasible if there is an example implementing "Gestures recognitions" using "OpenCV Java".
     
  24. Noego

    Noego

    Joined:
    Feb 20, 2013
    Posts:
    5
    Hello,

    We are using the Cascade Classifer for face features detection and we want to use the rejectLevels and levelWeights of the detectMultiScale function but they don't seem to work even with outputRejectLevels set to true.

    Aren't they supposed to return a value fore each detected Rect?
     
  25. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    307
    Is it possible to use "detectMultiScale" on a rotated image somehow? I want to be able to take into account the user turning the phone to a portrait mode for example but I don't know how to tell the cascade which direction is up! :)
     
  26. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    According to the following pages, it seems to be caused by a bug of OpenCV.
    http://haoxiang.org/2013/11/opencv-detectmultiscale-output-detection-score/
    http://answers.opencv.org/question/7270/detectmultiscale-using-rejectlevels-and-levelweights/
    http://code.opencv.org/issues/3064
     
  27. ik1004

    ik1004

    Joined:
    Feb 26, 2013
    Posts:
    2
    Hello,
    Does this plugin support unity 5? Is it safe to upgrade without having any problems?
    Thank you in advance.
     
  28. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    "OpenCV for Unity 1.0.9(Support for Unity5)" is in pending review now in AssetStore.
    I think that it is approved a few days later.

    As for the current version1.0.8, only Android and Win and Mac support Unity5.
    But, iOS does not support.
    If you do not use iOS, there is not the problem with upgrading.
     
    Mikael-H likes this.
  29. Mikael-H

    Mikael-H

    Joined:
    Apr 26, 2013
    Posts:
    307
    Will this remove the limitations on editor and standalone for non-pro users?
     
  30. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Of Course The limitations on editor and standalone for non-pro users is removed in Unity5.
     
    Mikael-H likes this.
  31. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Mikael-H likes this.
  32. Morieris

    Morieris

    Joined:
    Mar 10, 2015
    Posts:
    4
    I'm having problems getting the camera to work in WebCamTextureDetectFaceSample on iOS. I'm using ios8, iphone 6, xcode 6, unity 4.6.3f1 pro. I've got the sample working in the unity editor and osx standalone, and the iOS version builds and links properly, so I'm mostly sure that it is set up correctly. The sample just doesn't show the camera feed on the iphone 6. I'm getting the same issue on an ipad2 with ios 7.
     
    Last edited: Mar 10, 2015
  33. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    In the current Version "OpenCV for Unity1.0.8"
    Errors happen when set "Scripting Backend - IL2CPP" more than Unity4.6.2.

    But,This bug is fixed in next versions.

    "OpenCV for Unity 1.0.9(Support for Unity5)" is in pending review now in AssetStore.
    I think that it is approved a few days later.
     
    Morieris likes this.
  34. Morieris

    Morieris

    Joined:
    Mar 10, 2015
    Posts:
    4

    Thanks for your reply. I've now got the 1.0.9 update and am running in Unity 5.

    I have the WebCamTextureDetectFaceSample up and running fine on an ipad2/ios7, but it is still failing to show the camera on the iphone6/io8.
     
  35. cakeslice

    cakeslice

    Joined:
    Oct 18, 2014
    Posts:
    192
    Is it possible to implement a QR code reader with this asset?
     
  36. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    If use webCamTextureToMat(), Please choose OpenGL ES 2.0 or 3.0 in [PlayerSettings][Other Settings]-[Configuration]-[Graphics API].

    Because error occurs in GetPixels32() of WebCamTexture in iOS,"OpenCV for Unity1.0.9" uses LowLevelGraphics in iOS.
    LowLevelGraphics does not support Metal now, please choose OpenGL.
     
    Last edited: Mar 12, 2015
    Morieris likes this.
  37. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    Released Version 1.0.9

    Version changes
    1.0.9
    [Common] Support for Unity5.
     
    Morieris and Mikael-H like this.
  38. Morieris

    Morieris

    Joined:
    Mar 10, 2015
    Posts:
    4
    Got it, that worked. Thanks!
     
  39. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    MarkerBased AR Sample
    Because I succeed in recognizing a marker in an upper example,Probably I think that the reading of the QR cord is possible.
     
  40. yuvalal

    yuvalal

    Joined:
    Mar 15, 2015
    Posts:
    2
    Thanks for this great plugin.
    Is there a schedule for implementing openCV 3.0.0?
    I'm really looking forward to using OCRTesseract in unity.
     
  41. Pablanka

    Pablanka

    Joined:
    Mar 16, 2015
    Posts:
    1
    Hi,

    I need to find walls in a room and set the camera perspective ,aproximatelly with the image perspective. Like this:
    .
    The objective is avoid to set the perspective manually (
    ), I need to make it automatically
    Can I make something like that with OpenCV plugin? Can you guys guide me with it?.
    Thanks you so much!
     
  42. wirelesstaco

    wirelesstaco

    Joined:
    Apr 27, 2014
    Posts:
    1
    Hey Enox,

    Should there be a video.BackgroundSubtractor operator in the code somewhere? I am currently unable to find it.

    Otherwise everything is looking great!
    Thanks.
     
  43. Radical435

    Radical435

    Joined:
    Oct 20, 2014
    Posts:
    24
    Hello -- is Imgproc.goodFeaturesToTrack implemented?

    I am using this call:
    Imgproc.goodFeaturesToTrack(rgbaMat, keypoints, 1000, 0.01, 10);

    And it never finds features (I've played around with the numbers to see if that was it). rgbaMat is a Mat which is from the webcam.
    I am populating it as in your WebCam examples, using Utils.webCamTextureToMat (webCamTexture, rgbaMat, colors);

    To give a little more context,:

    if (webCamTexture.width > 16 && webCamTexture.height > 16) {
    Utils.webCamTextureToMat (webCamTexture, rgbaMat, colors);
    if(lastRgbaMat != null)
    {
    MatOfPointkeypoints = newMatOfPoint();
    Imgproc.goodFeaturesToTrack(rgbaMat, keypoints, 1000, 0.01, 10);

    Debug.Log("this many good features:"+keypoints.toArray().Length);

    lastRgbaMat exists, and the log prints out successfully.

    Looking for any kind of helpy.

    Thank you
     
  44. Morieris

    Morieris

    Joined:
    Mar 10, 2015
    Posts:
    4
    Does DescriptorMatcher.FLANNBASED work?

    I've tried using flann in the Feature2DSample and in my own project, and it seems to return no matches no matter what.

    -joe
     
  45. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    "OpenCV for Untiy" which implemented "OpenCV 3.0.0" is under development now.
    I am going to implement OCRTesseract.
     
  46. yuvalal

    yuvalal

    Joined:
    Mar 15, 2015
    Posts:
    2
    Thanks, that's great news.
     
  47. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    I think that it is possible if you can detect the line of the wall of the room using OpenCV method(example Imgproc.HoughLines() , Imgproc.findContour()...).
     
  48. ik1004

    ik1004

    Joined:
    Feb 26, 2013
    Posts:
    2
    Hello again,

    Thank you for your quick reply.

    I would now like to know what is the most efficient to load textures from resources folder in a Mat. Right now I am using Resources.Load(file) as Texture and then convert that texture to a mat.
    It is important to find the most efficient way since I will be using a large image set for my application.

    Thank you in advance!

    P.S. I am targeting mobile platforms (Android and iOS but mainly Android)
     
  49. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
  50. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,408
    parameter image of Imgproc.goodFeaturesToTrack must be single-channel.

    http://docs.opencv.org/java/org/ope....opencv.core.MatOfPoint, int, double, double)
    Parameters:
    image - Input 8-bit or floating-point 32-bit, single-channel image.
     
    Radical435 likes this.
unityunity