Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Enox Software - Open CV Rectangle Detection

Discussion in 'Scripting' started by 4thDimensionStudios, Nov 23, 2016.

  1. 4thDimensionStudios

    4thDimensionStudios

    Joined:
    Nov 10, 2014
    Posts:
    7
    I am trying to build a software like CamScanner. I am a beginner to Image Processing. I am trying to detect rectangles from images - ex Paper. I need some help regarding this. I am using Open CV from Enox Software. I tried converting C++ code to C# but since I am a beginner I failed. Expecting some superior help from you

    Thank You !!
     
  2. LeftyRighty

    LeftyRighty

    Joined:
    Nov 2, 2012
    Posts:
    5,148
  3. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    4,218
    Last year I used the OpenCV asset that LeftyRighty pointed out above and it is outstanding.

    You definitely do NOT want to hassle with integrating OpenCV yourself. The guy who made the above plugin really took care of a lot of stuff, and the examples/samples in the package show pretty much how to do all the interesting stuff you would need. Highly recommend it.
     
    xVergilx likes this.
  4. 4thDimensionStudios

    4thDimensionStudios

    Joined:
    Nov 10, 2014
    Posts:
    7
    @LeftyRighty ya I saw all the examples. The thing is this, I am new to this I don't understand some of the methods. I attached a sample jpg image to get and idea.


    public class HoughLinesPSample : MonoBehaviour
    {
    public Texture2D imgTexture;
    public GameObject Cube;
    // Use this for initialization
    void Start ()
    {

    Mat imgMat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC1);

    Utils.texture2DToMat (imgTexture, imgMat);



    Mat grayMat = new Mat ();
    Imgproc.bilateralFilter(imgMat, imgMat,2, 1, 1000);
    Imgproc.threshold(imgMat, grayMat, 0, 255, Imgproc.THRESH_BINARY | Imgproc.THRESH_OTSU);

    Imgproc.Canny(grayMat, grayMat, 255, 255);

    Mat lines = new Mat ();

    Imgproc.HoughLinesP (grayMat, lines, 1, Mathf.PI / 180, 50, 50, 10);


    int[] linesArray = new int[lines.cols () * lines.rows () * lines.channels ()];
    lines.get (0, 0, linesArray);
    Debug.Log("Number of Lines " + linesArray.Length/8);

    imgMat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC3);
    for (int i = 0; i < linesArray.Length; i=i+4) {
    int R=Random.Range(0,255);


    //Imgproc.line(imgMat, new Point(linesArray[i + 0], linesArray[i + 1]), new Point(linesArray[i + 2], linesArray[i + 3]), new Scalar(255, 0, 0), 2);
    Imgproc.rectangle(imgMat, new Point(linesArray[i + 0], linesArray[i + 1]), new Point(linesArray[i + 2], linesArray[i + 3]), new Scalar(255, 0, 0), 2);
    Debug.Log(linesArray [i + 0] +" "+linesArray [i + 1]+" "+linesArray [i + 2]+" "+ linesArray [i + 3]);

    }

    Texture2D texture = new Texture2D(imgMat.cols(), imgMat.rows(), TextureFormat.RGBA32, false);

    Utils.matToTexture2D(imgMat, texture);

    gameObject.GetComponent<Renderer>().material.mainTexture = texture;

    }


    }

    I want to only find the main rectangle/paper in the middle.. and extract its content
     

    Attached Files:

    Last edited: Nov 24, 2016
  5. LeftyRighty

    LeftyRighty

    Joined:
    Nov 2, 2012
    Posts:
    5,148
  6. 4thDimensionStudios

    4thDimensionStudios

    Joined:
    Nov 10, 2014
    Posts:
    7
    Thanks @LeftyRighty .. I am kind of new here
    @Kurt-Dekker , This is my piece of code.. I need to detect a rectangle from this code. I have tried everything I know. Please help me.


    Code (CSharp):
    1. public class HoughLinesPSample : MonoBehaviour
    2. {
    3. public Texture2D imgTexture;
    4. public GameObject Cube;
    5. // Use this for initialization
    6. void Start ()
    7. {
    8.  
    9. Mat imgMat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC1);
    10.  
    11. Utils.texture2DToMat (imgTexture, imgMat);
    12.  
    13.  
    14.  
    15. Mat grayMat = new Mat ();
    16. Imgproc.bilateralFilter(imgMat, imgMat,2, 1, 1000);
    17. Imgproc.threshold(imgMat, grayMat, 0, 255, Imgproc.THRESH_BINARY | Imgproc.THRESH_OTSU);
    18.  
    19. Imgproc.Canny(grayMat, grayMat, 255, 255);
    20.  
    21. Mat lines = new Mat ();
    22.  
    23. Imgproc.HoughLinesP (grayMat, lines, 1, Mathf.PI / 180, 50, 50, 10);
    24.  
    25.  
    26. int[] linesArray = new int[lines.cols () * lines.rows () * lines.channels ()];
    27. lines.get (0, 0, linesArray);
    28. Debug.Log("Number of Lines " + linesArray.Length/8);
    29.  
    30. imgMat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC3);
    31. for (int i = 0; i < linesArray.Length; i=i+4) {
    32. int R=Random.Range(0,255);
    33.  
    34.  
    35. //Imgproc.line(imgMat, new Point(linesArray[i + 0], linesArray[i + 1]), new Point(linesArray[i + 2], linesArray[i + 3]), new Scalar(255, 0, 0), 2);
    36. Imgproc.rectangle(imgMat, new Point(linesArray[i + 0], linesArray[i + 1]), new Point(linesArray[i + 2], linesArray[i + 3]), new Scalar(255, 0, 0), 2);
    37. Debug.Log(linesArray [i + 0] +" "+linesArray [i + 1]+" "+linesArray [i + 2]+" "+ linesArray [i + 3]);
    38.  
    39. }
    40.  
    41. Texture2D texture = new Texture2D(imgMat.cols(), imgMat.rows(), TextureFormat.RGBA32, false);
    42.  
    43. Utils.matToTexture2D(imgMat, texture);
    44.  
    45. gameObject.GetComponent<Renderer>().material.mainTexture = texture;
    46.  
    47. }
    48.  
    49.  
    50. }
    51.  
     

    Attached Files:

  7. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    4,218
    Looks like you have gotten pretty far along. Now basically this is where things get "interesting." You have coming back from OpenCV a stream of contours in the form of sets of connected points. What you need to do is decide what criteria you want to apply to that data stream in order to arrive at the rectangle you seek.

    In our case we knew that the rectangles we sought were about 15% of the width of the screen, so we first applied filters to try and coalesce blobby polygons into four-point rectangle lists, and threw out everything that didn't have exactly four points. I think there was something in OpenCV that helped us reduce the contour complexity, but we might have hand-written that too, I forget.

    We then analyzed the size of the remaining four-point contours with various if/then conditional tests:

    - is it "close enough to square" (i.e., one dimension not more than x% longer than the other)
    - is it "close enough to parallelogram" (i.e., opposing sides have a "similar enough" slope to them)
    - is it "close enough to rectangle" (i.e., are the angles of adjacent lines all close to perpendicular?)
    - is the overall size "close enough" to the size we are seeking? In your case it looks like you want something that is about 70% of the size of the source image, so that could be a critieria you use.

    Basically every frame this processing was applied to the contour datastream, and the results were displayed in various colors as we brought it up and got it going. It was some pretty interesting engineering, and it took a while to get it "just so," but the key was to just work in slow incremental steps. The steps and questions I listed above could be a possible template of steps to refine your data stream into something usable.
     
  8. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,120
    Kurt-Dekker likes this.
  9. alexchandriyaa

    alexchandriyaa

    Joined:
    Jan 18, 2017
    Posts:
    140
    w
    whether this worked or not?? am also trying to find square of image like each boxes in chessboard
     
  10. ginoman

    ginoman

    Joined:
    Oct 15, 2018
    Posts:
    1
    Hi, I'm not a specialist on this so I don't really know but I want to detect few square/ rectangle objects in the same time and since Vuforia isn't able to do that, I'm thinking on using Opencv for unity. I'd like to know if Opencv for unity will cover up what I want to do.
    Thank you.