Enox Software - Open CV Rectangle Detection

I am trying to build a software like CamScanner. I am a beginner to Image Processing. I am trying to detect rectangles from images - ex Paper. I need some help regarding this. I am using Open CV from Enox Software. I tried converting C++ code to C# but since I am a beginner I failed. Expecting some superior help from you

Thank You !!

 

you know they offer a unity integration asset?
https://www.assetstore.unity3d.com/en/#!/content/21088

 

Last year I used the OpenCV asset that LeftyRighty pointed out above and it is outstanding.

You definitely do NOT want to hassle with integrating OpenCV yourself. The guy who made the above plugin really took care of a lot of stuff, and the examples/samples in the package show pretty much how to do all the interesting stuff you would need. Highly recommend it.

 

@LeftyRighty ya I saw all the examples. The thing is this, I am new to this I don't understand some of the methods. I attached a sample jpg image to get and idea.


public class HoughLinesPSample : MonoBehaviour
{
public Texture2D imgTexture;
public GameObject Cube;
// Use this for initialization
void Start ()
{

Mat imgMat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC1);

Utils.texture2DToMat (imgTexture, imgMat);



Mat grayMat = new Mat ();
Imgproc.bilateralFilter(imgMat, imgMat,2, 1, 1000);
Imgproc.threshold(imgMat, grayMat, 0, 255, Imgproc.THRESH_BINARY | Imgproc.THRESH_OTSU);

Imgproc.Canny(grayMat, grayMat, 255, 255);

Mat lines = new Mat ();

Imgproc.HoughLinesP (grayMat, lines, 1, Mathf.PI / 180, 50, 50, 10);


int[] linesArray = new int[lines.cols () * lines.rows () * lines.channels ()];
lines.get (0, 0, linesArray);
Debug.Log("Number of Lines " + linesArray.Length/8);

imgMat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC3);
for (int i = 0; i < linesArray.Length; i=i+4) {
int R=Random.Range(0,255);


//Imgproc.line(imgMat, new Point(linesArray[i + 0], linesArray[i + 1]), new Point(linesArray[i + 2], linesArray[i + 3]), new Scalar(255, 0, 0), 2);
Imgproc.rectangle(imgMat, new Point(linesArray[i + 0], linesArray[i + 1]), new Point(linesArray[i + 2], linesArray[i + 3]), new Scalar(255, 0, 0), 2);
Debug.Log(linesArray [i + 0] +" "+linesArray [i + 1]+" "+linesArray [i + 2]+" "+ linesArray [i + 3]);

}

Texture2D texture = new Texture2D(imgMat.cols(), imgMat.rows(), TextureFormat.RGBA32, false);

Utils.matToTexture2D(imgMat, texture);

gameObject.GetComponent<Renderer>().material.mainTexture = texture;

}


}

I want to only find the main rectangle/paper in the middle.. and extract its content

 

https://forum.unity3d.com/threads/using-code-tags-properly.143875/


since kurt has some experience with this I suggest you'd be better tagging him and asking him the specifics

 

Thanks @LeftyRighty .. I am kind of new here
@Kurt-Dekker , This is my piece of code.. I need to detect a rectangle from this code. I have tried everything I know. Please help me.

Code (CSharp):

1. public class HoughLinesPSample : MonoBehaviour
2. {
3. public Texture2D imgTexture;
4. public GameObject Cube;
5. // Use this for initialization
6. void Start ()
7. {
8.  
9. Mat imgMat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC1);
10.  
11. Utils.texture2DToMat (imgTexture, imgMat);
12.  
13.  
14.  
15. Mat grayMat = new Mat ();
16. Imgproc.bilateralFilter(imgMat, imgMat,2, 1, 1000);
17. Imgproc.threshold(imgMat, grayMat, 0, 255, Imgproc.THRESH_BINARY | Imgproc.THRESH_OTSU);
18.  
19. Imgproc.Canny(grayMat, grayMat, 255, 255);
20.  
21. Mat lines = new Mat ();
22.  
23. Imgproc.HoughLinesP (grayMat, lines, 1, Mathf.PI / 180, 50, 50, 10);
24.  
25.  
26. int[] linesArray = new int[lines.cols () * lines.rows () * lines.channels ()];
27. lines.get (0, 0, linesArray);
28. Debug.Log("Number of Lines " + linesArray.Length/8);
29.  
30. imgMat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC3);
31. for (int i = 0; i < linesArray.Length; i=i+4) {
32. int R=Random.Range(0,255);
33.  
34.  
35. //Imgproc.line(imgMat, new Point(linesArray[i + 0], linesArray[i + 1]), new Point(linesArray[i + 2], linesArray[i + 3]), new Scalar(255, 0, 0), 2);
36. Imgproc.rectangle(imgMat, new Point(linesArray[i + 0], linesArray[i + 1]), new Point(linesArray[i + 2], linesArray[i + 3]), new Scalar(255, 0, 0), 2);
37. Debug.Log(linesArray [i + 0] +" "+linesArray [i + 1]+" "+linesArray [i + 2]+" "+ linesArray [i + 3]);
38.  
39. }
40.  
41. Texture2D texture = new Texture2D(imgMat.cols(), imgMat.rows(), TextureFormat.RGBA32, false);
42.  
43. Utils.matToTexture2D(imgMat, texture);
44.  
45. gameObject.GetComponent<Renderer>().material.mainTexture = texture;
46.  
47. }
48.  
49.  
50. }
51.  

 

Looks like you have gotten pretty far along. Now basically this is where things get "interesting." You have coming back from OpenCV a stream of contours in the form of sets of connected points. What you need to do is decide what criteria you want to apply to that data stream in order to arrive at the rectangle you seek.

In our case we knew that the rectangles we sought were about 15% of the width of the screen, so we first applied filters to try and coalesce blobby polygons into four-point rectangle lists, and threw out everything that didn't have exactly four points. I think there was something in OpenCV that helped us reduce the contour complexity, but we might have hand-written that too, I forget.

We then analyzed the size of the remaining four-point contours with various if/then conditional tests:

- is it "close enough to square" (i.e., one dimension not more than x% longer than the other)
- is it "close enough to parallelogram" (i.e., opposing sides have a "similar enough" slope to them)
- is it "close enough to rectangle" (i.e., are the angles of adjacent lines all close to perpendicular?)
- is the overall size "close enough" to the size we are seeking? In your case it looks like you want something that is about 70% of the size of the source image, so that could be a critieria you use.

Basically every frame this processing was applied to the contour datastream, and the results were displayed in various colors as we brought it up and got it going. It was some pretty interesting engineering, and it took a while to get it "just so," but the key was to just work in slow incremental steps. The steps and questions I listed above could be a possible template of steps to refine your data stream into something usable.

 

I think that the following article(Japanese) will be helpful.
http://qiita.com/cfiken/items/44b364b951b605f1e919

1. Imgproc.threshold()
2. Imgproc.findContours()
   
3. Imgproc.approxPolyDP()

 

w

whether this worked or not?? am also trying to find square of image like each boxes in chessboard

 

Hi, I'm not a specialist on this so I don't really know but I want to detect few square/ rectangle objects in the same time and since Vuforia isn't able to do that, I'm thinking on using Opencv for unity. I'd like to know if Opencv for unity will cover up what I want to do.
Thank you.

 

I really like your idea