[RELEASED] OpenCV for Unity

”DescriptorMatcher.FLANNBASED” seems to only be used in "SIFT, SURF".
But, Because "SIFT,SURT" is nonfree module, it is not implemented in "OpenCV for Untiy".

 

You can read Mat directly by using Highgui.imread().
However, iOS does not support jpg and png.

 

I want to pattern recognition for augmented reality
OPENCV for Unity can be use?

 

Hi,

I am working on an AR project which needs 'OpenCV for Unity' implementation, but before buying package i have few queries.

1> Can this package with 'MarkerBased AR' be used as image recognition for different markers in database ?
2> Will this work Offline completely ?
3> Will this work for mobile devices (Android/iOS/Windows Store Apps) ?
4> Can 'MarkerBased AR Sample' be use to do image recognition from mobile device database and mobile camera ?
5> How many images in mobile device database it can store as 'Marker'?
6> Can i use AssetBundle to download data at runtime and then do image recognition ?
7> What is the licensing scheme of this package ?

Regards
Selaphiel

 

Hi,

Same as the 2 guys above.

I need an asset that does image recognition for an AR project. The image provided as a marker must be custom.
It also need to be really fast and accurate (running on Windows PC), only 1 image must be recognised.

Is it possible to achieve this with your asset ?

Regards

 

All the functions of OpenCV Java 2.4.10 are available.
Please look for an implementation by the OpenCV Java.

 

”MarkerBasedARSample”Code is the rewrite of https://github.com/MasteringOpenCV/code/tree/master/Chapter2_iPhoneAR using the “OpenCV for Unity”.
It is a very simple code for tutorials.
Please refer to the book for the details of the code.

Because it is a simple code, please customize it by oneself.

1> Can this package with 'MarkerBased AR' be used as image recognition for different markers in database ?
can read the Marker of the default.
It is possible if you change a program.

2> Will this work Offline completely ?
yes.

3> Will this work for mobile devices (Android/iOS/Windows Store Apps) ?
only Android and iOS.

4> Can 'MarkerBased AR Sample' be use to do image recognition from mobile device database and mobile camera ?
yes.

5> How many images in mobile device database it can store as 'Marker'?
I think that it can be to increase by rewriting the program.

6> Can i use AssetBundle to download data at runtime and then do image recognition ?
yes

7> What is the licensing scheme of this package ?
Because using OpenCV, "OpenCV for Unity" is a BSD license.

 

Thanks Enox, that did it.

Regards
Selaphiel

 

”MarkerBasedARSample”Code is the rewrite of https://github.com/MasteringOpenCV/code/tree/master/Chapter2_iPhoneAR using the “OpenCV for Unity”.
It is a very simple code for tutorials.
Please refer to the book for the details of the code.

Run in macbook air 2012 Mid.

 

It looks like a good start, but can we register our own Image Markers ? I need to detect a Logo
Does it work out of the box like dropping a prefab, configure image and it automatically drive the transform of an object ?

Is there anyway to test your asset to ensure that it can do what we need for our project and then buy it if is ok ?

Regards

 

In this sample, 5 x 5 marker is detected.
It is not possible to detect such as photos.

Unfortunately there is no test asset.
A source code is uploaded in github.https://github.com/EnoxSoftware/MarkerBasedARSample

 

There is no way to detect even a simple logo which could be surrounded by a frame or stuff like that ?

 

I think that there are some methods.
by analyzing a frame surrounding the logo.
or
by using FeatureDetector class for the estimate of the logo image direction.(http://docs.opencv.org/ref/master/dc/d2c/tutorial_real_time_pose.html)

 

Released Version 1.1.0

Version changes
1.1.0
[Common]Divide asset for Unity4 and Unity5.

 

I am new to OpenCV and need little guidance. Is there any way to get position of detected Object from live camera and store in Vector2?

Regards.

 

Please refer to DetectFaceSample.
You can convert a detection result from MatOfRect into Rect.

Code (CSharp):

1. MatOfRect faces = new MatOfRect ();
2.      
3. if (cascade != null)
4.       cascade.detectMultiScale (grayMat, faces, 1.1, 2, 2,
5.                                            new Size (20, 20), new Size ());
6.  
7.      OpenCVForUnity.Rect[] rects = faces.toArray ();
8.      for (int i = 0; i < rects.Length; i++) {
9.           Debug.Log ("detect faces " + rects [i]);
10.  
11.           Core.rectangle (imgMat, new Point (rects [i].x, rects [i].y), new Point (rects [i].x + rects [i].width, rects [i].y + rects [i].height), new Scalar (255, 0, 0, 255), 2);
12.      }

 

Referring to WebCamTextureDetectFaceSample I was trying to extend the functionality to eyes detection. To get this feature I've added the haarcascade_eye.xml classifier. Since then test app have been crashing. Is it known feature or just sort of simple bug in my code? Is plugin supposed to handle other classifiers?

 

What platforms errors occurred?
Error did not occur in my environment.(Windows8.1 Unity4.5.5 Editor)

I changed only the following part of the ”WebCamTextureDetectFaceSample”.
and copy ”haarcascade_eye.xml” to "StreamingAssets".

Code (CSharp):

1. //                  cascade = new CascadeClassifier (Utils.getFilePath ("haarcascade_frontalface_alt.xml"));
2.                     cascade = new CascadeClassifier (Utils.getFilePath ("haarcascade_eye.xml"));
3.                                        

 

I have been using other classifiers (face, mouth, eyes) on win 7, win 8 and android without problems myself. I haven't had any problems. I've actually used the haarcascade_eye.xml as well.

 

My bad, it was corrupted xml file.

Another question is:
how to cast/convert Rect.x/y coordinates from detected thing to world position? Becasue to be honest I have no idea what is happening there and can't find any info in documentation.

@Edit1:
There is also another thing I can't handle. The sample code is not able to use any camera on android devices (tested on Samsung galaxy S2 and HTC Desire 510). Any ideas?

 

By world position are you talking about x,y,z-coordinates? As in determining a 3D shape from several images, ie structure from motion? If so then you have the Calib3d.findFundamentalMat-function but you need to calculate the camera matrices yourself I think. I am currently trying to get that to work myself right now If anyone has found any other functionality that I don't know of that is related to structure from motion then PLEASE let me know!

About the Android devices, I have it working on a low end Android but you need to follow the instructions in the manual to set up all the dll:s etc. Have you done that?

 

Please refer to the next code. It is a very simple example using in OpenCVObjectDetector.
https://github.com/EnoxSoftware/OpenCVObjectDetector/blob/master/Sample/ObjectDetectSample.cs

 

Is there no way to detect photo/images ?

 

I think it to be possible by implementing the following examples.
http://docs.opencv.org/ref/master/dc/d2c/tutorial_real_time_pose.html
https://github.com/MasteringOpenCV/code/tree/master/Chapter3_MarkerlessAR

 

List<MatOfPoint> contours = newList<MatOfPoint>();
Mathierarchy = newMat();
Imgproc.findContours(grayMat, contours, hierarchy, Imgproc.RETR_LIST,Imgproc.CHAIN_APPROX_SIMPLE )

MatOfPoint2f approxCurve =newMatOfPoint2f();

//For each contour found
for(int i=0; i<contours.size(); i++) // ---->error because not function size in opencvunity
{//Convert contours(i) from MatOfPoint to MatOfPoint2f
MatOfPoint2f contour2f =newMatOfPoint2f( contours.get(i).toArray()); // // ---->error because not function get
//Processing on mMOP2f1 which is in type MatOfPoint2f
double approxDistance =Imgproc.arcLength(contour2f,true)*0.02;
Imgproc.approxPolyDP(contour2f, approxCurve, approxDistance,true);

//Convert back to MatOfPointMatOfPoint points =newMatOfPoint( approxCurve.toArray());

// Get bounding rect of contourRect rect =Imgproc.boundingRect(points);

// draw enclosing rectangle (all same color, but you could use variable i to make them unique)
Core.rectangle(contoursFrame,newPoint(rect.x,rect.y),newPoint(rect.x+rect.width,rect.y+rect.height),(255,0,0,255),3);

}
I can't convert to c#
I want to draw rectangle on contour
Please Help me!!!!!!!

 

Hi,
I'm writing to query information about problems on this plugin.
I tried to use the face tracking feature (OpenCV's Haar Cascade algorithm) on a live input from a webcam and I always got 0 detected face. So I tried the simple "DetectFaceSample" from your demo app for Android, which should display a red rectangle, but just displays the picture of Lena. I tried this in an empty project but not only, with Unity 4.5.5p5 Pro, 5.0.0 and 5.0.1, inside the editor. There is no error, just this log:
imgMat dst ToString Mat [ 512*512*CV_8UC4, isCont=True, isSubmat=False, nativeObj=0x423869640, dataAddr=0x420675632 ]
UnityEngine.Debug:Log(Object)
OpenCVForUnitySample.DetectFaceSample:Start() (at Assets/OpenCVForUnity/Samples/DetectFaceSample/DetectFaceSample.cs:23)

What can I do about it?
Thank you in advance.

 

for(inti = 0; i < contours.Count; i++) // Actually it will be contours.Count not contours.Size()
{
MatOfPoint2fcontour2f =newMatOfPoint2f(contours.toArray ());
doubleapproxDistance =Imgproc.arcLength(contour2f,true) * 0.02f;
MatOfPoint2fapproxCurve = newMatOfPoint2f ();
Imgproc.approxPolyDP(contour2f, approxCurve, approxDistance,true);
MatOfPointpoints =newMatOfPoint( approxCurve.toArray());
OpenCVForUnity.Rectrect = Imgproc.boundingRect(points);
Core.rectangle(threshold,newPoint(rect.x,rect.y),newPoint(rect.x+rect.width,rect.y+rect.height), newScalar(255,0,0,255),3);
}

 

I can't change marker AR
I try to test other picture not appear model
Please Tutorial Step in change marker AR

 

If the red frame is not displayed, you might have failed to read the "haarcascade_frontalface_alt.xml".
Please confirm whether there is ”OpenCVForUnity/StreamingAssets”folder at the right position.

 

Ok referring to code you gave me https://github.com/EnoxSoftware/OpenCVObjectDetector/blob/master/Sample/ObjectDetectSample.cs

I managed to do ... sth? What's the point. I'm trying to detect face elements just as eyes, nose etc. and put a 2Dtexture on found position e.g. glasses in location of eyes. Sadly the conversion of points in ObjectDetectSample is totaly wrong (for my purpose). Giving me sort of "random" results.

 

Thank you for your promptness, it was exactly that. However I would like to say that an error message when the training file isn't found would be appreciated.
(The file was in Assets/OpenCVForUnity/StreamingAssets and it has to be in Assets/StreamingAssets.)

 

In this sample, a marker of 5 × 5 is detectable.
If you want to change the marker, please change the ids array of hammDistMarker method of Marker class.
0 represents black, 1 represents the white.

marker
[1, 0, 0, 0, 0]
[0, 1, 1, 1, 0]
[1, 0, 1, 1, 1]
[1, 0, 1, 1, 1]
[1, 0, 1, 1, 1]

↓

ids
new int[]{1,0,0,0,0},
new int[]{1,0,1,1,1},
new int[]{0,1,0,0,1},//???　Maybe it is dummy data.
new int[]{0,1,1,1,0}

 

Released Version 1.1.1

Version changes
1.1.1
[Common]Add OpticalFlowSampleScene.
[Common]Fix SampleScene.
[Common]Fix function name of CvANN_MLP_TrainParams class.

 

Hey Enox i can't convert code java to c#
I want to compare histogram image.
Help me please!!!!!

Mat image0 =...;//
Mat image1 =...;

Mat hist0 =newMat();
Mat hist1 =newMat();

int hist_bins =30;//number of histogram bins
int [] hist_range={0,180};//histogram range
MatOfFloat ranges =newMatOfFloat(0f,256f);
MatOfInt histSize =newMatOfInt(25);
// ERROR
Imgproc.calcHist(Arrays.asList(image0),new MatOfInt(0),new Mat(), hist0, histSize, ranges);
Imgproc.calcHist(Arrays.asList(image1),new MatOfInt(0),new Mat(), hist1, histSize, ranges);

double res =Imgproc.compareHist(hist0, hist1,Imgproc.CV_COMP_CORREL);

 

Thank you

 

Please try this code.

Code (CSharp):

1. List<Mat> image0_list = new List<Mat> ();
2. image0_list.Add (image0);
3. List<Mat> image1_list = new List<Mat> ();
4. image1_list.Add (image1);
5.  
6. Imgproc.calcHist(image0_list,new MatOfInt(0),new Mat(), hist0, histSize, ranges);
7. Imgproc.calcHist(image1_list,new MatOfInt(0),new Mat(), hist1, histSize, ranges);

 

Hi, Can you please tell. That how can we detect color in Detected Face rectangle. I have used different techniques to do so but its not working.
Thanks & Regards.

 

This is an example of a simple color detection.

Code (CSharp):

1. //roi mat of detected face
2. Mat faceMat = new Mat(imgMat, new Rect(faceX, faceY, faceW, faceH));
3.  
4. Mat redOnlyMat = new Mat();
5.  
6. //redOnlyMat is the same size as src and CV_8U type.
7. Core.inRange(faceMat, Scalar(0, 0, 0), Scalar(0, 0, 255), redOnlyMat);

 

Hi,
Can I use EigenDecomposite method from this asset's library ?

 

Unfortunately, the "EigenDecomposite" method is not implemented in "OpenCV for Unity".
http://enoxsoftware.github.io/OpenCVForUnity/doc/html/annotated.html

 

hi
I work with vuforia Version 4. can I use the Face Detection with Vuforia camera in Android 4/5

THX.

 

I was successful in displaying vuforia camera image, using Mat of "OpenCV for Unity".
I was referring to the following pages.
https://developer.vuforia.com/library/articles/Solution/How-To-Access-the-Camera-Image-in-Unity

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. using OpenCVForUnity;
5.  
6. public class CameraImageAccess : MonoBehaviour, ITrackerEventHandler
7. {
8.     //cube on which the texture will be applied
9.     public GameObject model;
10. //    private Image.PIXEL_FORMAT m_PixelFormat = Image.PIXEL_FORMAT.RGB888;
11.         private Image.PIXEL_FORMAT m_PixelFormat = Image.PIXEL_FORMAT.GRAYSCALE;
12.         private bool m_RegisteredFormat = false;
13.         private bool m_LogInfo = true;
14.         Mat inputMat;
15.         Texture2D outputTexture;
16.  
17.         void Start ()
18.         {
19.                 QCARBehaviour qcarBehaviour = (QCARBehaviour)FindObjectOfType (typeof(QCARBehaviour));
20.                 if (qcarBehaviour) {
21.                         qcarBehaviour.RegisterTrackerEventHandler (this);
22.                 }
23.         }
24.  
25.         public void OnInitialized ()
26.         {
27.      
28.         }
29.  
30.         public void OnTrackablesUpdated ()
31.         {
32.                 if (!m_RegisteredFormat) {
33.                         CameraDevice.Instance.SetFrameFormat (m_PixelFormat, true);
34.                         m_RegisteredFormat = true;
35.                 }
36.  
37.                 CameraDevice cam = CameraDevice.Instance;
38.                 Image image = cam.GetCameraImage (m_PixelFormat);
39.                 if (image == null) {
40.                         Debug.Log (m_PixelFormat + " image is not available yet");
41.                 } else {
42.  
43.                         if (inputMat == null) {
44.                                 inputMat = new Mat (image.Height, image.Width, CvType.CV_8UC1);
45.                         }
46. //                        Debug.Log ("inputMat dst ToString " + inputMat.ToString ());
47.  
48.                         inputMat.put (0, 0, image.Pixels);
49.  
50.  
51.                         if (outputTexture == null) {
52.                                 outputTexture = new Texture2D (inputMat.cols (), inputMat.rows (), TextureFormat.RGBA32, false);
53.                         }
54.              
55.                         Utils.matToTexture2D (inputMat, outputTexture);
56.  
57.                         model.renderer.material.mainTexture = outputTexture;
58.  
59.                 }
60.         }
61. }

 

Still don't understand, how am I supposed to generate this sort of rotation matrix?

 

Examples of when you want to use the default marker image


↓

convert the image into an array data.
0 represents black, 1 represents the white.

[1, 0, 0, 0, 0] //a
[0, 1, 1, 1, 0] //b
[1, 0, 1, 1, 1] //c
[1, 0, 1, 1, 1] //c
[1, 0, 1, 1, 1] //c

↓

set the array data of the marker in the "ids" array.
Duplicate row seems to do not have to add.
”ids" array has been defined in the "hammDistMarker" method of "Marker" class.

int[][] ids = new int[][]
{
new int[]{1,0,0,0,0}, //a
new int[]{0,1,1,1,0} //b
new int[]{1,0,1,1,1}, //c
};

To match the marker you want to use, please change the ”ids” array.

 

hi , EnoxSoftware
i have a question about your tool is that can I change the mark image and the image is download from server ??
its mean that the mark image is any image i offer...

question 2:
can i add more mark images to let user open the camera to track them ???

Tks !!!

 

The marker image which is detectable in "MarkerBased AR Sample" is only marker of 5x5.
You can change the markers that are detected as shown in the following article.
http://forum.unity3d.com/threads/released-opencv-for-unity.277080/page-3#post-2083671
This sample can be detected only one type of marker.

 

Hello,

Thank you for releasing this asset!
I am trying to do blob detection to identify black circles out of video. I am using FeatureDetector and Feature2D to do this. This is my code. I have added few lines ( after rgbamat is created)in your WebcamTexture.cs sample to achieve this . I get this error:

NullReferenceException: Object reference not set to an instance of an object
OpenCVForUnity.FeatureDetector.detect (OpenCVForUnity.Mat image, OpenCVForUnity.MatOfKeyPoint keypoints) (at Assets/OpenCVForUnity/org/opencv/features2d/FeatureDetector.cs:230)
OpenCVForUnitySample.WebCamTextureToMatSample+<init>c__Iterator3.MoveNext () (at Assets/OpenCVForUnity/Samples/WebCamTextureToMatSample/WebCamTextureToMatSample.cs:92)
UnityEngine.MonoBehaviour:StartCoroutine(IEnumerator)
OpenCVForUnitySample.WebCamTextureToMatSample:Start() (at Assets/OpenCVForUnity/Samples/WebCamTextureToMatSample/WebCamTextureToMatSample.cs:32)

Basically, line230 in featuredetect.cs throws error.
I have been trying to solve this since a long time, your time and help is greatly appreciated!

Thanks,
Priya

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. using OpenCVForUnity;
5.  
6. namespace OpenCVForUnitySample
7. {
8.     /// <summary>
9.     /// WebCamTexture to mat sample.
10.     /// </summary>
11.     public class WebCamTextureToMatSample : MonoBehaviour
12.     {
13.      
14.         WebCamTexture webCamTexture;
15.         Color32[] colors;
16.         public bool isFrontFacing = false;
17.         int width = 640;
18.         int height = 480;
19.        [COLOR=#ff8000][B] public Mat rgbaMat;
20.         Texture2D texture;
21.         bool initDone = false;
22.         public Mat GrayMat;
23.         public Mat KeypointMat;
24.         public FeatureDetector detector;
25.         public MatOfKeyPoint keypoint1; [/B][/COLOR]
26.  
27.      
28.         // Use this for initialization
29.         void Start ()
30.         {
31.          
32.             StartCoroutine (init ());
33.          
34.         }
35.      
36.         private IEnumerator init ()
37.         {
38.             if (webCamTexture != null) {
39.                 webCamTexture.Stop ();
40.                 initDone = false;
41.              
42.                 rgbaMat.Dispose ();
43.             }
44.          
45.             // Checks how many and which cameras are available on the device
46.             for (int cameraIndex = 0; cameraIndex < WebCamTexture.devices.Length; cameraIndex++) {
47.              
48.              
49.                 if (WebCamTexture.devices [cameraIndex].isFrontFacing == isFrontFacing) {
50.                  
51.                  
52.                     Debug.Log (cameraIndex + " name " + WebCamTexture.devices [cameraIndex].name + " isFrontFacing " + WebCamTexture.devices [cameraIndex].isFrontFacing);
53.                  
54.                     webCamTexture = new WebCamTexture (WebCamTexture.devices [cameraIndex].name, width, height);
55.  
56.                  
57.                  
58.                     break;
59.                 }
60.              
61.              
62.             }
63.          
64.             if (webCamTexture == null) {
65.                 webCamTexture = new WebCamTexture (width, height);
66.             }
67.          
68.             Debug.Log ("width " + webCamTexture.width + " height " + webCamTexture.height + " fps " + webCamTexture.requestedFPS);
69.          
70.          
71.          
72.             // Starts the camera
73.             webCamTexture.Play ();
74.          
75.          
76.             while (true) {
77.                 //If you want to use webcamTexture.width and webcamTexture.height on iOS, you have to wait until webcamTexture.didUpdateThisFrame == 1, otherwise these two values will be equal to 16. (http://forum.unity3d.com/threads/webcamtexture-and-error-0x0502.123922/)
78.                 if (webCamTexture.width > 16 && webCamTexture.height > 16) {
79.                     Debug.Log ("width " + webCamTexture.width + " height " + webCamTexture.height + " fps " + webCamTexture.requestedFPS);
80.                     Debug.Log ("videoRotationAngle " + webCamTexture.videoRotationAngle + " videoVerticallyMirrored " + webCamTexture.videoVerticallyMirrored);
81.                  
82.                     colors = new Color32[webCamTexture.width * webCamTexture.height];
83.                  
84.                     rgbaMat = new Mat (webCamTexture.height, webCamTexture.width, CvType.CV_8UC4);
85.                    [COLOR=#ffa64d][B] GrayMat = new Mat (webCamTexture.height, webCamTexture.width, CvType.CV_8UC1);
86.  
87.                     detector = FeatureDetector.create(FeatureDetector.SIMPLEBLOB);
88.                    
89.                     Debug.Log ("detector created");
90.                     //detector.detect(
91.                    
92.                     detector.detect(GrayMat,keypoint1);
93.                     Debug.Log ("keypoints created");
94.                    
95.                     //Scalar red = Scalar(0,255,0);
96.                    
97.                     Features2d.drawKeypoints(GrayMat,keypoint1,KeypointMat);[/B][/COLOR]
98.                  
99.                     texture = new Texture2D (webCamTexture.width, webCamTexture.height, TextureFormat.RGBA32, false);
100.                  
101.                  ....

 

EnoxSoftware

About ids, you're guessing or ever tested this? Cause the original matrix in code is:

new int[]{1,0,0,0,0},
new int[]{1,0,1,1,1},
new int[]{0,1,0,0,1},
new int[]{0,1,1,1,0}

and it has no relation to the marker sample it is able to detect. Also coding the marker in a way you showed us (just normal binarization) is not the way it is recognizing the sample - tested on multiple 5x5 markers.

I think it is somehow connected with the output ID value of the marker and matrix presented here is just sort of rotation matrix helping to find the orientation of the marker, so it's supposed to be constant.

So maybe next time first try to test your code because question about changing markers is not the first in this thread so something must be a bit tricki here.

So again I'm bumping my question - how to change marker?

 

”keypoint1” and "KeypointMat" does not seem to be initialized.
Please add the following line.
MatOfKeyPoint keypoint1 = new MatOfKeyPoint();
Mat KeypointMat = new Mat();

 

Since English is not my native language, I might not be able to understand your question correctly.
I tried some of the markers, but worked all of them correctly.

Code (CSharp):

1. new int[]{1,0,0,0,0},
2. new int[]{1,0,1,1,1},
3. new int[]{0,1,0,0,1}, //dummy data
4. new int[]{0,1,1,1,0}

Code (CSharp):

1. new int[]{0,0,1,0,0},
2. new int[]{1,1,1,1,1},
3. new int[]{1,0,1,0,1},
4. new int[]{1,1,0,1,1}

Code (CSharp):

1. new int[]{1,1,1,1,1},
2. new int[]{1,0,0,0,1},
3. new int[]{1,1,1,1,0},
4. new int[]{0,1,1,1,1}

By changing the "ids" array, and I think that it has been successful in changing the marker to be detected.