[RELEASED] OpenCV for Unity

For CalibrateCamera, the following functions have been implemented.
http://docs.opencv.org/java/org/opencv/calib3d/package-frame.html

 

I am working on creating object tracking using webcam texture face tracking but the the tracking is not stable. too noise how we can reduce the movement tracking point any help thanks in adv.

 

Hello,
I am hoping someone here can assist me. I am trying to create a script to take the standard unity C# vector2 and vector3 and put them into Mats for open CV.

Code (CSharp):

1. static public Mat vec2toMatV(Vector2 vec2)
2. {
3. Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC3);
4.  
5. tempMatofFloat.put (0, 0, vec2.x);
6. tempMatofFloat.put (0, 1, vec2.y);
7. returntempMatofFloat;
8. }

The error I am receiving is:

I am not sure if I am having the wrong the Type or what. I am trying to set up functions to create Mats out of the Screen position Vector2s and Word Position Vector3s to use with the calibrate camera function:

public static double calibrateCamera(java.util.List<Mat> objectPoints,
java.util.List<Mat> imagePoints,
Size imageSize,
Mat cameraMatrix,
Mat distCoeffs,
java.util.List<Mat> rvecs,
java.util.List<Mat> tvecs,
int flags,
TermCriteria criteria)

I am not the most advanced coder and some of this is still a bit beyond my grasp at the moment.

 

newMat is misspelling. Try: (whitespace between new & Mat)

Code (CSharp):

1.  
2. Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC3);
3.  

 

That was an error in copy and posting. The whitespace exists and still get error.

 

Where it was tested in my environment, the following code is working well.

Code (CSharp):

1. static public Mat vec2toMatV(Vector2 vec2)
2. {
3.        //Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC3);
4. 　　　Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC1);
5.        
6. 　　　tempMatofFloat.put (0, 0, vec2.x);
7. 　　　tempMatofFloat.put (0, 1, vec2.y);
8.  
9. 　　　return tempMatofFloat;
10. }

 

Are you testing on a Mac or Windows? I am working on Mac OSX on a Mac Pro. What about when you try using the function?
<Fixed error had plugins in wrong folder - must be asset level>

 

Fixed error. I didn't move the plugins folder to the proper place. I am an idiot. Thanks for the assistance!

However now I need to still figure out the proper way to pass float values into the proper type of Mat as needed.

I've created a function to debug my Mats and I can tell that I am creating them of at least appropriate size:

Code (CSharp):

1.     static public void debugMat(Mat incMat)
2.     {
3.         int cols = incMat.cols ();
4.         int rows = incMat.rows ();
5.  
6.         for (int i = 0; i < cols; i++)
7.         {
8.             for(int j = 0; j < rows; j++)
9.             {
10.                 Debug.Log("Row " + i + "Column " + j + "Value " + incMat.get(i,j));
11.             }
12.         }
13.  
14.     }

I have tried it with various operators of each CvType but in this manner:

Code (csharp):

1.  
2.   A.create (2, 1, CvType.CV_16S);
3. debugMat(A);
4. A.create (2, 1, CvType.CV_16U);
5. debugMat(A);
6. A.create (2, 1, CvType.CV_32F);
7. debugMat(A);
8. A.create (2, 1, CvType.CV_32S);
9. debugMat(A);
10. A.create (2, 1, CvType.CV_8S);
11. debugMat(A);
12. A.create (2, 1, CvType.CV_8U);
13. debugMat(A);
14. A.create (2, 1, CvType.CV_USRTYPE1);
15. debugMat(A);

However the debug result is the same each time:

I've tried many different ways to set float values into the Mats using .put(). I have tried adding double[] and recalling them at an index and still not receiving a value. Any ideas?

 

Code (CSharp):

1. Vector2 vec2 = new Vector2 (100.5f, 200.2f);
2. Mat matV = vec2toMatV (vec2);
3. Debug.Log ("matV " + matV.ToString ());
4. Debug.Log ("matVdump " + matV.dump ());
5.  
6. float f = (float)matV.get (0, 0) [0];
7. Debug.Log ("f " + f);
8.  
9. float[] f_Array = new float[matV.channels ()];
10. matV.get (0, 0, f_Array);
11. Debug.Log ("f_Array " + f_Array [0]);
12.  
13.  
14. static public Mat vec2toMatV (Vector2 vec2)
15. {
16.     Mat tempMatofFloat = new Mat (1, 2, CvType.CV_32FC1);
17.          
18.     tempMatofFloat.put (0, 0, vec2.x);
19.     tempMatofFloat.put (0, 1, vec2.y);
20.  
21.     return tempMatofFloat;
22. }

 

Oh thank you so very much sir that is awesome!

 

Can this be used to do edge detection ?

 

Yes, "Canny()" is implemented.
http://docs.opencv.org/java/org/opencv/imgproc/Imgproc.html#Canny(org.opencv.core.Mat, org.opencv.core.Mat, double, double, int, boolean)

 

Can i use this for Text Recognition (OCR)?

 

Unfortunately the OCR method is not implemented.

 

Released Version 1.0.8

Version changes
1.0.8
[Common]Update to OpenCV2.4.10

 

Hi,

I'm wondering if the face detection works in the latest version?
It seems nothing happened when I used camera to detect my face.

 

Does DetectFaceSample work fine?
If the red frame is not displayed, you might have failed to read the "haarcascade_frontalface_alt.xml".
Please confirm whether there is ”StreamingAssets”folder at the right position.

 

Thanks. It works like a charm.
One more question, it seems that the frame rate is pretty low.
I'm wondering how I can improve the performance?

 

Can improve speed by adjusting the parameter of the sample scene.

• Adjusting the parameters of detectMultiScale (). (minSize etc)
• Change Mat to handle in detectMultiScale() to the smaller size.

 

Hi!
Thanks for this useful plug in.
I bought and installed the plug in today.
Moved the sample file that I wanted to try in 'streaming assets' folder and also moved the opencv bundle to assets folder too.
I constantly keep on getting this error in unity editor - "ArgumentException: The output Mat object has to be of same size"
I am using a Mac system OS 10.9.4.

I tried changing the width and height variable in webcamTexture but it doesn't help. I also tried changing the unity game player size through edit>project setting>player>resolution and presentation.
What am I doing wrong?

Thank you!
-Priya

 

Thank you for an error report.

Sample worked without a problem when I tested it in my environment.(OSX 10.9.4 ,Unity 4.5.5)
I'm sorry, a setup method of ReadMe.pdf was wrong. Please move a file in reference to a capture image.

Mac Standalone Setup
×　Copy from “OpenCVForUnity/Plugins/opencvforunity.bundle” to “Assets/Plugins/opencvforunity.bundle” folder.
○　Copy from “OpenCVForUnity/Plugins/opencvforunity.bundle” to “Assets/Plugins/” folder.

 

Can we do Gestures recognitions? like Hand, mouth, and teeth etc.

 

In ”OpenCV for Unity”, ”Gestures recognitions” is not implemented as API.
Because "OpenCV for Unity" is a clone of "OpenCV Java ",I think that it is feasible if there is an example implementing "Gestures recognitions" using "OpenCV Java".

 

Hello,

We are using the Cascade Classifer for face features detection and we want to use the rejectLevels and levelWeights of the detectMultiScale function but they don't seem to work even with outputRejectLevels set to true.

Aren't they supposed to return a value fore each detected Rect?

 

Is it possible to use "detectMultiScale" on a rotated image somehow? I want to be able to take into account the user turning the phone to a portrait mode for example but I don't know how to tell the cascade which direction is up!

 

According to the following pages, it seems to be caused by a bug of OpenCV.
http://haoxiang.org/2013/11/opencv-detectmultiscale-output-detection-score/
http://answers.opencv.org/question/7270/detectmultiscale-using-rejectlevels-and-levelweights/
http://code.opencv.org/issues/3064

 

Hello,
Does this plugin support unity 5? Is it safe to upgrade without having any problems?
Thank you in advance.

 

"OpenCV for Unity 1.0.9(Support for Unity5)" is in pending review now in AssetStore.
I think that it is approved a few days later.

As for the current version1.0.8, only Android and Win and Mac support Unity5.
But, iOS does not support.
If you do not use iOS, there is not the problem with upgrading.

 

Will this remove the limitations on editor and standalone for non-pro users?

 

Of Course The limitations on editor and standalone for non-pro users is removed in Unity5.

 

http://forum.unity3d.com/threads/released-opencv-for-unity.277080/#post-1880419

 

I'm having problems getting the camera to work in WebCamTextureDetectFaceSample on iOS. I'm using ios8, iphone 6, xcode 6, unity 4.6.3f1 pro. I've got the sample working in the unity editor and osx standalone, and the iOS version builds and links properly, so I'm mostly sure that it is set up correctly. The sample just doesn't show the camera feed on the iphone 6. I'm getting the same issue on an ipad2 with ios 7.

 

In the current Version "OpenCV for Unity1.0.8"
Errors happen when set "Scripting Backend - IL2CPP"　more than Unity4.6.2.

But,This bug is fixed in next versions.

"OpenCV for Unity 1.0.9(Support for Unity5)" is in pending review now in AssetStore.
I think that it is approved a few days later.

 

Thanks for your reply. I've now got the 1.0.9 update and am running in Unity 5.

I have the WebCamTextureDetectFaceSample up and running fine on an ipad2/ios7, but it is still failing to show the camera on the iphone6/io8.

 

Is it possible to implement a QR code reader with this asset?

 

If use webCamTextureToMat(), Please choose OpenGL ES 2.0 or 3.0 in [PlayerSettings][Other Settings]-[Configuration]-[Graphics API].

Because error occurs in GetPixels32() of WebCamTexture in iOS,"OpenCV for Unity1.0.9" uses LowLevelGraphics in iOS.
LowLevelGraphics does not support Metal now, please choose OpenGL.

 

Released Version 1.0.9

Version changes
1.0.9
[Common] Support for Unity5.

 

Got it, that worked. Thanks!

 

MarkerBased AR Sample
Because I succeed in recognizing a marker in an upper example,Probably I think that the reading of the QR cord is possible.

 

Thanks for this great plugin.
Is there a schedule for implementing openCV 3.0.0?
I'm really looking forward to using OCRTesseract in unity.

 

Hi,

I need to find walls in a room and set the camera perspective ,aproximatelly with the image perspective. Like this:

.
The objective is avoid to set the perspective manually (

), I need to make it automatically
Can I make something like that with OpenCV plugin? Can you guys guide me with it?.
Thanks you so much!

 

Hey Enox,

Should there be a video.BackgroundSubtractor operator in the code somewhere? I am currently unable to find it.

Otherwise everything is looking great!
Thanks.

 

Hello -- is Imgproc.goodFeaturesToTrack implemented?

I am using this call:
Imgproc.goodFeaturesToTrack(rgbaMat, keypoints, 1000, 0.01, 10);

And it never finds features (I've played around with the numbers to see if that was it). rgbaMat is a Mat which is from the webcam.
I am populating it as in your WebCam examples, using Utils.webCamTextureToMat (webCamTexture, rgbaMat, colors);

To give a little more context,:

if (webCamTexture.width > 16 && webCamTexture.height > 16) {
Utils.webCamTextureToMat (webCamTexture, rgbaMat, colors);
if(lastRgbaMat != null)
{
MatOfPointkeypoints = newMatOfPoint();
Imgproc.goodFeaturesToTrack(rgbaMat, keypoints, 1000, 0.01, 10);

Debug.Log("this many good features:"+keypoints.toArray().Length);

lastRgbaMat exists, and the log prints out successfully.

Looking for any kind of helpy.

Thank you

 

Does DescriptorMatcher.FLANNBASED work?

I've tried using flann in the Feature2DSample and in my own project, and it seems to return no matches no matter what.

-joe

 

"OpenCV for Untiy" which implemented "OpenCV 3.0.0" is under development now.
I am going to implement OCRTesseract.

 

Thanks, that's great news.

 

I think that it is possible if you can detect the line of the wall of the room using OpenCV method(example Imgproc.HoughLines() , Imgproc.findContour()...).

 

Hello again,

Thank you for your quick reply.

I would now like to know what is the most efficient to load textures from resources folder in a Mat. Right now I am using Resources.Load(file) as Texture and then convert that texture to a mat.
It is important to find the most efficient way since I will be using a large image set for my application.

Thank you in advance!

P.S. I am targeting mobile platforms (Android and iOS but mainly Android)

 

Because ”OpenCV for Unity” is a clone of "OpenCV Java", please check a cord of "OpenCV Java".
http://stackoverflow.com/questions/9246528/opencv-android-background-subtraction

 

parameter image of Imgproc.goodFeaturesToTrack must be single-channel.

http://docs.opencv.org/java/org/ope....opencv.core.MatOfPoint, int, double, double)
Parameters:
image - Input 8-bit or floating-point 32-bit, single-channel image.