[RELEASED] OpenCV for Unity

@EnoxSoftware: I have PM'ed you.

 

hi
I see a lot of OpenCV Hand and Finger Detecting Videos on youtube.
All sampels use a Color Technology to detect Hand and Finger.
it is possible with your plugin, and have you a sample for testing



THX

 

Probably I think that it is feasible.
All the functions of OpenCV Java 2.4.11 are available in ”OpenCV for Unity”.
Please look for an Hand and Finger Detecting example by the OpenCV Java.

 

OpenCV for Unity
Released Version 1.2.1

Version changes
1.2.1
[Common] Add Beta5 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”(Add Linux support).
[Common]Rewrite SampleScene.

 

Hi,
First of all.. this plugin is awesome.

But I have a little error (maybe a bug).
When I initialize a Video Capture with a Video-URL Path, it isn't realy opened.

Example:

Code (CSharp):

1.     IEnumerator SOTMain()
2.     {
3.         capture = new VideoCapture(Utils.getFilePath(videoFeed));
4.         yield return new WaitForSeconds(2.0f); // Waiting for video
5.  
6.         if( !capture.isOpened() )
7.         {
8.             Debug.LogError("Acquiring Video Feed is not possible");
9.             yield break;
10.         }
11.  
12.         yield return new WaitForSeconds(1.0f);
13.         if( capture.read(frame1) )
14.         {
15.            
16.         }
17.         else
18.         {
19.             Debug.LogError("Weird.");
20.         }
21.  
22.         yield return new WaitForSeconds(1.0f);
23.         Debug.Log("ende");
24.    
25.     }

The path is correct (I tested it with File.Exists) but I got the message "Acquiring Video Feed is not possible");

Unity 5.2.1f1
Windows 10 Editor
Video-Format: .avi

Thanks

Yours sincerley,
Jordan

 

The setting of the pass to ffmpeg may be necessary.
Please refer to Win Standalone Setup Procedure of ReadMe.pdf.

 

THANK YOU!

 

Hey,

Currently working on spawning a prefab at a position based on the detected marker. Do you have any ideas on how to accomplish this?

 

Hello, I just bought your package today. Seems like exactly what I need.

Some questions:
1. Do you have any explanations for the samples? Like what exactly is each sample demonstrating?
2. What is the difference between WebCamTextureAsyncDetectFaceSample and WebCamTextureDetectFaceSample? What is the Async version actually doing?

I'm sorry if these are noobish questions as I'm very new to OpenCV.
Thank you

 

Please refer to a sample code of "OpenCVObjectDetector".
https://github.com/EnoxSoftware/OpenCVObjectDetector/blob/master/Sample/ObjectDetectSample.cs

 

EDIT: I am an idiot. The opencv2.framework in iOS was put there by me, and was used by the C++ code that I just rewrote with OpenCVForUnity... Probably should delete that...

On building for iOS, we get an immediate error "plugins collided with each other".

I'm assuming that you have 2 opencv.frameworks (in ios/opencv2.framework and iosForXCode/opencv2.framework) for some reason. How do you want us to set those up (assuming some manual step is involved)?​

 

WebCamTextureAsyncDetectFaceSample is the sample which speeded up WebCamTextureDetectFaceSample by asynchronous of the face detection.

 

Thanks for the link. I was able to get the position already by taking the detected markers points and calculating the center from them. I'm more curious about matching rotation of the marker and the distance from the camera.

For distance, I have an idea that I can base it off of the scale of the marker (I calculate scale from the length of an edge), which may serve my needs for detecting if the marker is being held or not.

I am stuck on matching the rotation, the closest thing I can think of being the right direction is having to do some perspective transformation but haven't had any luck.

 

in opticalflowsample, what are the variables cornersPrev and cornersThis?

 

it is possible to detect those found points grew in scale?

 

The cornersPrev and cornersThis variables in OpticalFlowSample has what meaning ?

 

cornersPrev is corner points of the previous frame.
cornersThis is corner points of the current frame.

 

Hello Enox
I build OpencvforUnity on andriod it work but
I build OpencvforUnity in iphone5s on iOS 9
Scene OpticalFlowSample,CamShiftSample,WebCamTextureToMatSample,ComicFilterSample,WebCamTextureAsyncDetectFaceSample,WebCamTextureDetectFaceSample cannot show result in real time
but other scene is work
Please Help me!!!!!!

 

Thanks, And What happens to these WHEN THE variables are changed constantly points ?

 

What are the Unity version you are using?

WebCamTexture.width/height bugs exist in the Unity5.2.
http://issuetracker.unity3d.com/issues/ios-webcamtexture-dot-width-slash-height-always-returns-16

if Unity5.2, Please insert the following code.

Code (CSharp):

1. while (true)
2.             {
3.                 //If you want to use webcamTexture.width and webcamTexture.height on iOS, you have to wait until webcamTexture.didUpdateThisFrame == 1, otherwise these two values will be equal to 16. (http://forum.unity3d.com/threads/webcamtexture-and-error-0x0502.123922/)
4. #if UNITY_IOS && !UNITY_EDITOR && (UNITY_4_6_3 || UNITY_4_6_4 || UNITY_5_0_0 || UNITY_5_0_1)
5.                                 if (webCamTexture.width > 16 && webCamTexture.height > 16) {
6. #else
7.                 if (webCamTexture.didUpdateThisFrame)
8.                 {
9. ///////////////////////////////////////////////////////////////////////////////////
10. #if UNITY_IOS && !UNITY_EDITOR && UNITY_5_2                                  
11.                                     while (webCamTexture.width <= 16)
12.                                     {
13.                                         webCamTexture.GetPixels32();
14.                                         yield return new WaitForEndOfFrame();
15.                                     }
16. #endif
17. ///////////////////////////////////////////////////////////////////////////////////
18. #endif
19.  
20.                     Debug.Log("width " + webCamTexture.width + " height " + webCamTexture.height + " fps " + webCamTexture.requestedFPS);
21.                     Debug.Log("videoRotationAngle " + webCamTexture.videoRotationAngle + " videoVerticallyMirrored " + webCamTexture.videoVerticallyMirrored + " isFrongFacing " + webCamDevice.isFrontFacing);

 

hi
I work with your Sample from Github VuforiaWithOpenCVForUnitySample.unity. It work fine for me. I have analyzed the 2 scripts CameraImageToMatSample.cs and PostRenderToMatSample.cs.
Now i compare your Script DetectFaceSample.cs for detecting the Woman Face and the Vuforia Functionality.
Can i ad this Script to the Camera under the ARCamera as your PostRenderToMatSample.
Can you gif me a tip or sample that i can use this Script with vuforia.

Thanks for help

 

Hey,

I am thinking of buying this plugin. Why might I choose your plugin over this library https://github.com/shimat/opencvsharp ?

My project needs are video encoding. Am I right in saying your plugin does not implement this functionality?

 

Advantage of ”OpenCV for Unity” is that it supports the iOS and Android.
However, VideoWriter Class is not implemented.

 

Hi
Nice job, I realy enjoy playing with OpenCV inside Unity.
I have a few questions:
- Do you plan to implement Common Interfaces like AdjusterAdapter and DynamicAdaptedFeatureDetector ?
- How to perform simple matrix multiplication ? The Mat operator * is not overriden...
Thanks for help

 

There is no plan implementing Common Interfaces now.

In the same way as a case using ”OpenCV Java”, please use Core.gemm().
http://enoxsoftware.github.io/OpenC...1_core.html#a7ec35e6f5d2b7cb428cb5bae2c1385ed
http://answers.opencv.org/question/10770/product-of-two-matrices/

 

in cornersPrev and cornersThis variables, when the current frame is new points he no arrow in the above it?

 

Code (CSharp):

1. Mat CM  =  new Mat( 3, 3, CvType_32FC1);
2.  
3. //C++ Code i'd like to translate to OpenCVForUnity C#
4.  
5. CM.at<float>(0,0) = 1;
6. CM.at<float>(1,1) = 1;

Hey there!

First of all your plugin rocks indeed!!! I'm rather new to both: Unity-C# and OpenCV so please alow me to ask a super noob question (i'm currently working on camera-calibration-script). Therefore i'm trying to access/set a single channel of an "Mat"-Object. In C++ you can use ".at"; in OpenCVSharp there is a ".mSet" method. But besides the "setTo"-method i can't find a "easy" way of setting up/change a single mat-value. So lets say you have a new 3x3Matrix (all channels are 0) and you want to set Mat-channel[0][0] = value 1 and Mat-channel[1][1] = value 1. How to do that? Thanks so much in advance!!!!

 

Hi,

I'm currently working on the same thing.

For this I used

Code (CSharp):

1. cameraMatrix = new Mat(3, 3, OpenCVForUnity.CvType.CV_32FC1);
2.  
3. cameraMatrix.put(0, 0, 1);
4. cameraMatrix.put(1, 1, 1);

I'm currently facing an issue right now though when I call the Calib3d.calibrateCamera I get NullReferenceException: Object reference not set to an instance of an object error it doesn't offer any more detail as to what is causing this though.

I you have any luck let me know

 

Hi,

I'm Currently looking to use the Calib3d.calibrateCamera

static double calibrateCamera (List< Mat > objectPoints, List< Mat > imagePoints, Size imageSize, Mat cameraMatrix, Mat distCoeffs, List< Mat > rvecs, List< Mat > tvecs)

We i call the function i get the following error:

NullReferenceException: Object reference not set to an instance of an object
OpenCVForUnity.Calib3d.calibrateCamera (System.Collections.Generic.List`1 objectPoints, System.Collections.Generic.List`1 imagePoints, OpenCVForUnity.Size imageSize, OpenCVForUnity.Mat cameraMatrix, OpenCVForUnity.Mat distCoeffs, System.Collections.Generic.List`1 rvecs, System.Collections.Generic.List`1 tvecs, Int32 flags) (at Assets/OpenCVForUnity/org/opencv/calib3d/Calib3d.cs:553)
WebCameraCalibration.OpenCVCalibrateCamera () (at Assets/StereoCameraCalibration/Scripts/02 - WebCamera/WebCameraCalibration.cs:262)

This doesn't really help me to figure out what the issue may be so if you could please have a look at the following and see if there's anything you can spot as to why this error is produced.
My object points and imagepoints are defined as follows with the image points being filled earlier in the code:

Code (CSharp):

1. //assign object points for each image - Must be equal in size to imagePoints List
2.         //in this case we use the same chess board for each image so we will use the same values for each intem in the list
3.         List<Mat> objectPoints = new List<Mat>();
4.  
5.         //create new mat and add to list
6.         for(int i =0; i < matCornerPositionsList.Count;i++)
7.         {
8.             Mat objectPointsMat = new Mat((int)size.height*(int)size.width,3,0);
9.  
10.             for (int j = 0; j < boardCorners;j++)
11.             {
12.                 Mat vectorMat = new Mat(3, 1, OpenCVForUnity.CvType.CV_32FC1);
13.                 vectorMat.put(0, 0, j / boardWidth);
14.                 vectorMat.put(0, 1, j % boardWidth);
15.                 vectorMat.put(0, 2, 0f);
16.  
17.                 objectPointsMat.push_back(vectorMat);
18.             }
19.  
20.             Debug.Log(objectPointsMat);
21.  
22.             objectPoints.Add(objectPointsMat);
23.         }
24.  
25.  
26.         //convert imagepoints to List<Mat>
27.         List<Mat> imagePoints = new List<Mat>();
28.  
29.         foreach(MatOfPoint2f p in matCornerPositionsList)
30.         {
31.             Point[] cornerPoints = p.toArray();
32.  
33.             Mat imagePointsMat = new Mat((int)size.height * (int)size.width, 3, 0);
34.  
35.             foreach (Point point in cornerPoints)
36.             {
37.                 Mat vectorMat = new Mat(3, 1, OpenCVForUnity.CvType.CV_32FC1);
38.                 vectorMat.put(0, 0, point.x);
39.                 vectorMat.put(0, 1, point.y);
40.                 vectorMat.put(0, 2, 0f);
41.  
42.                 if(point.x == null)
43.                 {
44.                     Debug.LogError("Point X null");
45.                 }
46.  
47.                 if (point.y == null)
48.                 {
49.                     Debug.LogError("Point Y null");
50.                 }
51.  
52.                 imagePointsMat.push_back(vectorMat);
53.             }
54.  
55.             Debug.Log(imagePointsMat.get(0,0)[0]);
56.  
57.             imagePoints.Add(imagePointsMat);
58.         }

The size is the size of the images used to find the checkerboard pattern, defined as follows:

Code (CSharp):

1. Size imageSize = new Size(calibrationTextureList[0].width, calibrationTextureList[0].height);

and the rest of the arguments:

Code (CSharp):

1. cameraMatrix = new Mat(3,3,OpenCVForUnity.CvType.CV_32FC1);
2. //init camera matrix
3. cameraMatrix.put(0, 0, 1);
4. cameraMatrix.put(1, 1, 1);
5.  
6. distCoeffs = new Mat();
7. translationVectors = new List<Mat>();
8. rotationVectors = new List<Mat>();

Any help would be appreciated, Thanks

 

Super cool! Thanks a lot Marc! May i ask another simple question like how to debug a opencv mat? Means how to display the stored values? In C++ you use cout << mat bla ... the opencvforunity documention is rather limited with examples and although a lot of great c++ examples are out there it's really hard to find good c#-documentations

 

Hey Marc,

since i'm new to this i'm not sure but after studying your code i guess your errors are coming from the fact that you initialize the Mat vectorMat with 3 rows and 1 column but then you assign values via .put to just the first row-element but for 3 columns. But if you have done this by purpose i'd like to know why!

 

hate ios9~~~~~~

 

Your right that was a mistake. still getting an error when I call CalibrateCamera.

Think it is an issue with the data or not set up as the proper structure

 

I have a problem. I wanted to know how to check the range of points within the video.
it's possible?

 

If cornersPrev and cornersThis are of the same value, the line connecting the cornersPrev and cornersThis will not be displayed.

 

Use the following page as a reference, I made a sample code.
http://computervisionandjava.blogspot.jp/2013/10/camera-cailbration.html

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using System.Collections.Generic;
4.  
5. using OpenCVForUnity;
6.  
7. namespace OpenCVForUnitySample
8. {
9.         /// <summary>
10.         /// CameraCalibrationSample
11.         /// </summary>
12.         public class CameraCalibrationSample : MonoBehaviour
13.         {
14.  
15.  
16.                 int flagsCorner = Calib3d.CALIB_CB_ADAPTIVE_THRESH
17.                         | Calib3d.CALIB_CB_FAST_CHECK
18.                         | Calib3d.CALIB_CB_NORMALIZE_IMAGE;
19.                 int flagsCalib = Calib3d.CALIB_ZERO_TANGENT_DIST
20.                         | Calib3d.CALIB_FIX_PRINCIPAL_POINT
21.                         | Calib3d.CALIB_FIX_K4
22.                         | Calib3d.CALIB_FIX_K5;
23.                 TermCriteria criteria = new TermCriteria (TermCriteria.EPS
24.                         + TermCriteria.MAX_ITER, 40, 0.001);
25.                 Size winSize = new Size (5, 5), zoneSize = new Size (-1, -1);
26.                 Size patternSize;
27.                 List<Mat> objectPoints, imagePoints = new List<Mat> ();
28.                 List<Mat> vCorners;
29.                 List<Mat> vImg;
30.                 Mat cameraMatrix = Mat.eye (3, 3, CvType.CV_64F);
31.                 Mat distCoeffs = Mat.zeros (8, 1, CvType.CV_64F);
32.                 List<Mat> rvecs = new List<Mat> ();
33.                 List<Mat> tvecs = new List<Mat> ();
34.        
35.                 bool getCorners (Mat gray, MatOfPoint2f corners)
36.                 {
37.                         if (!Calib3d.findChessboardCorners (gray, patternSize,
38.                                                corners, flagsCorner))
39.                                 return false;
40.                         Imgproc.cornerSubPix (gray, corners, winSize, zoneSize,
41.                                  criteria);
42.                         return true;
43.                 }
44.        
45.                 MatOfPoint3f getCorner3f ()
46.                 {
47.                         MatOfPoint3f corners3f = new MatOfPoint3f ();
48.                         double squareSize = 50;
49.                         Point3[] vp = new Point3[(int)(patternSize.height *
50.                                 patternSize.width)];
51.                         int cnt = 0;
52.                         for (int i = 0; i < patternSize.height; ++i)
53.                                 for (int j = 0; j < patternSize.width; ++j, cnt++)
54.                                         vp [cnt] = new Point3 (j * squareSize,
55.                                          i * squareSize, 0.0d);
56.                         corners3f.fromArray (vp);
57.                         return corners3f;
58.                 }
59.        
60.                 void calibrate ()
61.                 {
62.                         double errReproj = Calib3d.calibrateCamera (objectPoints,
63.                                                        imagePoints, vImg [0].size (), cameraMatrix,
64.                                                        distCoeffs, rvecs, tvecs, flagsCalib);
65.                         Debug.Log ("done, \nerrReproj = " + errReproj);
66.                         Debug.Log ("cameraMatrix = \n" + cameraMatrix.dump ());
67.                         Debug.Log ("distCoeffs = \n" + distCoeffs.dump ());
68.                 }
69.        
70.                 void getAllCornors (string path)
71.                 {
72.                         vImg = new List<Mat> ();
73.                         objectPoints = new List<Mat> ();
74.                         imagePoints = new List<Mat> ();
75.                         MatOfPoint3f corners3f = getCorner3f ();
76.  
77.                         for (int i = 1; i < 15; i++) {
78.    
79.                                 Mat mat = Highgui.imread (path + i.ToString ("D2") + ".jpg",
80.                                          Highgui.CV_LOAD_IMAGE_COLOR);
81.  
82.                                 if (mat == null || mat.channels () != 3)
83.                                         continue;
84.  
85.                                 Debug.Log ("mat.channels() = " + mat.channels ()
86.                                         + ", " + mat.cols () + ", " + mat.rows ());
87.                                 Mat gray = new Mat ();
88.                                 Imgproc.cvtColor (mat, gray, Imgproc.COLOR_BGR2GRAY);
89.                                 MatOfPoint2f corners = new MatOfPoint2f ();
90.                                 if (!getCorners (gray, corners))
91.                                         continue;
92.                                 objectPoints.Add (corners3f);
93.                                 imagePoints.Add (corners);
94.                                 vImg.Add (mat);
95.                         }
96.                 }
97.  
98.                 // Use this for initialization
99.                 void Start ()
100.                 {
101.                         patternSize = new Size (8, 6);
102.                         Debug.Log ("Utils.getFilePath " + Utils.getFilePath ("left01.jpg").Remove (Utils.getFilePath ("left01.jpg").LastIndexOf ("left") + 4));
103.                         getAllCornors (Utils.getFilePath ("left01.jpg").Remove (Utils.getFilePath ("left01.jpg").LastIndexOf ("left") + 4));
104.                         calibrate ();
105.  
106.                 }
107.    
108.                 // Update is called once per frame
109.                 void Update ()
110.                 {
111.  
112.                 }
113.  
114.                 void OnGUI ()
115.                 {
116.                         float screenScale = Screen.width / 240.0f;
117.                         Matrix4x4 scaledMatrix = Matrix4x4.Scale (new Vector3 (screenScale, screenScale, screenScale));
118.                         GUI.matrix = scaledMatrix;
119.        
120.        
121.                         GUILayout.BeginVertical ();
122.                         if (GUILayout.Button ("back")) {
123.                                 Application.LoadLevel ("OpenCVForUnitySample");
124.                         }
125.        
126.        
127.                         GUILayout.EndVertical ();
128.                 }
129.         }
130. }

left01-14.jpg is here.
https://github.com/Itseez/opencv/tree/master/samples/data

 

Thanks so much! This is extremely usefull! One tiny question: In getCorners3f() you're using squareSize = 50. I guess this is the "real" size of a quad from the printed chessboard. But what does "50" means? 50mm in length or height?

 

It is to slow down the execution of the frames? or decrease the variation points found ? because the points varies greatly and is difficult to work on them

 

Maybe I think that it is such a meaning.I don't know a lot about Camera Calibration.
http://opencv-java-tutorials.readthedocs.org/en/latest/05 - Camera Calibration.html#id1

 

As i've found out this "squareSize"-value seams to be important for the extrensic parameters (especially for correcr translation values) although i couldn't exactly figure out how many "real"-units "50" means (in your example) anyways ... i guess it's mm. One more question: I'd like to save out my stored camMatrix-datas and distortionCoeffs to a file but there is no such object-type like "CvFileStorage" / "OpenCvSharp.FileStorage" like from the OpenCvSharp-class which makes it super easy to create an ".yml" file. I'm currently trying to inherit from the OpenCvSharp-class + OpenCvForUnity-class but it would be nice using just one. Thanks so much for all your support!

 

Thanks I'll have a look through this.

 

it is possible to measure depth with opticalFlow ?

 

it is possible to measure distance of the detected points ?

 

Hello Enox
I have a question.
I want to save image to folder resources
This code
Highgui.imwrite("Asssets/OpenCVForUnity/Samples/Resources/GGG.jpg",rgbaMat);
Can't work
Please Help me

 

Quick question:

When using FeatureDetector.SimpleBlob do I need a parameters object like when using OpenCV's SimpleBlobDetector class? Right now when I try to use SimpleBlob it crashes unity, but if I use FAST it doesn't crash. Any ideas?

 

Unfortunately,　Because I do not know a lot about OpenCV and image processing, I cannot answer your question.

 

Please specify the full path.
It is as follows.
Highgui.imwrite("C:/UnityProjectFolder/Asssets/OpenCVForUnity/Samples/Resources/GGG.jpg",rgbaMat);

 

@EnoxSoftware You could create a sample script of how to identify a hand or fingers? Reading on this forum I would not be the only one to need it! Especially because there are so many beginners just like me..

Or maybe a simpler example of how to track an object of a certain color!

 

The following code worked well.

Code (CSharp):

1.                         Texture2D imgTexture = Resources.Load ("blob") as Texture2D;
2.            
3.                         Mat imgMat = new Mat (imgTexture.height, imgTexture.width, CvType.CV_8UC1);
4.  
5.                         Utils.texture2DToMat (imgTexture, imgMat);
6.                         Debug.Log ("imgMat dst ToString " + imgMat.ToString ());
7.                        
8.                         Mat outImgMat = new Mat ();
9.  
10.                         FeatureDetector blobDetector = FeatureDetector.create (FeatureDetector.SIMPLEBLOB);
11.  
12.                         blobDetector.read (Utils.getFilePath ("blobparams.yml"));
13.  
14.                         MatOfKeyPoint keypoints = new MatOfKeyPoint ();
15.                         blobDetector.detect (imgMat, keypoints);
16.                         Features2d.drawKeypoints (imgMat, keypoints, outImgMat);
17.  
18.            
19.                         Texture2D texture = new Texture2D (outImgMat.cols (), outImgMat.rows (), TextureFormat.RGBA32, false);
20.            
21.                         Utils.matToTexture2D (outImgMat, texture);
22.            
23.                         gameObject.GetComponent<Renderer> ().material.mainTexture = texture;

Assets/StreamingAssets/blobparams.yml

Code (CSharp):

1. %YAML:1.0
2. thresholdStep: 20.
3. minThreshold: 10.
4. maxThreshold: 200.

Strangely, when excluding "thresholdStep: 20.", crash has occurred.

 

OpenCV for Unity
Released Version 1.2.2

Version changes
1.2.2
[iOS]Move “OpenCVForUnity/ iOSforXcode/opencv2.framework” to “OpenCVForUnity/Plugins/iOS/”folder.
[iOS]Fix WebCamTexture bug of SampleScene in Unity5.2.