[RELEASED] OpenCV for Unity

Perhaps MarkerLessARExample is useful for what you want to do.
https://www.assetstore.unity3d.com/en/#!/content/77560

 

Hi!
1. I found the strange problem: the video recorded with built-in MJPEG OpenCV "plugin" cannot be played without strange pauses in several seconds at VideoCapture.read(). But the recorded video can be played by VLC player without problems, and other videos (for instance, got on the Internet) can be played without problems using reading video file frame-by-frame (VideoCapture.read()).
Looks like internal representation of MJPEG for reading and writing are slightly different.
Pauses at VideoCapture.read() occurs randomly. The highest probability (and easiest way to reproduce) gives running in Editor mode, also reproduced at Android devices (did not test with iOs devices)

2. Do you have the versions of OpenCV for MOBILE devices build with ffmpeg?

3. It can be a good idea to provide downloadable versions of binaries in "Plugin" folder for most popular subsets of OpenCV. You did it for "with contrib modules" and "without contrib modules" versions

 

Hello!
I have a mask and a picture. I am trying to apply morphology to the mask to smoother its edge, and then cut out the picture with the mask. The mask Mat seems to be "fliped" after I apply morphology to it. I was wondering if this is a normal behavior or if I missed something?
(The 1st image is my game view screenshot, and the 2nd one is my expected result which is edited with Photoshop.)

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using UnityEngine.UI;
5. using OpenCVForUnity;
6.  
7. public class TestMorph : MonoBehaviour
8. {
9.  
10.     public RawImage img_mask;
11.     public Texture2D tex_mask;
12.     Mat mat_mask;
13.     Texture2D texToDisplay_mask;
14.  
15.     public RawImage img_picture;
16.     public Texture2D tex_picture;
17.     Mat mat_picture;
18.     Texture2D texToDisplay_picture;
19.  
20.     public RawImage img_maskMorphed;
21.     Texture2D tex_maskMorphed;
22.     Mat mat_maskMorphed;
23.     Texture2D texToDisplay_maskMorphed;
24.  
25.     public RawImage img_result;
26.     Texture2D tex_result;
27.     Mat mat_result;
28.     Texture2D texToDisplay_result;
29.  
30.     [Range(1, 500)]
31.     public int erode = 5;
32.     [Range(1, 500)]
33.     public int dilate = 8;
34.  
35.     void Start()
36.     {
37.  
38.     }
39.  
40.  
41.     void Update()
42.     {
43.  
44.         // Generate mat for picture
45.         mat_picture = new Mat(tex_picture.height, tex_picture.width, CvType.CV_8UC4);
46.         Utils.texture2DToMat(tex_picture, mat_picture);
47.         // Display
48.         texToDisplay_picture = new Texture2D(mat_picture.cols(), mat_picture.rows(), TextureFormat.RGBA32, false);
49.         Utils.matToTexture2D(mat_picture, texToDisplay_picture);
50.         img_picture.texture = texToDisplay_picture;
51.  
52.         // Generate mat for mask
53.         mat_mask = new Mat(tex_mask.height, tex_mask.width, CvType.CV_8UC1);
54.         Utils.texture2DToMat(tex_mask, mat_mask);
55.         // Display
56.         texToDisplay_mask = new Texture2D(mat_mask.cols(), mat_mask.rows(), TextureFormat.RGB24, false);
57.         Utils.matToTexture2D(mat_mask, texToDisplay_mask);
58.         img_mask.texture = texToDisplay_mask;
59.  
60.         // Generate mat for morphed mask mat
61.         mat_maskMorphed = new Mat(tex_mask.height, tex_mask.width, CvType.CV_8UC1);
62.         // Process morph
63.         MorphOps(mat_mask, mat_maskMorphed);
64.         // Display
65.         texToDisplay_maskMorphed = new Texture2D(mat_maskMorphed.cols(), mat_maskMorphed.rows(), TextureFormat.RGB24, false);
66.         Utils.matToTexture2D(mat_maskMorphed, texToDisplay_maskMorphed);
67.         img_maskMorphed.texture = texToDisplay_maskMorphed;
68.  
69.         // Generate mat for result
70.         mat_result = new Mat(tex_mask.height, tex_mask.width, CvType.CV_8UC4);
71.  
72.         // Process mask
73.         mat_picture.copyTo(mat_result, mat_maskMorphed);
74.         //Core.bitwise_and(mat_picture, mat_picture, mat_result, mat_maskMorphed);
75.  
76.         // Display
77.         texToDisplay_result = new Texture2D(mat_result.cols(), mat_result.rows(), TextureFormat.RGBA32, false);
78.         Utils.matToTexture2D(mat_result, texToDisplay_result);
79.         img_result.texture = texToDisplay_result;
80.  
81.     }
82.  
83.     private void MorphOps(Mat input, Mat output)
84.     {
85.  
86.         Mat erodeElement = Imgproc.getStructuringElement(Imgproc.MORPH_ELLIPSE, new Size(erode, erode));
87.         //Mat dilateElement = Imgproc.getStructuringElement(Imgproc.MORPH_RECT, new Size(dilate, dilate));
88.  
89.         Imgproc.morphologyEx(input, output, Imgproc.MORPH_OPEN, erodeElement);
90.     }
91.  
92. }
93.  

Thank you!

 

1. Perhaps that strange problem is a bug in OpenCV 3.4.1 itself. https://github.com/opencv/opencv/issues/11126
This bug will be fixed in the next version OpenCVForUnity.

2. I don't have the versions of OpenCV for MOBILE devices build with ffmpeg.

3. Thank you for your valuable ideas.
Currently, plugin files that exclude the contrib module for Android and iOS platform are included with OpenCVForUnity.

 

When reusing mat used for Utils.matToTexture2D (Mat mat, Texture2D texture2D, bool flip = true, int flipCode = 0, bool flipAfter = false), please set flipAfter flag to true.
http://enoxsoftware.github.io/OpenC..._utils.html#a302b2811f9ca301edcb115a73ea8ee86

Code (CSharp):

1.  
2.         // Generate mat for mask
3.         ---
4.         Utils.matToTexture2D(mat_mask, texToDisplay_mask, true, 0, true);
5.         ---
6.  

 

Thank you!!

 

Hello
I need to crop the detected face from the webcam feed as 2d texture or even mesh and put it in another place is that possible and how to achieve this I'm using dlib face landmark and please I'm a beginner so I don't know what to do

 

I think that this example will be helpful for what you want to realize.
https://www.assetstore.unity3d.com/#!/content/66602

 

Hello!
I am trying to set flipAfter flag to true as you said. However I got "No overload for method 'matToTexture2D' takes 5 arguments" error and I am not able to figure it out. It would be great if you would tell me what I've done wrong.
THank you!

 

I am new to OpenCV but working to make a face detection feature.
The AsyncFaceDetectionWebCamTextureExample works well on Unity editor but the camera texture does not show in the Android app. It just shows all the buttons and the hourglass symbol. How can I fix this?

 

hi, i have some question for opencv for unity:
i For some special reason, I use the video stream i unity , that format is "Texture Render", i want make “Texture Render” convert to "mat" ,but it only has Texture2d to mat and Cam to mat

 

Is the version of OpenCVForUnity 2.2.9 or later?

 

Does FaceDetectionWebCamTextureExample work without problems on Android?

 

There is no method to convert RenderTexture directly to Mat. Please use the Utils.textureToTexture2D () method.
https://github.com/EnoxSoftware/Ope...xample/ImwriteScreenCaptureExample.cs#L74-L75

 

I am level dsigner, totally new with coding. For a learning purpose and to know how to use Face filter concept I went searching google and finnaly got open CV is one of the way, I do have asked my whole friends they dont have Idea about this but I got Older version of Open CV 2.1.3 and tried following this tutorial

But I m getting some error in Console, as I attached image of my console

kindly let me know how to get rid of this ...


Thanks in advance

 

Could you tell me about your test environment?
OpenCV for Unity version :
Unity version :

 

Hey,
It worked with your code, thanks.


I am currently trying to track some colored edges as markers like the picture above. And I am using

moment.get_m10() /moment.get_m00()

to caculate the Xpos and

moment.get_m01() /moment.get_m00()

for Ypos.

I notice that this creates a pretty huge deviation between the real center of the triangle and the trapezoid shown on the screen, causing the overlayed picture to be unparalleled. I am wondering if there is a method around this? I have found a findCandidates function in the Arco sample where it can rule out the markers that are not square. I am wondering if we can utilize this.

It would be best if this method would be applicable in a complex background (the edges of the colored square might be blurry). I am hoping to build something that can recognize thin colored edges.

Thanks in advance.

 

I've been having a problem for some time now. I'm using Unity 2018 and opencvforunity version 2.4.1. I was testing using Android 7 nouget.

In unity when I call core.flip in order to flip my webcam texture in works. However, on an Android build, the webcam texture displays the image at 90 degrees. It doesn't detect my face on android unless I turn the phone horizontally to match the flipped webcamtexture.

When I debugged I saw that in android both the src mat and dst mat's nativeObj intptr has a negative number. I'm guessing this is pointing to junk, because in unity on my computer the intptr is a positive number.

Any idea why Core.flip isn't working in android?

 

Is there the simple code to reproduce this problem?
Also, Is there the screenshot of bad results?

 

It's basically the problem that this person has/had: https://stackoverflow.com/questions/28927223/webcamtexture-rotation-differs-in-unity3d-under-android.

I used the following code to try to flip and fix it:

Code (CSharp):

1. if (webCamDevice.isFrontFacing)
2.                 {
3.                     if (webCamTexture.videoRotationAngle == 0)
4.                     {
5.                         Core.flip(rgbaMat, rgbaMat, 1);
6.                     }
7.                     else if (webCamTexture.videoRotationAngle == 90)
8.                     {
9.                         Core.flip(rgbaMat, rgbaMat, 0);
10.                     }
11.                     else if (webCamTexture.videoRotationAngle == 270)
12.                     {
13.                         Core.flip(rgbaMat, rgbaMat, 1);
14.                     }
15.                 }
16.                 else
17.                 {
18.                     if (webCamTexture.videoRotationAngle == 90)
19.                     {
20.  
21.                     }
22.                     else if (webCamTexture.videoRotationAngle == 270)
23.                     {
24.                         Core.flip(rgbaMat, rgbaMat, -1);
25.                     }
26.                 }

The code I'm using is different though. Mine is basically the code for the FaceDetections example
https://github.com/EnoxSoftware/Ope...nExample/FaceDetectionWebCamTextureExample.cs

In update though before I show the rectangle I try to flip the webcamtexture using this code:

Code (CSharp):

1. if (webCamDevice.isFrontFacing)
2.                 {
3.                     if (webCamTexture.videoRotationAngle == 0)
4.                     {
5.                         Core.flip(rgbaMat, rgbaMat, 1);
6.                     }
7.                     else if (webCamTexture.videoRotationAngle == 90)
8.                     {
9.                         Core.flip(rgbaMat, rgbaMat, 0);
10.                     }
11.                     else if (webCamTexture.videoRotationAngle == 270)
12.                     {
13.                         Core.flip(rgbaMat, rgbaMat, 1);
14.                     }
15.                 }
16.                 else
17.                 {
18.                     if (webCamTexture.videoRotationAngle == 90)
19.                     {
20.  
21.                     }
22.                     else if (webCamTexture.videoRotationAngle == 270)
23.                     {
24.                         Core.flip(rgbaMat, rgbaMat, -1);
25.                     }
26.                 }

The code works in unity editor but in Android, the image stays flipped at 90 degrees. The image never gets flipped. I know this because when I debug the intptr of the mat objects nativeObj is equal to a negative number. Interestingly, this doesn't cause the app to close and it doesn't seem to give me errors. Kind of hard to tell though because even with filtering out only unity there's a lot of stuff that logcat shows.

Could there be something wrong with the way I set up OpenCV? Does OpenCV normally work in Unity Android build?

 

It seems that it works without problems in my environment.
OpenCVforUnity version :2.3.1
Unity version : 2018.2.5f1
Device : Nexus7

Could you send the code you tested to Technical Inquiry?
https://enoxsoftware.com/opencvforunity/contact/technical-inquiry/

 

Hi @EnoxSoftware, I'm trying to develop semantic segmentation using OpenCVforUnity based on torch_enet.cpp
But I don't know how to write colorizeSegmentation function in torch_enet.cpp using OpenCVforUnity.
I tried to access mat data returned from blobFromImage function referencing "Accessing Pixel Values" example, but it didn't work maybe because the blob is 4 dimensional mat and its cols and rows are -1.
I tried to copy its data from the blob to a byte array using Utils.copyFromMat but get no luck because I couldn't find size array of mat in OpenCVforUnity and define the size of the byte array properly.
Do you have any good examples for that?
Or do I need to reshape the blob mat? If so, what size is appropriate for the result of enet?
Thank you in advance.

 

Unfortunately,I do not have an example of SemanticSegmentation.

Since this package is a clone of OpenCV Java, you are able to use the same API as OpenCV Java 3.4.2. OpenCV Java does not seem to be able to get the size of multidimensional Mat. However, Mat.total () method seems to work.
Perhaps the blobs that can be obtained from "Enet-model-best.net" are as follows.
total 10485760 = 1024(width) * 512(height) * 20(classes)

 

Hi
Trying to modify 2 of the MainModules in the objdetect folder
1- Add webcam support to the "HOGDescriptorExample"
and
2- Allow detection of smaller faces in the "AsynchronousFaceDetectionWebCamTextureExample"

For the latter could you tell me specifically what needs changing in the AsynchronousFaceDetectionWebCamTextureExample.ca class?

Any feedback appreciated!

 

Hello eco_bach.

1- I don't have an example combining WebCamTexture and HOGDescriptor.
Could you refer to an example of using WebCamTexture? (cf. FaceDetectionWebCamTextureExample or ComicFilterExample)
2- Could you adjust the parameters of the CascadeClassifier.detectMultiScale function?
https://docs.opencv.org/3.0-beta/modules/objdetect/doc/cascade_classification.html
(minSize – Minimum possible object size. Objects smaller than that are ignored.)
In order to detect smaller faces, you need to adjust the "minSize" to a smaller size.

 

Thanks!

 

Thanks for your suggestion!
I hard-coded the width and height as a workaround and now ENet is working on Unity properly.
If I find any good ways to get the size information of the blob, I will share it here.

 

Hello!
I'm trying to make a game with aruco marker for mac and windows. I'm using example ArUcoWebCamTextureExample and I have 6-8 fps on mac and 10fps on windows. Is there any way to increase fps?
Can I change webcamera resolution? Can it increase fps?

 

Hello All and Enox,

I know you have shared good directions in terms of writing this code for Panorama stitching. However, the result is only 2nd image rather than stitched image. Below is the code. It would be great if somebody has an answer to this as this code is very close to whats written here in this link which is in Java. Thanks in advance.

http://privateblog.info/sozdanie-panoramy-s-pomoshhyu-opencv-i-java/

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using System.Collections.Generic;
4.  
5. #if UNITY_5_3 || UNITY_5_3_OR_NEWER
6. using UnityEngine.SceneManagement;
7. #endif
8. using OpenCVForUnity;
9.  
10. namespace OpenCVForUnityExample
11. {
12.     public class Panorama : MonoBehaviour
13.     {
14.  
15.         // Use this for initialization
16.         void Start()
17.         {
18.  
19.             Texture2D imgTexture = Resources.Load("img_11") as Texture2D;
20.  
21.             Mat img1Mat = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC3);
22.             Utils.texture2DToMat(imgTexture, img1Mat);
23.             Debug.Log("img1Mat.ToString() " + img1Mat.ToString());
24.  
25.             Texture2D img2Texture = Resources.Load("img_12") as Texture2D;
26.  
27.             Mat img2Mat = new Mat(img2Texture.height, img2Texture.width, CvType.CV_8UC3);
28.             Utils.texture2DToMat(img2Texture, img2Mat);
29.             Debug.Log("img2Mat.ToString() " + img2Mat.ToString());
30.  
31.             Texture2D img3Texture = Resources.Load("3") as Texture2D;
32.  
33.             Mat img3Mat = new Mat(img3Texture.height, img3Texture.width, CvType.CV_8UC3);
34.             Utils.texture2DToMat(img3Texture, img3Mat);
35.             Debug.Log("img3Mat.ToString() " + img3Mat.ToString());
36.  
37.             Mat pr1 = Pano(img1Mat, img2Mat, 1024, 1024);
38.             //Mat pr2 = Pano(img3Mat, pr1, 1024, 1024);
39.  
40.             Texture2D texture = new Texture2D(pr1.cols(), pr1.rows(), TextureFormat.RGBA32, false);
41.  
42.             Utils.matToTexture2D(pr1, texture);
43.  
44.             gameObject.GetComponent<Renderer>().material.mainTexture = texture;
45.  
46.         }
47.  
48.         // Update is called once per frame
49.         void Update()
50.         {
51.  
52.         }
53.  
54.         Mat Pano(Mat src1, Mat src2, int width, int height)
55.         {
56.  
57.             Mat gray1 = new Mat();
58.             Mat gray2 = new Mat();
59.  
60.             // Descriptors
61.             Mat des1 = new Mat();
62.             Mat des2 = new Mat();
63.  
64.             ORB detector = ORB.create(); // 3000 is the size of list matches that will be computed
65.             ORB extractor = ORB.create();
66.  
67.            
68.             MatOfKeyPoint keyPoints1 = new MatOfKeyPoint();
69.             MatOfKeyPoint keyPoints2 = new MatOfKeyPoint();
70.  
71.             Imgproc.cvtColor(src1, gray1, Imgproc.COLOR_RGB2GRAY);
72.             Imgproc.cvtColor(src2, gray2, Imgproc.COLOR_RGB2GRAY);
73.  
74.             detector.detect(gray1, keyPoints1);
75.             detector.detect(gray2, keyPoints2);
76.  
77.             extractor.compute(gray1, keyPoints1, des1);
78.             extractor.compute(gray2, keyPoints2, des2);
79.  
80.             DescriptorMatcher matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMINGLUT);
81.  
82.             MatOfDMatch matches = new MatOfDMatch();
83.  
84.             matcher.match(des1, des2, matches);
85.  
86.             // Vector stuff and for loop goes here
87.  
88.             double max_dist = 0;
89.             double min_dist = 100;
90.  
91.             List <DMatch> listMatches = matches.toList();
92.  
93.             for (int i = 0; i < listMatches.Count; i++)
94.             {
95.                 double dist = listMatches[i].distance;
96.                 if (dist < min_dist) min_dist = dist;
97.                 if (dist > max_dist) max_dist = dist;
98.             }
99.  
100.             Debug.Log("Min:" + min_dist);
101.             Debug.Log("Max:" + max_dist);
102.  
103.             List<DMatch> good_matches = new List<DMatch>();
104.  
105.             MatOfDMatch goodMatches = new MatOfDMatch();
106.  
107.             for (int i = 0; i < listMatches.Count; i++)
108.             {
109.              
110.                 if (listMatches[i].distance < 2 * min_dist)
111.                 {
112.                     good_matches.Add(listMatches[i]); // This is just add in list instead of addlast in linked list
113.                 }
114.             }
115.  
116.             goodMatches.fromList(good_matches);
117.  
118.             Mat result = new Mat(new Size(src1.cols() + src2.cols(), src1.rows()), CvType.CV_32FC2);
119.  
120.             List<Point> imgPoints1List = new List<Point>();
121.             List<Point> imgPoints2List = new List<Point>();
122.  
123.             List<KeyPoint> keypoints1List = keyPoints1.toList();
124.             List<KeyPoint> keypoints2List = keyPoints2.toList();
125.  
126.             for (int i = 0; i < good_matches.Count; i++)
127.             {
128.                 imgPoints1List.Add(keypoints1List[good_matches[i].queryIdx].pt);
129.                 imgPoints2List.Add(keypoints2List[good_matches[i].trainIdx].pt);
130.             }
131.  
132.             MatOfPoint2f obj = new MatOfPoint2f();
133.             obj.fromList(imgPoints1List);
134.             MatOfPoint2f scene = new MatOfPoint2f();
135.             scene.fromList(imgPoints2List);
136.  
137.             Mat H = Calib3d.findHomography(obj, scene, Calib3d.RANSAC, 3);
138.  
139.             //size of the new image - i.e. image 1 + image 2
140.  
141.             Size s = new Size(src1.cols() + src2.cols(), src1.rows());
142.  
143.             Imgproc.warpPerspective(src1, result, H, s);
144.  
145.             Mat m = new Mat(result, new OpenCVForUnity.Rect(0, 0, src2.cols(), src2.rows()));
146.  
147.             src2.copyTo(m);
148.  
149.             //Features2d.drawMatches(src1, keyPoints1, src2, keyPoints2, goodMatches, result);
150.  
151.             return result;
152.  
153.         }
154.     }
155. }
156.  

 

Hi bf_glue.

You can change the resolution of WebCamTextue by setting the size value in "requestedWidth" and "requestedHeight" of WebCamTextureToMatHelper.

And you should also consider downscaling an image size by using the Imgproc.resize function before doing image processing.

 

Thanks for your reply!
Requested width and height already installed on 640x480
I'm trying to resize rgbMat

Code (CSharp):

1. Imgproc.resize(rgbaMat, rgbaMat, new Size (640, 480));

Code (CSharp):

1. void Update ()
2.         {          
3.             if (webCamTextureToMatHelper.IsPlaying () && webCamTextureToMatHelper.DidUpdateThisFrame ()) {
4.                
5.                 Mat rgbaMat = webCamTextureToMatHelper.GetMat ();
6.  
7.                 Imgproc.cvtColor (rgbaMat, rgbMat, Imgproc.COLOR_RGBA2RGB);
8.  
9.                 Imgproc.resize(rgbaMat, rgbaMat, new Size (640, 480));
10.  
11.                 // detect markers.
12.                 Aruco.detectMarkers (rgbMat, dictionary, corners, ids, detectorParams, rejectedCorners, camMatrix, distCoeffs);
13.  
14.                 // refine marker detection.
15.                 if (refineMarkerDetection && (markerType == MarkerType.GridBoard || markerType == MarkerType.ChArUcoBoard)) {
16.                     switch (markerType) {
17.                     case MarkerType.GridBoard:
18.                         Aruco.refineDetectedMarkers (rgbMat, gridBoard, corners, ids, rejectedCorners, camMatrix, distCoeffs, 10f, 3f, true, recoveredIdxs, detectorParams);
19.                         break;
20.                     case MarkerType.ChArUcoBoard:
21.                         Aruco.refineDetectedMarkers (rgbMat, charucoBoard, corners, ids, rejectedCorners, camMatrix, distCoeffs, 10f, 3f, true, recoveredIdxs, detectorParams);
22.                         break;
23.                     }
24.                 }
25.  
26.                 // if at least one marker detected
27.                 if (ids.total () > 0) {
28.                     if (markerType != MarkerType.ChArUcoDiamondMarker) {
29.  
30.                         if (markerType == MarkerType.ChArUcoBoard) {
31.                             Aruco.interpolateCornersCharuco (corners, ids, rgbMat, charucoBoard, charucoCorners, charucoIds, camMatrix, distCoeffs, charucoMinMarkers);
32.  
33.                             // draw markers.
34.                             Aruco.drawDetectedMarkers (rgbMat, corners, ids, new Scalar (0, 255, 0));
35.                             if (charucoIds.total () > 0) {
36.                                 Aruco.drawDetectedCornersCharuco (rgbMat, charucoCorners, charucoIds, new Scalar (0, 0, 255));
37.                             }
38.                         } else {
39.                             // draw markers.
40.                             Aruco.drawDetectedMarkers (rgbMat, corners, ids, new Scalar (0, 255, 0));
41.                         }
42.                            
43.                         // estimate pose.
44.                         if (applyEstimationPose) {
45.                             switch (markerType) {
46.                             default:
47.                             case MarkerType.CanonicalMarker:
48.                                 EstimatePoseCanonicalMarker (rgbMat);
49.                                 break;
50.                             case MarkerType.GridBoard:
51.                                 EstimatePoseGridBoard (rgbMat);
52.                                 break;
53.                             case MarkerType.ChArUcoBoard:
54.                                 EstimatePoseChArUcoBoard (rgbMat);
55.                                 break;
56.                             }
57.                         }
58.                     } else {
59.                         // detect diamond markers.
60.                         Aruco.detectCharucoDiamond (rgbMat, corners, ids, diamondSquareLength / diamondMarkerLength, diamondCorners, diamondIds, camMatrix, distCoeffs);
61.  
62.                         // draw markers.
63.                         Aruco.drawDetectedMarkers (rgbMat, corners, ids, new Scalar (0, 255, 0));
64.                         // draw diamond markers.
65.                         Aruco.drawDetectedDiamonds (rgbMat, diamondCorners, diamondIds, new Scalar (0, 0, 255));
66.  
67.                         // estimate pose.
68.                         if (applyEstimationPose)
69.                             EstimatePoseChArUcoDiamondMarker (rgbMat);
70.                     }
71.                 }
72.  
73.                 if (showRejectedCorners && rejectedCorners.Count > 0)
74.                     Aruco.drawDetectedMarkers (rgbMat, rejectedCorners, new Mat (), new Scalar (255, 0, 0));
75.                
76.                
77.                 //Imgproc.putText (rgbaMat, "W:" + rgbaMat.width () + " H:" + rgbaMat.height () + " SO:" + Screen.orientation, new Point (5, rgbaMat.rows () - 10), Core.FONT_HERSHEY_SIMPLEX, 1.0, new Scalar (255, 255, 255, 255), 2, Imgproc.LINE_AA, false);
78.  
79.                 Utils.fastMatToTexture2D (rgbMat, texture);
80.             }
81.         }

and have "ArgumentException: The Mat object has to be of the same size"

Please give an example of how to work with this. Will changing the resolution affect the FPS?

 

I don't have a concrete example code.
This type of image downscaling can also be easily executed by using ImageOptimizationHelper.
Please refer to PolygonFilterExample (Examples/Advanced/PolygonFilterExample) for usage.

 

Would I run deep learning model with this plugin like the link below?
https://docs.opencv.org/3.4.2/d2/d58/tutorial_table_of_content_dnn.html

 

Hey there,
I've been a customer of OpenCV for Unity for quite a while and today also purchased the Dlib example.

However, it does not work – I'm getting a NullReferenceException when running the ARHeadWebcamTextureExample:

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. OpenCVForUnity.Calib3d.solvePnP (OpenCVForUnity.MatOfPoint3f objectPoints, OpenCVForUnity.MatOfPoint2f imagePoints, OpenCVForUnity.Mat cameraMatrix, OpenCVForUnity.MatOfDouble distCoeffs, OpenCVForUnity.Mat rvec, OpenCVForUnity.Mat tvec, System.Boolean useExtrinsicGuess, System.Int32 flags) (at Assets/OpenCVForUnity/org/opencv/calib3d/Calib3d.cs:989)
3. DlibFaceLandmarkDetectorExample.ARHeadWebCamTextureExample.Update () (at Assets/DlibFaceLandmarkDetectorWithOpenCVExample/ARHeadExample/ARHeadWebCamTextureExample.cs:525)

Debugging only shows that objectPoints_mat and thus objectPoints are empty and thus the exception occurs. I'm running Unity 2018.2.5. Any ideas?

 

Since this package is a clone of OpenCV Java, you are able to use the same API as OpenCV Java 3.4.2. So, Halide backend and Custom deep learning layers are not supported.

 

Hello, Core.flip is now working (thank you for your earlier help). My problem is different from what I initially thought. On my android phone (ZTE ZMAX) the image is flipped 90 degrees clockwise when vertical. I thought I could get its transpose and then flip it in order to rotate the image 90 degrees in the correct rotation but I got the error in the image below (ArgumentException: the output Mat object has to be the same size).
This is my code:

Code (CSharp):

1. at temp = new Mat(rgbaMat.cols(), rgbaMat.rows(), CvType.CV_8UC4);
2.             Core.transpose(rgbaMat, temp);
3.             rgbaMat = temp;
4.             Core.flip(rgbaMat, rgbaMat, 1);

How should I go about rotating the image if this doesn't work?
Thank you.

 

As far as I can see the error you got, it seems that a wrong argument was passed to Utils.webCamTextrueToMat function. (webCamTexture and output Mat size are different)

 

Yea I kind of figured this was the problem since after the transpose rgba width and height are flipped but webcamTexture didn't. The problem is the webCamTexture can only have width and height supported by webcam, and (at least for my phone and computer) they don't support sizes where height is bigger than the width.

I did make my own transpose function that rotates only some of the pixels, but some of the pixels don't get rotated making it look bad (example below).

Is there another way to rotate the webCamTexture (or rgba matrix) 90 degrees so that I don't get this error?

 

WebCamTexture usually gets a landscape-sized image on a mobile platform because it does not transpose the image direction according to device orientation.
By using WebCamTextureToMatHelper, an image mat transposed in the correct direction can be acquired.
Usage example:
https://github.com/EnoxSoftware/Ope...xamples/Basic/WebCamTextureToMatHelperExample

 

Hello,
Trying to use WebCamTextureMarkerBasedARExample, e.g. just adding a human character model to the ARObjects. The character object is not visible in editor at runtime, and is not visible in an Android build.

If i add an additional cube to the existing ARObjects content, the new cube displays ok.

Are there limitations to the type of object, or complexity, that can be added to ARObjects ?

thanks

 

Have you set the layer of the new game object to "UI"?
If you want to render a game object by "ARCamera", you need to set the layer of the game object to "UI".

 

setting it to layer "AR" solved it. Thankyou Enox.

 

Hello!
I am working on a project in which we are trying to implement some CV Features through OpenCV for Unity at the moment. We need to apply an algorithm to recognize a dice (our problem is to find out how to count the dots), a problem which is usually very common and easy to solve if i'm right. In OpenCV there exists a class SimpleBlobDetector which fits our needs perfectly. However we cannot find something similar in OpenCV for Unity. Every other Featuredetector is availabe and i also found a forum post from some years ago where it seems this class also existed, but right now it seems if it does not. I guess we have to use one of the other featuredetectors and just create it with some parameters to get our desired functionality then, but since we are new in the field of computer vision we currently have no more idea how to approach it.

Is it possible to implement some rudimentary dice detection (it is enough to count the dots of always only one single dice) in Opencv for unity and if so, can someone give us an idea how to approach it?

Thank you for your help in advance

 

Hello, I'm doing some testing with MarkerLess AR Example ( https://assetstore.unity.com/packages/templates/tutorials/markerless-ar-example-77560 ) , but I'm having a lot of bug problems coming from the following scripting code ceiling WebCamTextureMarkerLessARExample.cs:

patternFound = patternDetector.findPattern (grayMat, patternTrackingInfo);

What is making the use of AR functionality extremely uncomfortable. Do you have any solutions to improve your performance?

 

Since this package is a clone of OpenCV Java, you are able to use the same API as OpenCV Java 3.4.2. So, SimpleBlobDetector class is not supported for now.
Also,
Since OpenCV is open source, it is possible to refer to SimpleBlobDetector's algorithm.
https://github.com/opencv/opencv/blob/master/modules/features2d/src/blobdetector.cpp

 

MarkerLessARExample Code is a rewrite of https://github.com/MasteringOpenCV/code/tree/master/Chapter3_MarkerlessAR using “OpenCV for Unity”.
Since this example is a tutorial code, I recommend using Vuforia etc. for more advanced functions.
I think that this page is helpful for hints on performance improvement.
https://github.com/MasteringOpenCV/code/issues/52

 

It's been 2 years since you originally mentioned in this thread that you did not plan on implementing SURF. Two questions:

- Has this plan (or lack of plan) changed since then, and have/will you reconsidered adding SURF?
- If you still have no intention to add SURF, can you expand on this choice?

A google search led me to this page, which seems to be class reference for a SURF class in your library but obviously the file "org/opencv_contrib/xfeatures2d/SURF.cs" is currently missing. https://enoxsoftware.github.io/Open...tml/class_open_c_v_for_unity_1_1_s_u_r_f.html What's that about?

 

Unfortunately, the SURF and SIFT algorithms have patent issues, so their classes are not planned to be added to OpenCVForUnity. I recommend that you consider using ORB or AKAZE instead of SIFT or SURF.

https://enoxsoftware.github.io/Open...tml/class_open_c_v_for_unity_1_1_s_u_r_f.html
This page is of an older version. Currently, this page is not linked.
https://enoxsoftware.github.io/Open.../class_open_c_v_for_unity_1_1_feature2_d.html

 

Hi Enox,

I am trying to calibrate my iPhone camera. I am planning on taking the photos and importing it to Unity. Is there any documentation or c# scripts I can follow to do this?

Thanks!!

 

Hello QAQAQAQAQAQ.

The photo capture of mobile phones is an area not included in the OpenCV feature.
If you want to use native feature APIs such as taking photos in Unity, you need to use useful assets such as NatCam.
https://assetstore.unity.com/packages/tools/integration/natcam-webcam-api-52154
https://medium.com/@olokobayusuf/natcam-tutorial-series-3-photos-e28361b83cf8
https://github.com/EnoxSoftware/NatCamWithOpenCVForUnityExample