[RELEASED] OpenCV for Unity

Currently I do not have an example of object detection from cameraPreview of MLCamera class.

 

Hello, when running the TensorFlowWebCamTextureExample with a custom .pb file created via Darkflow I receive the following error.

Any idea how I should resolve this?

dnn::forward_11() : OpenCV(4.0.0-dev) C:\Users\satoo\Desktop\opencv\modules\dnn\src\dnn.cpp:413: error: (-2:Unspecified error) Can't create layer "init" of type "NoOp" in function 'cv::dnn::dnn4_v20180917::LayerData::getLayerInstance'

UnityEngine.Debug:LogError(Object)

OpenCVForUnity.UnityUtils.Utils:debugLogFunc(String) (at Assets/OpenCVForUnity/org/opencv/unity/Utils.cs:1200)

OpenCVForUnity.DnnModule.Net:dnn_Net_forward_11()

OpenCVForUnity.DnnModule.Net:forward() (at Assets/OpenCVForUnity/org/opencv/dnn/Net.cs:96)

OpenCVForUnityExample.TensorFlowWebCamTextureExample:Update() (at Assets/OpenCVForUnity/Examples/MainModules/dnn/TensorFlowExample/TensorFlowWebCamTextureExample.cs:228)

 

I appreciate the response, and determining a PNP problem solution seems possible if I have 4 points. I noticed that the HoloLensARHeadExample code requires that you can determine the focal length of the camera. I saw that you use a camera intrinsics checker for the HoloLens, referenced in your script at this link: https://github.com/EnoxSoftware/Hol...nsicsChecker/CameraIntrinsicsCheckerHelper.cs. Looking over the Magic Leap documentation, I can't seem to find anything regarding the focal length or information about the camera. In fact, I only found a forum post describing that they do not provide the intrinsic camera details here: https://forum.magicleap.com/hc/en-us/community/posts/360033081772/comments/360003340051

Would you know if there would be a way I could calculate the focal length and principal points myself?
Thanks, you've been very helpful so far, and I appreciate your input.

 

This is usually done through camera calibration. What you are looking for is the intrinsic properties of the camera. Here are some OpenCV docs you can use to get started: https://docs.opencv.org/3.4.3/dc/dbb/tutorial_py_calibration.html

//C

 

I appreciate the extra information. I attempted to implement a test such as the given example to get the intrinsic information, but to no avail. Considering the project requirements, I decided to move on, but it was worth the try. If only Magic Leap provided more information in their documentation, I'd have less issues working with their device... Oh well.

 

Dear EnexSoftware,

I am working on a project which involves the camera pose estimation.
Now I am trying to translate the Python code into C#. However, with the same data I am getting different results: more realistics in Python and totally different with C# and OpenCV for Unity.
The function I am using is:

Calib3d.solvePnPRansac(ObjectPoints, ImagePoints, CameraMatrix, Distortion, rvec, tvec);

 

About the solvePnP problem:

the strange thing is that it returns a bool, which should tell if the computation was successfull or not, and it is False.
However, the vectors rvec and tvec have been modified and their content is always [0,0,0] (or numbers close to it).
What should I check?

 

Hi,

I was thinking of buying this but was wondering if there was an example of tracking a color object in 3d space? I would really like to know if there is a way to track depth and transfer that to a game object inside of unity.

 

Thank you very much for reporting.
Could you send me the code you tried? https://enoxsoftware.com/opencvforunity/contact/technical-inquiry/

 

I think MakerLessARExample is useful. This example is a Marker-Less Augmented Reality Example using the actual environment as input instead of printed square markers and display 3d model in WebCamTexture in real-time.
https://assetstore.unity.com/packages/templates/tutorials/markerless-ar-example-77560

 

1. Can I use a custom image as a marker with Open CV, such as a photo of a complex painting?

Is there a tutorial somewhere?

 

"MarkerBased AR Example" is the Augmented Reality Example that detect and recognize only N x N block markers.
but, "MarkerLess AR Example" is the Augmented Reality Example that detect and recognize natural 2d images ( a photo of a complex painting ).

 

Hi I have a question. (kind of bug report..?)


This is very simple Mat operation but It makes a crash. (crash like Unity shuts down all of the sudden without any warning or message. )
I think it occurs because 'Mat b' has channels more than 4.

Is this normal? or a bug?

My Unity version is 2018.3.7f1 and I downloaded this asset (your unity opencv) just two days before.

 

Thank you very much for reporting.
Since this asset is a clone of OpenCV Java, you are able to use the same API as OpenCV Java.http://enoxsoftware.github.io/OpenCVForUnity/3.0.0/doc/html/index.html
The Mat (int rows, int cols, int type, Scalar s) constructor seems to support only the initialization of Mat with 4 or less channels. If you want to initialize Mat of 5 channels or more, you can initialize the value of Mat as follows.

Code (CSharp):

1.             Mat b = new Mat (1, 3, CvType.CV_32FC (10));
2.  
3. //            CV_8U and CV_8S -> byte[],
4. //            CV_16U and CV_16S -> short[],
5. //            CV_32S -> int[],
6. //            CV_32F -> float[],
7. //            CV_64F-> double[].
8.             b.put (0, 0, 1.0f, 2.0f, 3.0f, 4.0f, 5.0f, 6.0f, 7.0f, 8.0f, 9.0f, 10.0f);
9.             b.put (0, 1, 11.0f, 12.0f, 13.0f, 14.0f, 15.0f, 16.0f, 17.0f, 18.0f, 19.0f, 20.0f);
10.             b.put (0, 2, 21.0f, 22.0f, 23.0f, 24.0f, 25.0f, 26.0f, 27.0f, 28.0f, 29.0f, 30.0f);
11.  
12.             Debug.Log (b);
13.             Debug.Log ("b.dump() " + b.dump ());

 

Oh.. Thank you for your fast, kind reply
Can I have another question?

Now I see I can put my data into Mat manually.
But Unity shuts down If I try operations with my Mats(46 x 46 x 21). Including Mat and Mat operations, Mat and primitive type(float, int) operations occur shut-down too.
Is this intended?

What would be the best way if I want to divide my Mat(46 x 46 x 21) by 2?

Thank you.

 

My custom object detection (tensorflow) work smoothly with python but when i export to unity it run very slow because it didn't use gpu anymore. Do you know how to fix this issue?

 

Your Mats(46 x 46 x 21) is 3 Dimensional Mat ?
The following page will be helpful for the operation of 3 Dimensional Mat.
http://answers.opencv.org/question/32358/how-to-access-3d-histogram-data-opencv-java/

Code (CSharp):

1.             //create 3D Mat
2.             Mat sourceMat = Imgcodecs.imread (Utils.getFilePath ("pca_test1.jpg"));      
3.             List<Mat> sourceImages = new List<Mat> ();        
4.             sourceImages.Add (sourceMat);      
5.             Mat histogram = new Mat ();            
6.             MatOfInt histSizes = new MatOfInt (10, 10, 10);      
7.             MatOfFloat histRanges = new MatOfFloat (0, 256, 0, 256, 0, 256);
8.             MatOfInt channels = new MatOfInt (0, 1, 2);
9.             Imgproc.calcHist (sourceImages, channels, new Mat (), histogram, histSizes, histRanges);
10.  
11.             Debug.Log (histogram.ToString ());
12.             Debug.Log ("size(0) " + histogram.size (0));
13.             Debug.Log ("size(1) " + histogram.size (1));
14.             Debug.Log ("size(2) " + histogram.size (2));
15.             Debug.Log ("dims() " + histogram.dims ());
16.             //            Debug.Log ("b.dump() " + histogram.dump ());
17.             Debug.Log ("histogram.type() " + histogram.type ());
18.             Debug.Log ("histogram.total() " + histogram.total ());
19.  
20.             //get Mat data
21.             float[] data = new float[10 * 10 * 10];
22.             Utils.copyFromMat<float> (histogram, data);
23.             Debug.Log ("data[0] " + data [0]);
24.  
25.  
26.             //devide Mat by 2
27.             Mat result = new Mat ();
28.             Core.divide (histogram, new Scalar (2, 2, 2), result);
29.  
30.             //get devided Mat data
31.             Utils.copyFromMat<float> (result, data);
32.             Debug.Log ("data[0] " + data [0]);

 

Unfortunately, the OpenCVforUnity dnn module does not support GPU ( OpenCL backend ).

CPU Performance Comparison of OpenCV and other Deep Learning frameworks
https://www.learnopencv.com/cpu-performance-comparison-of-opencv-and-other-deep-learning-frameworks/

 

Thank you very much.
I think I can use Core.divide

 

Could you consider developing your MagicLeapWithOpenCVForUnityExample a bit further (real time)?

 

I plan to update MagicLeapWithOpenCVForUnityExample after Unity2019 ( support for MagicLeap platform ) is officially released.

 

On UWP - Getting the error, "Cascade file is not loaded. Please copy from "FaceTrackerExample/StreamingAssets/" to "Assets/StreamingAssets/" folder.

The issue is there is no "FaceTrackerExample/StreamingAssets/". How do I fix this?

It does work in editor though. Very well.

 

Can you tell me the version of the asset you tested?
Unity version :
OpenCVforUnity version :
FaceTrackerExample version :

 

@EnoxSoftware But how do I draw a Cube on top of the nose. Can anyone help me

 

Hi @EnoxSoftware,
The demo you have hosted here, does not import into unity.
It says that there is nothing to import.
This happens even in a new empty project.
Please provide help in this matter.
I am using 2018.3.5, if it helps.

Thanking you in anticipation of the help.

Regards,
Hitarth

 

We're working on using the Intel DL IE in combination with your OpenCV for Unity asset to allow us to offload the model inferences to Intel Movidius NCS2. Is this something that could be officially supported? Do you anticipate any issues?

 

@EnoxSoftware,
Thanks for the fast reply!

Unity Version: 2018.3.6f1
OpenCVforUnity version: 2.3.4
FaceTrackerExample version: 1.1.9

 

In my environment, the files are imported without problems.
OpenCVForUnity2.3.4TrialVersion.unitypackage
Unity2018.3.5f1
Windows10

 

Unfortunately, OpenCVforUnity does not support Intel's Deep Learning Inference Engine backend.

 

The cascade.empty () method does not seem to work properly on the UWP platform. This issue is under investigation.

Code (CSharp):

1.             CascadeClassifier cascade = new CascadeClassifier ();
2.             cascade.load (haarcascade_frontalface_alt_xml_filepath);
3. //            if (cascade.empty())
4. //            {
5. //                Debug.LogError("cascade file is not loaded.Please copy from “FaceTrackerExample/StreamingAssets/” to “Assets/StreamingAssets/” folder. ");
6. //            }

 

@EnoxSoftware can you please reply on this.

 

@EnoxSoftware, thanks for letting us know! Please, if you can fix it, we would love the fix. We are using this for a big project. Do you think there is an alternative route to face detection.


@Rajesh_Holopundits you can try this example!
https://assetstore.unity.com/packages/templates/tutorials/facemask-example-79999

 

Hi @EnoxSoftware ,

I am trying load one of the Intel pre-trained model using:

public string model_file_path = "Model_file_path.xml";
public string config_file_path = "Config_file_path.bin";
Net net;
net = Dnn.readNetFromModelOptimizer(model_file_path, config_file_path);

However, Unity crashes without giving me any clue. Can you show me how can I load the model.

Thanks in advance for the answer!
Best Regards,
Emre

 

Trying to run the mask rnn example from a fresh import.

The link to the .pb file does not work. Substituted with frozen_inference_graph.pb

Also, received these errors:

Code (CSharp):

1. NullReferenceException: Object reference not set to an instance of an object
2. OpenCVForUnityExample.MaskRCNNExample.Run () (at Assets/OpenCVForUnity/Examples/MainModules/dnn/TensorFlowExample/MaskRCNNExample.cs:143)
3. OpenCVForUnityExample.MaskRCNNExample.Start () (at Assets/OpenCVForUnity/Examples/MainModules/dnn/TensorFlowExample/MaskRCNNExample.cs:91)

 

I think ARHeadExample is useful for you.
Could you place Cube on the child of "ARObject"?

 

Could you rename frozen_inference_graph.pb to mask_rcnn_inception_v2_coco_2018_01_28.pb?

 

Unfortunately, OpenCVforUnity does not support Intel's Deep Learning Inference Engine backend.
https://docs.opencv.org/4.0.0/d6/d0f/group__dnn.html#ga4f3b552113d2bff48a54e168791c448e
Returns
Net object. Networks imported from Intel's Model Optimizer are launched in Intel's Inference Engine backend.

 

@EnoxSoftware,
Do you think there is a way to avoid/bypass the cascade file?

 

HI, @EnoxSoftware ,
I meet problem at first step with Unity 2019.1.0f2 and OpenCV for Unity 2.3.4.

I can't understand real solution that change your code or change compile option.

This trouble is just simple, that I get "All Compiler error hove to be fix before enter playmode!".
I found following errors on console.
".... error CS1644: Feature 'out variable declaration' cannot be used because it is not part of the C# 4.0 langue specification."

This mean as following that
error on fastAction.cs line 24

if (lookup.TryGetValue(rhs, out LinkedListNode<System.Action> node))

I can remove this error as following that

LinkedListNode<System.Action> node;
if (lookup.TryGetValue(rhs, outnode))

But is this solution of this problem or I missing some compile option with Unity ?

 

Instead of the cascade.empty () method, you can use the result of cascade.load () method to determine if the cascade file load was successful.

Code (CSharp):

1.             CascadeClassifier cascade = new CascadeClassifier ();
2. //          cascade.load (haarcascade_frontalface_alt_xml_filepath);
3.             if (!cascade.load (haarcascade_frontalface_alt_xml_filepath))
4.             {
5.                   Debug.LogError("cascade file is not loaded.Please copy from “FaceTrackerExample/StreamingAssets/” to “Assets/StreamingAssets/” folder. ");
6.             }

 

@EnoxSoftware,

Thank you. This fixed the error.

 

Does the error occur when importing only OpenCVforUnity into an empty project?

 

Yes, can you update the directions?

 

Any advice for Converting OpenCVForUnity Rect to Unity World Point?

 

Unfortunately, I do not know the simple way to convert from OpenCV's point to Unity's point.

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using OpenCVForUnity.CoreModule;
4. using OpenCVForUnity.UnityUtils;
5. using OpenCVForUnity.ObjdetectModule;
6. using OpenCVForUnity.ImgprocModule;
7.  
8. using UnityEngine.SceneManagement;
9.  
10.  
11. namespace OpenCVForUnitySample
12. {
13.     /// <summary>
14.     /// DetectFace sample.
15.     /// </summary>
16.     public class DetectFace2DTo3DExample : MonoBehaviour
17.     {
18.  
19.         public GameObject point3D;
20.  
21.         // Use this for initialization
22.         void Start ()
23.         {
24.    
25.             Texture2D imgTexture = Resources.Load ("lena") as Texture2D;
26.            
27.             Mat imgMat = new Mat (imgTexture.height, imgTexture.width, CvType.CV_8UC4);
28.            
29.             Utils.texture2DToMat (imgTexture, imgMat);
30.             Debug.Log ("imgMat dst ToString " + imgMat.ToString ());
31.  
32.  
33.             //CascadeClassifier cascade = new CascadeClassifier (Utils.getFilePath ("lbpcascade_frontalface.xml"));
34.             CascadeClassifier cascade = new CascadeClassifier (Utils.getFilePath ("haarcascade_frontalface_alt.xml"));
35. //                        if (cascade.empty ()) {
36. //                                Debug.LogError ("cascade file is not loaded.Please copy from “OpenCVForUnity/StreamingAssets/” to “Assets/StreamingAssets/” folder. ");
37. //                        }
38.  
39.             Mat grayMat = new Mat ();
40.             Imgproc.cvtColor (imgMat, grayMat, Imgproc.COLOR_RGBA2GRAY);
41.             Imgproc.equalizeHist (grayMat, grayMat);
42.  
43.  
44.             MatOfRect faces = new MatOfRect ();
45.        
46.             if (cascade != null)
47.                 cascade.detectMultiScale (grayMat, faces, 1.1, 2, 2,
48.                                            new Size (20, 20), new Size ());
49.  
50.             OpenCVForUnity.CoreModule.Rect[] rects = faces.toArray ();
51.             for (int i = 0; i < rects.Length; i++) {
52.                 Debug.Log ("detect faces " + rects [i]);
53.  
54.                 Imgproc.rectangle (imgMat, new Point (rects [i].x, rects [i].y), new Point (rects [i].x + rects [i].width, rects [i].y + rects [i].height), new Scalar (255, 0, 0, 255), 2);
55.  
56.  
57.  
58.             }
59.  
60.  
61.  
62.             Texture2D texture = new Texture2D (imgMat.cols (), imgMat.rows (), TextureFormat.RGBA32, false);
63.  
64.             Utils.matToTexture2D (imgMat, texture);
65.  
66.             gameObject.GetComponent<Renderer> ().material.mainTexture = texture;
67.    
68.  
69.  
70.  
71.             Matrix4x4 point3DMatrix4x4 = Get2DTo3DMatrix4x4 (new Point (rects [0].x + rects [0].width / 2, rects [0].y + rects [0].height / 2), gameObject);
72.            
73.             Vector3 point3DVec = new Vector3 (0, 0, 0);
74.             point3DVec = point3DMatrix4x4.MultiplyPoint3x4 (point3DVec);
75.            
76.             point3D.transform.position = point3DVec;
77.             point3D.transform.eulerAngles = gameObject.transform.eulerAngles;
78.         }
79.        
80.         // Update is called once per frame
81.         void Update ()
82.         {
83.  
84.         }
85.  
86.         public void OnBackButton ()
87.         {
88.            
89.             SceneManager.LoadScene ("OpenCVForUnitySample");
90.          
91.         }
92.  
93.         /// <summary>
94.         /// Get2s the D to3 D matrix4x4.
95.         /// </summary>
96.         /// <returns>The D to3 D matrix4x4.</returns>
97.         /// <param name="point2D">Point2 d.</param>
98.         /// <param name="quad">Quad.</param>
99.         private Matrix4x4 Get2DTo3DMatrix4x4 (Point point2D, GameObject quad)
100.         {
101.             float textureWidth = GetComponent<Renderer> ().material.mainTexture.width;
102.             float textureHeight = GetComponent<Renderer> ().material.mainTexture.height;
103.            
104.             Matrix4x4 transCenterM =
105.                 Matrix4x4.TRS (new Vector3 (((float)point2D.x) - textureWidth / 2, (textureHeight - (float)point2D.y) - textureHeight / 2, 0), Quaternion.identity, new Vector3 (1, 1, 1));
106.            
107.            
108.             Vector3 translation = new Vector3 (quad.transform.localPosition.x, quad.transform.localPosition.y, quad.transform.localPosition.z);
109.            
110.             Quaternion rotation =
111.                 Quaternion.Euler (quad.transform.localEulerAngles.x, quad.transform.localEulerAngles.y, quad.transform.localEulerAngles.z);
112.            
113.             Vector3 scale = new Vector3 (quad.transform.localScale.x / textureWidth, quad.transform.localScale.y / textureHeight, 1);
114.            
115.             Matrix4x4 trans2Dto3DM =
116.                 Matrix4x4.TRS (translation, rotation, scale);
117.  
118.             Matrix4x4 resultM = trans2Dto3DM * transCenterM;
119.  
120.             return resultM;
121.         }
122.     }
123. }

 

@EnoxSoftware,

Does this plugin also have tools to rotate images or mats?

 

You can rotate Mat by using the code below.
https://github.com/EnoxSoftware/Ope.../Feature2DExample/Feature2DExample.cs#L21-L38

 

Hi @EnoxSoftware (and for the googlers),

A point of feedback: currently when directly instantiating a Mat on declaration as a member, a built Unity app crashes.

Code (CSharp):

1. class Analyser {
2.     private Mat input = new Mat (analysisHeight, analysisWidth, CvType.CV_8UC4);

Note, this only happens in standalone mode, in the editor there is no issue with this code.

The crash happens immediately when the app launches, and does not even result in a crash report. The output_log.txt file does contain a stack trace including

ERROR: SymGetSymFromAddr64, GetLastError: 'The specified module could not be found.' (Address: 00000209E66103A3)

ERROR: SymGetModuleInfo64, GetLastError: 'A dynamic link library (DLL) initialization routine failed.' (Address: 00000209E66103A3)

So it looks like a dll is not yet initialized at that time. The fix was to initialize all Mats later, for example in the Start method:

Code (CSharp):

1. class Analyser {
2.     private Mat input;
3.  
4.     void Start() {
5.         input = new Mat (analysisHeight, analysisWidth, CvType.CV_8UC4);

As you can imagine it took some time to figure this one out, so it might be nice if when instantiating a Mat, you could do a check if the dll is loaded and post a warning if not, rather than letting the app crash.

 

Thank you very much for reporting.
It seems that crash does not occur when "Load On startup" is enabled.

 

Ah, yes, that makes sense. Thanks for the tip!