[RELEASED] OpenCV for Unity

BackgroundSubtractorMOG2Example is included in OpenCVForUnity.
https://github.com/EnoxSoftware/Ope...OG2Example/BackgroundSubtractorMOG2Example.cs

 

did you solved your problem?
i have the same error

 

@EnoxSoftware I am trying the Openpose example however the script is not able to read .prototxt file, and I get the error. model file is not loaded. The model and prototxt file can be downloaded here .....

Secondly! would it be possible to use the model on VideoTexture on Mobile Devices, I am worried about the fps.
please guide me in this regard.
Thanks

 

Have you downloaded prototxt file as a raw text file? In my environment, the file size of openpose_pose_mpi_faster_4_stages.prototxt is 32,813 bytes.
https://raw.githubusercontent.com/opencv/opencv_extra/master/testdata/dnn/ope npose_pose_mpi_faster_4_stages.prototxt

for now, Since OpenCV's Dnn module uses a CPU, OpenPoseExample takes more than 900 ms to estimate human pose. Perhaps real-time processing is difficult.
But, this repository using OpenCVForUnity seems to work in real-time.
https://twitter.com/yukihiko_a/status/1131174274708910080
https://github.com/yukihiko/ThreeDPoseUnityForiOS

 

@EnoxSoftware i have sent a technical inquiry regarding the line segment detector (LSD) module of imgproc, but i'll just ask here as well. we are attempting to recreate this (http://www.kellymcclinton.com/automatically-extracting-floor-plans) on a vuforia(AR)-based android application. we are trying to apply the LSD onto the results of a probabilistic hough line transform pass on a floor plan. currently, the LSD module is not producing any results. attached is our LSD.cs class. thank you in advance!

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using UnityEngine.SceneManagement;
4. using OpenCVForUnity;
5.  
6. namespace OpenCVForUnityExample
7. {
8.     public class LSD : MonoBehaviour
9.     {
10.         Mat grayMat, lines, imgMat;
11.         public Texture2D image;
12.         public void Start ()
13.         {
14.             imgMat = new Mat (image.height, image.width, CvType.CV_8UC4);
15.             Utils.texture2DToMat (image, imgMat);
16.             grayMat = new Mat (image.height, image.width, CvType.CV_8UC1);
17.             Imgproc.cvtColor (imgMat, grayMat, Imgproc.COLOR_RGB2GRAY);
18.             Imgproc.Canny (grayMat, grayMat, 50, 200);
19.             lines = new Mat (image.height, image.width, CvType.CV_8UC4);
20.  
21.             LineSegmentDetector lsd = Imgproc.createLineSegmentDetector();
22.             lsd.detect(grayMat, lines);
23.             lsd.drawSegments(imgMat, lines);
24.  
25.             Texture2D texture = new Texture2D (imgMat.cols (), imgMat.rows (), TextureFormat.RGBA32, false);
26.             Utils.matToTexture2D (imgMat, texture);
27.             gameObject.GetComponent<Renderer> ().material.mainTexture = texture;
28.         }
29.     }
30. }

 

Yes, I downloaded the file as a raw and I have the same size but I couldn't make it run. 900 ms are toom much.
if I train my own mobilenet model with tensorflow lite, would it be possible to run it with opencvforunity package?

 

LineSegmentDetector Implementation has been removed due original code license issues.
You can use FastLineDetector instead.
https://github.com/opencv/opencv/issues/14576
https://github.com/opencv/opencv/co...6dae6b6#diff-ad263766480c395858af1137a84226ef

Code (CSharp):

1. using OpenCVForUnity.CoreModule;
2. using OpenCVForUnity.ImgprocModule;
3. using OpenCVForUnity.UnityUtils;
4. using OpenCVForUnity.XimgprocModule;
5. using UnityEngine;
6.  
7. namespace OpenCVForUnityExample
8. {
9.     public class FLD : MonoBehaviour
10.     {
11.         Mat grayMat, lines, imgMat;
12.         public Texture2D image;
13.         public void Start()
14.         {
15.             Utils.setDebugMode(true);
16.  
17.             imgMat = new Mat(image.height, image.width, CvType.CV_8UC3);
18.             Utils.texture2DToMat(image, imgMat);
19.             grayMat = new Mat(image.height, image.width, CvType.CV_8UC1);
20.             Imgproc.cvtColor(imgMat, grayMat, Imgproc.COLOR_RGB2GRAY);
21.             Imgproc.Canny(grayMat, grayMat, 50, 200);
22.             lines = new Mat(image.height, image.width, CvType.CV_8UC4);
23.  
24.  
25.             FastLineDetector fld = Ximgproc.createFastLineDetector();
26.             fld.detect(grayMat, lines);
27.             fld.drawSegments(imgMat, lines);
28.  
29.             Texture2D texture = new Texture2D(imgMat.cols(), imgMat.rows(), TextureFormat.RGBA32, false);
30.             Utils.matToTexture2D(imgMat, texture);
31.             gameObject.GetComponent<Renderer>().material.mainTexture = texture;
32.  
33.             Utils.setDebugMode(false);
34.         }
35.     }
36. }

 

@EnoxSoftware Hello, we sent a Technical Inquiry through the enoxsoftware website but thought it would be better to reach you here.

We have an issue when building our project or even the OpenCV Examples for iOS.

Our environment is:
Unity Version: 2019.3.0a3
OpenCVforUnity version: 2.3.5

The build in Xcode fails with the following error message:

2CD173127361/OurApp.app/Frameworks/UnityFramework.framework/UnityFramework: dlopen(/var/containers/Bundle/Application/2BF88547-9B4E-4C35-9BEB-
2CD173127361/OurApp.app/Frameworks/UnityFramework.framework/UnityFramework, 265): Library not loaded: @rpath/opencv2.framework/opencv2
Referenced from: /var/containers/Bundle/Application/2BF88547-9B4E-4C35-9BEB-2CD173127361/OurApp.app/Frameworks/UnityFramework.framework/UnityFramework
Reason: image not found

It looks like the @rpath does not contain the right path for the opencv2 shared-lib.

We recently updated from OpenCVforUnity version:2.2.8 which was building ok with the same Unity version we are using (2019.3.0a3).

Would you have any clue about this problem?
Thanks

 

Are downloaded model files placed in the “Assets / StreamingAssets / dnn /” folder?

 

Hi EnoxSoftware, thanks for your AlphaBlendingExample first , it almost solve my question.
However, there is some question which I am facing.
From the picture, I think there is only a few part which I have to blend the alpha
so I think I need to customize a alpha Mat for comparing picture which white area with same pixel and black are with different pixel, but I am fail to do that since there is two Mat with different size
It make a wrong comparing though my following code.
So may I ask how to solve this case? Thanks.

Code (CSharp):

1. byte[] fg_byte = new byte[first_mat.total() * first_mat.channels()];
2.         Utils.copyFromMat<byte>(first_mat, fg_byte);
3.         byte[] bg_byte = new byte[warp_mat.total() * warp_mat.channels()];
4.         Utils.copyFromMat<byte>(warp_mat, bg_byte);
5.         byte[] alpha_byte = new byte[alphaMat.total() * alphaMat.channels()];
6.         Utils.copyFromMat<byte>(alphaMat, alpha_byte);
7.         int pixel_i1 = 0;
8.         int pixel_i2 = 0;
9.         double alpha = 255;
10.         int channels = (int)first_mat.channels();
11.         int total = (int)warp_mat.total();
12.         for (int i = 0; i < total; i++)
13.         {
14.             if (bg_byte[pixel_i1] == fg_byte[pixel_i2] && bg_byte[pixel_i1 + 1] == fg_byte[pixel_i2 + 1] && bg_byte[pixel_i1 + 2] == fg_byte[pixel_i2 + 2])
15.             {
16.                 if (i < first_mat.total() - 1)
17.                 {
18.                     alpha = 255;
19.                 }
20.                 else
21.                 {
22.                     alpha = 133;
23.                 }
24.             }
25.             else
26.             {
27.                 alpha = 0;
28.             }
29.  
30.             alpha_byte[i] = (byte)alpha;
31.             pixel_i1 += channels;
32.             if (i < first_mat.total() - 1)
33.                 pixel_i2 += channels;
34.         }
35.         Utils.copyToMat(alpha_byte, alphaMat);

 

It seems to have been changed to use UnityFramework.framework from Unity2019.3. Unity 2011.3 is still in alpha, so there may be bugs remaining in the framework's path settings.

In my environment, I succeeded in building with the following settings.
1. Uncheck "Add to Embedded Binaries".

2. Add "opencv2.framework" to "Embedded Binaries".

 

Hello,

Any update on this fix? We are using 2018.4.3 and we are getting the same error.

 

I have an Android app that uses this asset, I'm doing a bunch of conversions and line detection and edge detection. The app runs fine for 20-30 seconds, and then the app crashes.

Whats the best way to investigate the cause?
Thanks.

 

just curious if super resolution and other modules might be included. i noticed that some modules are included as Unity C# and others are not - curious why you picked those?

 

If I think of the simple way, it is necessary to match the size of both images, such as by temporarily copying the area of them to the same size image Mat.

 

Thank you very much for reporting.
This error seems to occur in the latest version of Unity2018. This error will be fixed in the next version of OpenCVForUnity.
For now, please use Unity2019.1 or later.

 

Hi ! finaly got Yolo examples to work .. question how can I generate my own training models to replace the ones from github? any links or tutorials?

Thanks.

 

Sadly, our project has a lot of issues in 2019.1 and later. Is there any workaround for solving this in 2018? If not, could you please let us know an ETA on the next version?

Best,
Max

 

A memory leak may have occurred. Could you check if the used Mat is Dispose？
void Update ()
{
Mat mat = new Mat();
-------------------------
-------------------------
mat.Dispose();
}


Also,
In order to display the native opencv's error code, please enclose the code in Utils.setDebugMode(true) and Utils.setDebugMode(false).
Errors are displayed in the console?
Utils.setDebugMode(true);
------------
Utils.setDebugMode(false);

 

Since this package is a clone of OpenCV Java, you are able to use the same API as OpenCV Java 4.1.0.

 

Could you tell me the Invoice number? https://enoxsoftware.com/opencvforunity/contact/other-inquiry/
I will send you the download link for the modified "OpenCVForUnity/Plugins/WebGL" folder.

 

Sent! Thanks.

 

Thanks for your help, this solved our issue

 

Simply trying to play the DemoScenes. Unity 2019.1.10f1 on Mac 10.14.5
Every scene except Robot following Line and Thresholds crashes when I try to play in the editor.

 

Hi @EnoxSoftware
I am following this github repo to make the hand tracking app using OpenCV and TensorFlow.
But the unity editor crashes.
https://github.com/victordibia/handtracking

Need your immediate help.
This project is in tight deadline.

 

I think the following post will be helpful.

The error was caused by the wrong input .pbtxt file passed into the function readNetFromTensorflow because the .pbtxt has to be geneated by tf_text_graph_ssd.py as describe here:​

https://stackoverflow.com/questions...t-work-with-opencv-after-retraining-mobilenet
https://stackoverflow.com/questions...n-failed-in-getmemoryshapes?noredirect=1&lq=1
https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API

 

I can't upload my app to iTunes Connect:

App Store Connect Operation Error
ERROR ITMS-90087: "Unsupported Architectures. The executable for doodlelens.app/Frameworks/opencv2.framework contains unsupported architectures '[x86_64, i386]'."

App Store Connect Operation Error
ERROR ITMS-90209: "Invalid Segment Alignment. The app binary at 'doodlelens.app/Frameworks/opencv2.framework/opencv2' does not have proper segment alignment. Try rebuilding the app with the latest Xcode version."

App Store Connect Operation Error
ERROR ITMS-90125: "The binary is invalid. The encryption info in the LC_ENCRYPTION_INFO load command is either missing or invalid, or the binary is already encrypted. This binary does not seem to have been built with Apple's linker."

App Store Connect Operation Warning
WARNING ITMS-90080: "The executable 'Payload/doodlelens.app/Frameworks/opencv2.framework' is not a Position Independent Executable. Please ensure that your build settings are configured to create PIE executables. For more information refer to Technical Q&A QA1788 - Building a Position Independent Executable in the iOS Developer Library."

 

Thank you very much for reporting.
Could you tell me about your test environment?
OpenCVForUnity version :
Unity version :
Xcode version :

 

Hi,

I'm building an interactive table like the picture attached , and am hoping to use a realsense 415 with OpenCVforUnity to work out where people are touching the projection.

I've got two main issues

1 - When I try the examples in your asset with the realsense I get a "could not connect pins - renderstream()" error. The examples work fine however with a regular webcam. Any idea what the fix might be ?

2 - Am i going to be able to tell the difference between the table (3 m from the projector) and the hands above it. Is there a specific example in the unityopenCV asset that will help with this detection ?

OpenCVForUnity version : 2.3.5
Unity version : 2017.4.17
Windows 10

 

Hi again - I'm at the point where I'm using this functionality, but want to access those blob points - to map things in the GUI to their position. How can I access that data please ?

Code (CSharp):

1.  Mat outImgMat = new Mat();
2.  
3.             SimpleBlobDetector blobDetector = SimpleBlobDetector.create();
4.          
5.             blobDetector.read(blobparams_yml_filepath);
6.          
7.             MatOfKeyPoint keypoints = new MatOfKeyPoint();
8.             blobDetector.detect(imgMat, keypoints);
9.             Features2d.drawKeypoints(imgMat, keypoints, outImgMat);
10.          
11.             Texture2D texture = new Texture2D(outImgMat.cols(), outImgMat.rows(), TextureFormat.RGBA32, false);
12.  
13.             Utils.matToTexture2D(outImgMat, texture);

 

Unfortunately I have not tried the realsense camera.
1 - Is this post related to "could not connect pins - renderstream()" error?
https://community.nuitrack.com/t/un...pins-renderstream-to-intel-realsense-d435/644

2 - An example of hand detection is included with OpenCVForUnity. However, I do not have multiple hand detection example.
https://github.com/EnoxSoftware/Ope...y/Examples/Advanced/HandPoseEstimationExample

 

thanks for the reply - the 'could not connect pins' issue stopped occuring when I used a different PC ! - the hand-tracking one doesn't work for me as I'm looking at tracking 4+ hands from above at about 2m and the users won't be able to click to focus

My issue now is getting openCV to be able to ignore the objects that are closer than where the table surface/hands interaction will be happening (so heads/shoulders don't trigger things)

so on this image - the dark tripod is one i'd like to ignore (it's where the head/shoulder would be) and the lighter one (which is roughly where the hands meet the table surface ) it the one i was to blob detect.

Currently I'm getting the opposite :O

 

Hi I have downloaded the OpenCVForUnity2.3.5TrialVersion.What I am trying to do is trying to detect text from physical objects.To read engraved text from steel objects by using an IOS device's(Iphone or ipad) camera.I want to try in IOS but when I take build many errors are coming.

Code (CSharp):

1. ArgumentException: The Assembly UnityEditor is referenced by OpenCVForUnityCSharpDLL ('Assets/OpenCVForUnity/org/OpenCVForUnityCSharpDLL.dll'). But the dll is not allowed to be included or could not be found.
2. UnityEditor.AssemblyHelper.AddReferencedAssembliesRecurse (System.String assemblyPath, System.Collections.Generic.List`1[T] alreadyFoundAssemblies, System.String[] allAssemblyPaths, System.String[] foldersToSearch, System.Collections.Generic.Dictionary`2[TKey,TValue] cache, UnityEditor.BuildTarget target) (at /Users/builduser/buildslave/unity/build/Editor/Mono/AssemblyHelper.cs:154)
3. UnityEditor.AssemblyHelper.AddReferencedAssembliesRecurse (System.String assemblyPath, System.Collections.Generic.List`1[T] alreadyFoundAssemblies, System.String[] allAssemblyPaths, System.String[] foldersToSearch, System.Collections.Generic.Dictionary`2[TKey,TValue] cache, UnityEditor.BuildTarget target) (at /Users/builduser/buildslave/unity/build/Editor/Mono/AssemblyHelper.cs:160)
4. UnityEditor.AssemblyHelper.FindAssembliesReferencedBy (System.String[] paths, System.String[] foldersToSearch, UnityEditor.BuildTarget target) (at /Users/builduser/buildslave/unity/build/Editor/Mono/AssemblyHelper.cs:194)
5. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)
6.  

Is this achievable.I have seen it in Samsung Bixby application reading the text from objects quite easily.

 

Code (CSharp):

1.             List<KeyPoint> keypointList = keypoints.toList();
2.             foreach (var keypoint in keypointList)
3.             {
4.                 Debug.Log("keypoint.pt " + keypoint.pt);
5.             }

 

OpenCVForUnity2.3.5TrialVersion supports UnityEditor only. Building for iOS is not supported.
https://enoxsoftware.com/opencvforunity/get_asset/

 

If I buy the plugin can the below questions be answered?
So how to test on physical objects...I ran on mac but it takes a built in image as input I believe.How to use device cam and run the test.How to run mac camera and check the physical object?
My Ultimate aim is to run on mobile devices and scan using device camera to detect the text.So basically it is taking video and processing.Can this be done?

 

Unfortunately, I do not have TextDetectionExample using WebCamTexture class or VideoCapture class. However, I think this post will be helpful.
https://forum.unity.com/threads/released-opencv-for-unity.277080/page-28#post-3256861

 

Hello! Maybe my question has already been answered, but I can't find where.

How can I combine two Mat in order to have a new Mat with the original Mats placed side by side horizontally?

Thnaks

 

You can combine multiple Mats by using the Core.hconcat () and Core.vconcat () methods.

Code (CSharp):

1.         void Start ()
2.         {
3.             //if true, The error log of the Native side OpenCV will be displayed on the Unity Editor Console.
4.             Utils.setDebugMode (true);
5.  
6.  
7.             Texture2D imgTexture = Resources.Load("lena") as Texture2D;
8.  
9.             Mat imgMat1 = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC4);
10.  
11.             Utils.texture2DToMat(imgTexture, imgMat1);
12.             Debug.Log("imgMat.ToString() " + imgMat1.ToString());
13.  
14.             Mat imgMat2 = new Mat(imgTexture.height, imgTexture.width, CvType.CV_8UC4);
15.  
16.             Utils.texture2DToMat(imgTexture, imgMat2);
17.             Debug.Log("imgMat2.ToString() " + imgMat2.ToString());
18.  
19.             Core.flip(imgMat2, imgMat2, 1);
20.  
21.  
22.  
23.             Mat dst = new Mat();
24.  
25.             List<Mat> src = new List<Mat>(new Mat []{ imgMat1, imgMat2 });
26.  
27.             Core.hconcat(src, dst);
28.  
29.  
30.             Texture2D texture = new Texture2D(dst.cols(), dst.rows(), TextureFormat.RGBA32, false);
31.  
32.             Utils.matToTexture2D(dst, texture);
33.  
34.             gameObject.GetComponent<Renderer>().material.mainTexture = texture;
35.  
36.  
37.             Utils.setDebugMode (false);
38.         }

 

Hi!!

Amazing plugin, got a few question:
i've tried the free trial and it works perfectly but my goal is to have a simple update/ callback /trigger/anything that tell me "Ehy i saw someone in front of a camera" while hiding the triangle stuff. Some way to know that camera "find" a face. No matter what face it is, just "i saw a face" or not.
Can't find much in the example code, is something unlocked with the full version? I'm missing something???

 

Hello and thanks for replying to my previous post here, @EnoxSoftware!

I have another, kind of general question: is there any way to detect curves? As much as possible, ONLY the curves?

 

You can use the following code to determine if a face is detected.

Code (CSharp):

1.                 OpenCVForUnity.CoreModule.Rect[] rects = faces.toArray ();
2.                 if( rects.Length > 0) {
3.                     Debug.Log ("face detected.");
4.                 }else{
5.                     Debug.Log ("face not detected.");
6.                 }

https://github.com/EnoxSoftware/Ope...aceDetectionWebCamTextureExample.cs#L183-L188

 

Updated to OpenCV for Unity 2.3.6. Having problems getting iOS working on either a universal or 64bit only build. Using Unity 2018.3.0f2. The build succeeds and gets all the way to linking the IL2CPP code and complains about the following native bindings not existing

Undefined symbols for architecture arm64[0m

> Symbol: _shape_ThinPlateSplineShapeTransformer_setRegularizationParameter_10

> Referenced from: _ThinPlateSplineShapeTransformer_setRegularizationParameter_m1EC749B7908EA5E47B80994A8ED25291E5C07C9D in Bulk_Assembly-CSharp_9.o







[31m❌  ld: symbol(s) not found for architecture arm64[0m







[31m❌  clang: error: linker command failed with exit code 1 (use -v to see invocation)[0m

I think I don't actually need these for my application - so I comment out and rebuild and it complains on some more missing bindings out of the shape/ folders. I'm commenting these out to see if I can get into a reasonable state - but wondering if anyone else is seeing this or if you have some recommendations on how to get back in action

 

I haven't tried it yet, but it seems that HoughLinesP method is effective for curve detection.
https://answers.opencv.org/question/185375/how-do-i-detect-the-curvy-lines-in-opencv/

 

It may be the cause of the error that the files of the old version of OpenCVForUnity remain.
Please delete "OpenCVForUnity" folder once and re-import.
Also,
If you do not use opencv_contrib module, build size will be reduced by using native plugin file excluding opencv_contrib module. Please see ReadMe.pdf for details.
1. Replace OpenCVForUnity/Plugins/iOS/libs folder to OpenCVForUnity/Extra/exclude_contrib/iOS/libs folder.
2. Delete OpenCVForUnity/Assets/OpenCVForUnity/org/opencv_contrib folder and OpenCVForUnity/Examples/Contrib folder.

 

Hi,

I would like to evaluate the lib before purchasing, but I can not download the trial package.
If I try to download it with Safari on OSX all that happens is that I go in a circle from the Main page to the Get Package, when I click Get package I'm redirected to Main Page.
If I try to Download with Chrome, the download starts but is slow (around 200Kbit/sec) and whenever I'm close to the end (around 83 MB) I suddenly get a "Network issue" and the download aborts.

I can guarantee that I'm in a good network as I'm in the office with like a 500k line and everything else is working.
Is there another way to get the trial? I wasted to much time on that already.

Thank you.

 

This fixed things - Thanks a ton!

Cheers

 

I have an app that uses this asset and works on desktop and on Android it will crash after a random time, usually around 1 minute.

How do I go about debugging this?

 

Could you send your email address to the contact form? https://enoxsoftware.com/opencvforunity/contact/other-inquiry/
I will send you the download link.

 

Thank you very much for reporting.
In which example does a crash occur?