[RELEASED] OpenCV for Unity

Hello,
i have a problem i am hoping you can give a solution for it.
i am using color detection and i want the user to select the color.
when the player select a color using camera i make a new ColorObject the problem is that i dont know which value the Scalar uses RGB or HSV values and i don't know how to find max and min values of that color.
So I have RGB values and from it i want find max and min and assign them to the ColorObject.
Thanks.

 

Cannot seem to run the yolov4 example without crashing in Unity 2020.3 LTS

 

So the links to tinyyolo3 do not work for tinyyolo4 ... this is causing crashes. What are the correct links to the v4?

 

I just tested it in the same environment as yours, and it worked fine in both yolov3-tiny and yolov4-tiny.

My environment :
Unity2020.3.0f1(LTS)
OpenCVForUnity2.4.3

yolov3-tiny files:
https://raw.githubusercontent.com/pjreddie/darknet/master/cfg/yolov3-tiny.cfg
https://pjreddie.com/media/files/yolo3-tiny.weights

yolov4-tiny files:
https://github.com/AlexeyAB/darknet/raw/master/cfg/yolov4-tiny.cfg
https://github.com/AlexeyAB/darknet/releases/download/darknet_yolo_v4_pre/yolov4-tiny.weights

 

Hi there,

I can't find the scene for AR face tracking. I need to create some AR content like snapchat filters.

Where can I find it?

 

dyld: Library not loaded: @ rpath / opencv2.framework

I'm getting exactly the same error on unity 2020.2.1f1 using xcode 12.4 and ios 14.4
Any suggestions?

Thank you very much!

Michal

 

Edit: Okay was me being a noob, use the Tools > OpenCVForUnity > Set plugin import settings menu item to fix this

Same here on 2020.3 in an IOS project too, using Xcode 12.4


UnityFramework, 265): Library not loaded: @rpath/opencv2.framework/opencv2

 

libiconv.2.dylib not found

ran into this issue while trying to build to iOS XCode an old OpenCVForUnity project from 2019.

 

Hi, I was wondering if anyone has managed to use the dnn module in OpenCV for Unity, specifically a yolov4-tiny model, with GPU and OpenGL. I believe the OpenCV wrapper currently uses OpenCL, but this is rather limiting when running a model on a mobile device. We have already developed an entire app and interface for a customer project using this wrapper, but have ran into some performance limitations that we can't seem to overcome without having access to utilizing GPU for these dnn algorithms.

 

Ignore OpenCV if its Yolo implementation is CPU only, there is an implementation of Tiny Yolo v3 here which I have recently ported from using AR (which is obviously limited to CPU only on Android with ARCore as ARCore does kot support Vulkan and Unity Barracuda only really uses Vulkan on Android). IOS uses GPU with metal which means Iphone 6 and onwards you are good.

Also important to note that Unity Barracuda 1.3.2 (I think) also has some drastic speed improvements for CPU inference.

If your use case is for non-AR, feel free to DM me I can share what I did to get it working with webcamtexture and also you can pass any kind of texture to it if you needed to run inference on a video or similar.

All you need to do is remove the AR stuff in the scene and where the script for detection uses XR CPU image just provide it with a texture from elsewhere

https://github.com/derenlei/Unity_Detection2AR

 

https://forum.unity.com/threads/no-reference-to-libiconv-2-dylib-on-xcode-7.353999/

 

Hello. I have a question. I want to pass data in Mat format captured through this asset from C# to Java. . . How can I convert C#'s Mat to Android Java's Mat??

In general, an attempt is made to insert Intptr in C# Mat into long in Java Mat by calling Java Class existing in Java AAR Plugin File, but it fails.

 

Hello I have a question.Problem with processing Mat into Imgproc.floodfill.Mat is from WebCamTexture.All process works except floodfill function,it doesnt give me back exact Point where he should floodfill.Could you help me which paramaters of Point and Rect should be for the floodfill to work(because it's WebCamTexture that is real time process per frame)?

 

There is probably no way to convert it directly. Once you get the data array from the C# mat, you need to set the data array in the Java mat.

The following code can be used to retrieve data from Mat in C#.

Code (CSharp):

1. byte[] data = new byte[ csharpMat.rows() * csharpMat.cols() * csharpMat.elemSize()];
2. csharpMat.get( 0, 0, data);

 

Hi,

I am working on a project using the MarkerBased AR Reader from Open CV for Unity. All of my builds have worked fine both in Android and iOS in testing, but once I try to upload my project into the App Store for iOS through XCode, I am receiving the error message "No suitable application records were found. Verify your bundle identifier 'org.opencv' is correct."

I have
1. Checked my import setting in Unity so that the plugins are in the plugin folder
2. Turned of Enable Bitcode in Xcode
3. Turned on Verify Workspace in Xcode
4. Embed and Signed the opencv2 framework
5. Added a script in the build phase to override the Unsupported Architecture Error.

Code (CSharp):

1. APP_PATH="${TARGET_BUILD_DIR}/${WRAPPER_NAME}"
2.  
3. # This script loops through the frameworks embedded in the application and
4. # removes unused architectures.
5. find "$APP_PATH" -name '*.framework' -type d | while read -r FRAMEWORK
6. do
7.     FRAMEWORK_EXECUTABLE_NAME=$(defaults read "$FRAMEWORK/Info.plist" CFBundleExecutable)
8.     FRAMEWORK_EXECUTABLE_PATH="$FRAMEWORK/$FRAMEWORK_EXECUTABLE_NAME"
9.     echo "Executable is $FRAMEWORK_EXECUTABLE_PATH"
10.  
11.     EXTRACTED_ARCHS=()
12.  
13.     for ARCH in $ARCHS
14.     do
15.         echo "Extracting $ARCH from $FRAMEWORK_EXECUTABLE_NAME"
16.         lipo -extract "$ARCH" "$FRAMEWORK_EXECUTABLE_PATH" -o "$FRAMEWORK_EXECUTABLE_PATH-$ARCH"
17.         EXTRACTED_ARCHS+=("$FRAMEWORK_EXECUTABLE_PATH-$ARCH")
18.     done
19.  
20.     echo "Merging extracted architectures: ${ARCHS}"
21.     lipo -o "$FRAMEWORK_EXECUTABLE_PATH-merged" -create "${EXTRACTED_ARCHS[@]}"
22.     rm "${EXTRACTED_ARCHS[@]}"
23.  
24.     echo "Replacing original executable with thinned version"
25.     rm "$FRAMEWORK_EXECUTABLE_PATH"
26.     mv "$FRAMEWORK_EXECUTABLE_PATH-merged" "$FRAMEWORK_EXECUTABLE_PATH"
27.  
28. done
29.  

I have built and upload an application using the OpenCv plugin before, but I cannot get his particular project working.

Is there anyway to solve this problem?

I am working on
XCode : 12.3
Unity : 2019.2.15f1
macOS : Catalina 10.15.5

Thank you,
Panu

 

SOLVED

I reimported all the open cv asset plugins and setting the import setting again and it now works

 

Currently having a serious issue of being unable to upload or verify an IOS archive build of Unity apps using OpenCV that work fine when built and installed to ipad/iphone

We get the error unexpected Mach-O header code: 0x72613c21 and it doesn't go away until I go to the Build Phases for the app target in Xcode - Embed frameworks and remove libopencvforunity.a as well as OpenCVForUnityAppController.mm which probably has broken OpenCV in my app

Using OpenCV 2.4.3, Xcode 12.4 and Unity 2020.3.0f1

Issue link on github:
https://github.com/EnoxSoftware/OpenCVForUnity/issues/103

 

I'll reply to this issue on github.

 

Hi @EnoxSoftware and all,

do you know how I can delete the 'OpenCV For Unity' sentence that appears on the textures (as in the picture attached) when clicking play button?

Thank you

 

It is a watermark that appears only when using the trial version.

 

Thank you @EnoxSoftware. So there is no way to eliminate it when using the free trial version?

Another question. Is the price for which the package is sold in the Asset Store una tantum? I mean, I pay 95$ once or do I have to renew the purchase periodically?

Thank you

 

Of course, once you purchase the software, you can use it permanently, and you can also download updated versions based on AssetStore's terms and conditions.

 

Thank you very much for your reply!

 

Hi @EnoxSoftware, first of all thank you for the great work, I purchased your plugin in 2017 and happily used it since then!

One short question: We urgently need one feature from the OpenCV 4.5.1 release (from Dec 2020) or even better 4.5.2 from this month. Can you say approximately when you are going to integrate these two releases into your plugin?

 

hello, in cpp, i can directly edit the value using Mat.at() function,

how can i edit pixel value directly?

i found that there are utils.matToTexture2d().
beside convert the mat to texture2D and using getPixels function,

any other faster way?, ty

 

The next version of OpenCVForUnity, which integrates OpenCV 4.5.2, will be released this month.

 

The AccessingPixelValueExample of MatBasicProcessingExample is a good reference for manipulating Mat pixel values.
https://github.com/EnoxSoftware/Ope...mple/MatBasicProcessingExample.cs#L2010-L2320

 

I plan to purchase this asset but due to Covid 19 so I will be working on 2 computers, one is my personal and the other is from the company, do I need to purchase this asset 2 times to be able to use it in my 2 computers or as long as the two using the same account, I can use the OpenCVforUnity on both of them (1-time purchase for my account)?

 

Hello, could you please add Mat creation from native (user-allocated) data pointer? For example if user have color buffer stored somewhere copying it to the new mat is not efficient at all.

 

Unity seat license allows "one" user per seat to install and use Unity on up to "two" computers simultaneously.

https://unity3d.com/jp/legal/as_terms
2.3.2 END-USER is granted a single seat license to install and use any Asset categorized in the Asset Store as an "Editor Extension" “Scripting”, or “Services” (collectively, “Extension Asset”) only on a maximum of 2 computers. For the avoidance of doubt, Extension Assets are licensed on a per seat basis and may not be shared or used concurrently on more than 2 different computers. As an exception, build farm servers and virtual machine instances used only for running, testing, or building projects with Extension Assets do not require separate seat license(s) or constitute use on more than 2 different computers.

 

Original Post: 6740311

So I have the same crash (ok if created in Editor, crashes on reload - Windows 10, OpenCVForUnity 2.4.3, Unity 2020.3.4f1 + other 2020 & 2019 versions), but for a slightly different scenario in that the offending line was for MatOfKeyPoint which was created in the class constructor. This class would have been instantiated during the Editor startup as it was a Serializable property. Placing the MatOfKeyPoint creation in a separate Init() method which in turn was called from a Start() method solved the problem.

Code (CSharp):

1. // this class crashes
2. [Serializable]
3. public class abc
4. {
5.     MatOfKeyPoint _keypoints;
6.  
7.     public abc()
8.     {
9.         _keypoints = new MatOfKeyPoint(); // crash here
10.     }
11. }
12.  
13. // This class loads fine when Init() is called from Start()
14. [Serializable]
15. public class abc
16. {
17.     MatOfKeyPoint _keypoints;
18.  
19.     public void Init()
20.     {
21.         _keypoints = new MatOfKeyPoint();
22.     }
23. }

Hopefully this helps, confirms that it's a general problem (MatOfKeyPoint& MatOfDouble + others?) and not just MatOfDouble. Because the solution revolves around class construction and sequencing it could mean that the opencv libs are not fully loaded/initialized during the Editor load process, maybe?

Thanks for a great product.

 

Thank you for the useful information.
I would like to find a workaround for this problem.

 

Hi @EnoxSoftware and all.
I have seen that using, for example, OpenCV for Python I can change the color of one pixel or of a region of pixels in this way:

Code (CSharp):

1. img[50,50] = (0,0,255)  // for a single pixel
2.  
3. img[0:150, 0:300] = [0,0,255] // for a region of pixels
4.  
5.  

I want to make the same thing using OpenCV for Unity. I have created a mat

Code (CSharp):

1. rgbaMat = new Mat(h, w, CvType.CV_8UC4);

and I want to change first the color of one of its pixels and then of a region of pixels... However I cannot find the proper syntax to do so in Unity... Can you help me solve?

I tried for example

Code (CSharp):

1. rgbaMat.Set<Vec3b>(i, j, new Vec3b(0, 0,255));

but it tells me
The type or namespace name 'Vec3b' could not be found (are you missing a using directive or an assembly reference?)

 

Code (CSharp):

1.             img.put(50, 50, new int[] { 0, 0, 255 });  // for a single pixel
2.  
3.             Mat roi = new Mat(img, new OpenCVForUnity.CoreModule.Rect(0, 0, 150, 300));
4.             roi.setTo(new Scalar(0, 0, 255)); // for a region of pixels

OpenCVForUnity is a C# wrapper for OpenCV 4.5.0. You will need to rewrite the code in C#.Since this asset is a clone of OpenCV4.5.0 Java, you are able to use the same API as OpenCV4.5.0 Java.
https://docs.opencv.org/4.5.0/javadoc/index.html
https://enoxsoftware.github.io/OpenCVForUnity/3.0.0/doc/html/annotated.html

 

@EnoxSoftware thanks for your reply! I tried to implement your suggestion for a single pixel but it gives me an error. Here is my piece of code... I have a rgbaMat (of which I want to change one color pixel) and a destination Mat called newdstMat. I am showing on texture left and texture right the left and right views from HTC Vive Pro Eye cameras...

Code (CSharp):

1. private void UpdateFilter(byte[] framebuffer)
2.     {
3.         rgbaMat.put(0, 0, framebuffer);
4.  
5.         if (enableFilter)
6.         {
7.            
8.             rgbaMat.put(50, 50, new int[] { 0, 0, 255 });
9.             rgbaMat.copyTo(newdstMat);
10.  
11.  
12.         }
13.         else
14.         {
15.             rgbaMat.copyTo(newdstMat);
16.         }
17.  
18. OpenCVForUnity.UnityUtils.Utils.matToTexture2D(newdstMat.rowRange((int)size.height / 2, (int)size.height), textureLeft);
19.  
20. OpenCVForUnity.UnityUtils.Utils.matToTexture2D(newdstMat.rowRange(0, (int)size.height / 2), textureRight);
21. }

When I hit play it gives me this error:
CvException: Provided data element number (3) should be multiple of the Mat channels count (4)
OpenCVForUnity.CoreModule.Mat.put (System.Int32 row, System.Int32 col, System.Int32[] data) (at <d43a46b111fb46658de86fc016687261>:0)
FilterEffect.UpdateFilter (System.Byte[] framebuffer)

What can the problem be?

 

Could you change new int[] { 0, 0, 255,} to new int[] { 0, 0, 255, 255 }?

Code (CSharp):

1. img.put(50, 50, new int[] { 0, 0, 255, 255 });  // for a single pixel

 

@EnoxSoftware I tried but it tells me that Mat data type is not compatible: 24

my Mat is a CvType.CV_8UC4

 

int -> byte

Code (CSharp):

1. img.put(50, 50, new byte[] { 0, 0, 255, 255 });  // for a single pixel

 

@EnoxSoftware thank you so much! Now it works both for a single pixel and a region.. I'm quite new to OpenCV for Unity and you gave me a great help. One last thing... I have seen it modifies only the texture in which is rendered the right camera view... do you know how it can be possible?

Code (CSharp):

1. private void UpdateFilter(byte[] framebuffer)
2.     {
3.         rgbaMat.put(0, 0, framebuffer);
4.  
5.         if (enableFilter)
6.         {
7.             rgbaMat.copyTo(newdstMat);
8.             Mat roi = new Mat(newdstMat, new OpenCVForUnity.CoreModule.Rect(0, 0, 200, 460));
9.             roi.setTo(new Scalar(127, 255, 0, 255));    
10.         }
11.         else
12.         {
13.             rgbaMat.copyTo(newdstMat);
14.         }
15.  
16.         OpenCVForUnity.UnityUtils.Utils.matToTexture2D(newdstMat.rowRange(0, (int)size.height / 2), textureRight);
17.         OpenCVForUnity.UnityUtils.Utils.matToTexture2D(newdstMat.rowRange((int)size.height / 2, (int)size.height), textureLeft);
18.     }

 

I have created the test scene to verify the issue. (based on the Texture2DToMatExample)

Code (CSharp):

1.  
2.             Texture2D imgTexture = Resources.Load ("face") as Texture2D;
3.  
4.             Mat imgMat = new Mat (imgTexture.height, imgTexture.width, CvType.CV_8UC4);
5.  
6.             Utils.texture2DToMat (imgTexture, imgMat);
7.             Debug.Log ("imgMat.ToString() " + imgMat.ToString ());
8.  
9.  
10.             // Cleate textures.
11.             Texture2D textureRight = new Texture2D(imgMat.cols(), imgMat.rows() / 2, TextureFormat.RGBA32, false);
12.             Texture2D textureLeft = new Texture2D(imgMat.cols(), imgMat.rows() / 2, TextureFormat.RGBA32, false);
13.             RightQuad.GetComponent<Renderer>().material.mainTexture = textureRight;
14.             LeftQuad.GetComponent<Renderer>().material.mainTexture = textureLeft;
15.  
16.             // Fill in the central area of the image.
17.             Mat roi = new Mat(imgMat, new OpenCVForUnity.CoreModule.Rect(223, 223, 64, 64));
18.             roi.setTo(new Scalar(127, 255, 0, 255));
19.  
20.             // Upload different areas of image data to the left and right textures.
21.             Utils.matToTexture2D (imgMat.rowRange(0, (int)imgMat.size().height / 2), textureRight);
22.             Utils.matToTexture2D(imgMat.rowRange((int)imgMat.size().height / 2, (int)imgMat.size().height), textureLeft);

In my tests, this code seems to work fine.

 

@EnoxSoftware sorry for the late reply. I managed to make my project work.
Thanks again for the help!

 

Hello @EnoxSoftware sorry for bothering you again...

I tried to use the code 'unsafe' (OPENCV_USE_UNSAFE_CODE) for accessing the pixel of my image but I have seen that this part of code is not executed, even if I set project settings --> player --> configuration --> use unsafe at TRUE.

Did I made something wrong or did I miss something?

I also tried the example scene but also there the part of code included in the unsafe context is not executed.

Thank you!

 

If you want to use optimization code using NativeArray class, select MenuItem[Tools/OpenCV for Unity/Use Unsafe Code] and check "Arrow unsafe code". ( require Unity2018.1 or later )

 

Hi there, I am working on implementing a Text Recognition plugin using OpenCV and Unity ,till now I have used 2 modules provided by OpenCV Unity asset
The OCR Example of MainModules is working fine in Pc and Android Device but it is slow. So I decided to use a Contrib Module ie TextRecognitionExample which seems to be more fast in PC. However when I make an android build and try to run it , the build crashes.

I have attached my Crash Logs , please look into it and help me to debug and fix it.

 

Thank you very much for reporting.

https://stackoverflow.com/questions/4666098/why-does-android-aapt-remove-gz-file-extension-of-assets
When I add a GZIP-ed file to my Android project's assets, the ".gz" extension is stripped when the project is packaged. (So, for instance, "foo.gz" in my assets folder needs to be accessed in code using getAssets().open("foo").)

Could you change the code as follows?

https://github.com/EnoxSoftware/Ope...ext/TextExample/TextRecognitionExample.cs#L68

Code (CSharp):

1.         #if UNITY_ANDROID && !UNITY_EDITOR
2.         protected static readonly string OCRHMM_KNN_MODEL_FILENAME = "text/OCRHMM_knn_model_data.xml";
3.         #else
4.         protected static readonly string OCRHMM_KNN_MODEL_FILENAME = "text/OCRHMM_knn_model_data.xml.gz";
5.         #endif      

 

Dear EnoxSoftware,
We have been happy to use OpenCV for Unity and Dlib FaceLandmark Detector since last year. However now we have problems distributing our application and depend on your support. We are facing the following two problems:

1. Could you 'move' the purchased "OpenCV for Unity" and "Dlib FaceLandmark Detector" assets to our account?
Apparently it is impossible as a Unity customer to move purchased asset packs between accounts ourselves
The suggested resolution in the article from Unity is to contact the Publisher of the Asset directly.
An external contractor purchased the "OpenCV for Unity" asset from the asset store on our behalf.
This made sense at that time - however, the availability of the contractor to work for us is now very limited.
It is important for us to be able to work with the asset ourselves.
Primarily we want to be able to update the asset within our project, which leasds me to problem number 2.

2. Code signing of the XCode project fails for us:
The ends of last two lines of IDEDistributionPipeline.log (using fastlane) are

Here is our related Issue at fastlane and another post here on the uniy forum by me.
We suspect, that this could be an issue with opencv and that we would be pointed in this direction.
Something which I would have wanted to try is to update the asset pack - however, this right now impossible for me, due to problem number 1.

Thanks in advance for your support

 

Thank you very much for reporting.
Could you send me an email using the contact form below?
https://enoxsoftware.com/opencvforunity/contact/technical-inquiry/

 

Thank you for the fast reply @EnoxSoftware - I have filled out the contact form, with the text from my above post and added a log file from the build pipeline.

 

Thanks a lot it worked!!!
Although I fixed it last week by manually pasting the OCRHMM KNN model file to my android /data/... and it worked but obviously that was just a temporary fix, Thanks again!

 

Dear EnoxSoftware,

I am working on a Text Recognition based game where I want to detect what letters are visible in front of my camera .
I am using Unity's Webcam Texture which I later pass to Mat Class for processing. Text Detection and Recognition is working fine ,but now I have to inject a Depth based filtration process where only the letters which are within a specified distance range from the camera will be send for Processing and the letters which are beyond the range will not be included in the Detection and Recognition process.