[RELEASED] OpenCV for Unity

@EnoxSoftware I can't get the bilateral filter operation to actually work, I'm trying to use it in a realtime image processing pipeline but to verify it was not performing the operation I created an editor script that loads an image, attempts to apply the filter, and saves a copy of the image. I verified that the input and output images are identical, I'm not sure why this operation in particular is failing as I'm using plenty of other Imgproc methods succesfully. Below is my test code as well as the input and output image.

Code (CSharp):

1.     [ContextMenu("Test")]
2.     void Test()
3.     {
4.         Mat image = Imgcodecs.imread("[OBFUSCATED]\\cat.jpg");
5.         Imgproc.bilateralFilter(image, image, 9, 150, 150);
6.         Imgcodecs.imwrite("[OBFUSCATED]\\bilateral.jpg", image);
7.     }

Original image


Filtered image

 

Last test, it's a mistery

Just new photo (not trained) converted to jpg and voila...
I don't know what happen with all other photo, format ? compression ? or ???

 

Just to be sure, I have to ask, are the defects you are currently encountering that only occur on specific models the result of testing the YOLOv5 example (v1.0.2) on a clean project?
Several of your recent posts seem to be about modified code.

 

Curiously, OpenCV seems to generate an internal error if the src and dst of the bilateralFilter function are the same mat.
It is necessary to temporarily prepare a mat of the same size for output.

Code (CSharp):

1.  
2.     [ContextMenu("Test")]
3.     void Test()
4.     {
5.         Mat image = Imgcodecs.imread("[OBFUSCATED]\\cat.jpg");
6.         Mat dst = image.clone();
7.         Imgproc.bilateralFilter(image, dst, 9, 150, 150);
8.         dst.copyTo(image);
9.         Imgcodecs.imwrite("[OBFUSCATED]\\bilateral.jpg", image);
10.     }
11.  

This error can be confirmed by enabling debug mode.

Code (CSharp):

1.  
2. //if true, The error log of the Native side OpenCV will be displayed on the Unity Editor Console.
3. Utils.setDebugMode(true);
4.  
5. Imgproc.bilateralFilter(image, image, 9, 150, 150);
6.  
7. Utils.setDebugMode(false);
8. //
9. imgproc::bilateralFilter_11() : OpenCV(4.7.0-dev) C:\Users\satoo\Desktop\opencv\modules\imgproc\src\bilateral_filter.dispatch.cpp:166: error: (-215:Assertion failed) (src.type() == CV_8UC1 || src.type() == CV_8UC3) && src.data != dst.data in function 'cv::bilateralFilter_8u'
10. //
11.  

 

Clean project or modified code is same problem with the model, like I said in my previous post all photo used in model not work, only new photo... (my last post with good result is modified code and new photo)

 

Hmmm, that is really strange behavior.
I can't think of a cause or solution...

 

@ibompuis don't do this

Code (CSharp):

1. Imgcodecs.imread(Application.persistentDataPath+"/test.jpg");

Do this

Code (CSharp):

1. Imgcodecs.imread(Path.Combine(Application.persistentDataPath, "test.jpg"));

 

Hi - thanks for all your work and support on OpenCV for Unity!

I am working with a RealSense 455d camera and have used the RealSense SDK to do some image processing on the camera input and I now want to pass that to the OpenCV of Unity SDK to do perform blob detection. Right now the output from the RealSense pipeline is a binding to a RawImage.texture on a canvas like so:



However, now I'd like to send this output to something like your `
MultiObjectTrackingBasedOnColorExample` script, but I'm unsure how to connect the dots between the `RawImage.texture` I have and the process in your script. It appears that the `
WebCamTextureToMatHelper` is a key part of this interface, but how could I connect my `RawImage.texture` output to this or some other part of the script? Thanks!

 

Unfortunately I don't have a RealSence so I haven't tried it, but I did find the following examples.
https://qiita.com/UnagiHuman/items/f877da98d7600e3d579c
According to this example article, integration with OpenCVForUnity seems to be possible.

 

Hello! Quick question. Does this API allow turn on/off phone flashlight while AR is active?

 

OpenCV does not have the ability to control mobile phone flashlight.
Third party assets must be linked.

 

@EnoxSoftware hi guys, back on the DNN Object Detection track, thank you for adding Yolov7, but you did not include Yolov5 in the distribution / release version, also you are using .weights file in the Yolov7 example and not ONNX, the main purpose of my original request was to use ONNX files.

Is the issues that you can't covert the weigts file to ONNX, because the main use case is to take Pytorch .pt files to .ONNX, this works 100% for Yolov5 example you created.

I did some digging on this and found the following:
How export to onnx be used by opencv?
https://github.com/WongKinYiu/yolov7/issues/49

I also read your notes about ONNX in the example, if you can have a look at this alternative solution as per the above mentioned issue.

Opencv dnn success run custom yolov7
1. reparam trained best.pt
https://github.com/WongKinYiu/yolov7/blob/main/tools/reparameterization.ipynb
nc = 1 #change with your nc
model = Model('cfg/deploy/yolov7.yaml', ch=3, nc=nc).to(device)
for i in range( (nc + 5) * 3):

2 use branch https://github.com/WongKinYiu/yolov7/tree/u5 to export onnx model
3 run with the same as yolov5 6.1
https://github.com/VITA-Alchemy/yolov5_6.0_opencvdnn_python/blob/main/main_dnn.py

 

Hello, I'm getting an almost black screen from a WebGL build on Chrome. To be specific, the texture shows one frame that was captured, then just goes black. Building HandPoseEstimationExample using SimpleWebServer if it matters.

 

There seems to be a bug in Unity 2022.2.0f1 and above where webcamTexture.didUpdateThisFrame only updates the first frame in WebGL build.
Could you try a build with Unity2021?

 

Hello, the SimpleBlobExample doesn't seem to work. In Unity and the OpenGL example I get an error: "error: (-5:Bad argument) minDistBetweenBlobs>0 in function 'validateParameters'". When setting minDistBetweenBlobs=10, minArea=1, maxArea=100000, minCircularity=1, maxCircularity=100000, minInertiaRatio=1, maxInertiaRatio=100000, minConvexity=1, maxConvexity= 100000 in code I don't get an error anymore.

 

Thank you very much for reporting. This problem will be fixed in the next version.

 

Hello, I'm having a problem with hconcat, it works for 2 Mats but not for 3.
I'm using:

Code (CSharp):

1. Size max = new Size(Math.Max(one.size().width, Math.Max(two.size().width, colourMatch.size().width)), Math.Max(one.size().height, Math.Max(two.size().height, colourMatch.size().height)));
2. Imgproc.resize(one, one, max);
3. Imgproc.resize(two, two, max);
4. Imgproc.resize(colourMatch, colourMatch, max);
5.  
6. Mat combined = new Mat();
7. Core.hconcat(new List<Mat>() { one, two, colourMatch }, combined);

If I don't include colourMatch it works, if I do it returns a Mat that is 0x0 in size, so matToTexture2D fails with GetPixels32 on degenerate image. I can use matToTexture2D on all three Mats (one, two, and colourMatch) individually and it works fine, but after combining it creates a Mat that is 0x0 and therefore causes an error when trying to use matToTexture2D.

Note that it only fails with colourMatch so it's likely a problem with that Mat, it's created like so:

Code (CSharp):

1. Mat mat = new Mat(75, 50, CvType.CV_8UC3);
2.         for (int x = 0; x < 50; x++)
3.         {
4.             for (int y = 0; y < 75; y++)
5.             {
6.                 if (x <= 24 && y <= 24)
7.                 {
8.                     // top left corner
9.                     mat.put(y, x, new double[] { color1.r, color1.g, color1.b });
10.                 }
11.                 else if (x <= 24 && y <= 49)
12.                 {
13.                     // middle left
14.                     mat.put(y, x, new double[] { color2.r, color2.g, color2.b });
15.                 }
16.                 else if (x <= 24 && y <= 74)
17.                 {
18.                     // bottom left corner
19.                     mat.put(y, x, new double[] { color3.r, color3.g, color3.b });
20.                 }
21.                 else if (x > 24 && y <= 24)
22.                 {
23.                     // top right corner
24.                     mat.put(y, x, new double[] { color4.r, color4.g, color4.b });
25.                 }
26.                 else if (x > 24 && y <= 49)
27.                 {
28.                     // middle right
29.                     mat.put(y, x, new double[] { color5.r, color5.g, color5.b });
30.                 }
31.                 else if (x > 24 && y <= 74)
32.                 {
33.                     // bottom right corner
34.                     mat.put(y, x, new double[] { color6.r, color6.g, color6.b });
35.                 }
36.             }
37.         }
38.         return mat;

Why is this happening and how do I fix it? Thanks.

EDIT: Fixed, one and two were 8UC4 and colourMatch was 8UC3.

 

What's the expected framerate on the lightweight pose estimation? I'm only getting 15fps on a good computer with the trial version

 

Our simple test code did not generate any errors.

Code (CSharp):

1.  
2. using OpenCVForUnity.CoreModule;
3. using OpenCVForUnity.ImgprocModule;
4. using OpenCVForUnity.UnityUtils;
5. using System;
6. using System.Collections.Generic;
7. using UnityEngine;
8.  
9. public class HconcatTest : MonoBehaviour
10. {
11.  
12.     // Start is called before the first frame update
13.     void Start()
14.     {
15.         //if true, The error log of the Native side OpenCV will be displayed on the Unity Editor Console.
16.         Utils.setDebugMode(true);
17.  
18.  
19.         Mat one = new Mat(100, 100, CvType.CV_8UC3);
20.         Mat two = new Mat(200, 200, CvType.CV_8UC3);
21.         Mat colourMatch = new Mat(300, 300, CvType.CV_8UC3);
22.  
23.         //
24.         Size max = new Size(Math.Max(one.size().width, Math.Max(two.size().width, colourMatch.size().width)), Math.Max(one.size().height, Math.Max(two.size().height, colourMatch.size().height)));
25.         Imgproc.resize(one, one, max);
26.         Imgproc.resize(two, two, max);
27.         Imgproc.resize(colourMatch, colourMatch, max);
28.  
29.         Mat combined = new Mat();
30.         Core.hconcat(new List<Mat>() { one, two, colourMatch }, combined);
31.         //
32.  
33.  
34.         Debug.Log("one:" + one);
35.         Debug.Log("two:" + two);
36.         Debug.Log("colourMatch:" + colourMatch);
37.  
38.         Debug.Log("combined:" + combined);
39.  
40.  
41.         Utils.setDebugMode(false);
42.     }
43. }
44.  

Enabling Utils.setDebugMode(true); may help identify the cause of the error.

 

That was about the same FPS on my PC.
It may be possible to increase the FPS if we find another even lighter model.
Although not related to OpenCV, a plugin for easy Human Pose Estimation in Unity is available here.
I think this is faster since it uses Barracuda for inference.
https://github.com/keijiro/BodyPixBarracuda

 

Hi, I'm currently attempting to implement a WebGL webcam video recorder. The problem I'm having so far happens right on the Editor where for some reason I am unable to save both video (with VideoWriter) and images (with imwrite) to the Application.persistentDataPath folder.
Saving on other folders, such as Desktop, works completely fine.
Also, I tested creating a simple text file on that folder using File.WriteAllText and it works correctly, which lead me to believe it has something to do with the plugin itself.

The code I wrote for the camera operation is as follows:

Code (CSharp):

1.     private IEnumerator _CameraOperationCoroutine(){
2.         VideoWriter writer = new VideoWriter();
3.         string save_path = Application.persistentDataPath;
4.         do{
5.             switch (this.camStatus){
6.                 case CameraStatus.NoDevice:
7.                 case CameraStatus.Unauthorized:
8.                 case CameraStatus.Unavailable:
9.                     ToggleErrorMessage(true);
10.                     break;
11.                 case CameraStatus.Initializing_Feed:
12.                     _webcamDisplay.material.mainTexture = _webCamTexture;
13.                     _webCamTexture.Play();
14.                     this.camStatus = !_webCamTexture.isPlaying ? CameraStatus.Unavailable : CameraStatus.Idle;
15.                     break;
16.                 case CameraStatus.Starting_Recording:
17.                     string filename = DateTime.Now.ToString("ddMMyyyyHHmmssff") + ".mp4";
18.                     writer = new VideoWriter(Path.Combine(save_path,filename),VideoWriter.fourcc('h','2','6','4'),30, new Size(_webCamTexture.width,_webCamTexture.height));
19.                     this.camStatus = CameraStatus.Recording;
20.                     break;
21.                 case CameraStatus.Recording:
22.                     if (_webCamTexture.didUpdateThisFrame){
23.                         Mat frame = new Mat(_webCamTexture.height, _webCamTexture.width, CvType.CV_8UC3);
24.                         Utils.webCamTextureToMat(_webCamTexture,frame);
25.                         Imgproc.cvtColor(frame,frame,Imgproc.COLOR_BGR2RGB);
26.                         writer.write(frame);
27.                     }
28.                     break;
29.                 case CameraStatus.Stopping_Recording:
30.                     writer.release();
31.                     string[] files = Directory.GetFiles(Application.persistentDataPath);
32.                     foreach (string file in files)
33.                     {
34.                         Debug.Log("FILE: "+file);
35.                     }
36.                     this.camStatus = CameraStatus.Idle;
37.                     break;
38.                 case CameraStatus.Idle:
39.                 default:
40.                     break;
41.             }
42.             yield return null;
43.         }
44.         while (this.camStatus != CameraStatus.NoDevice && this.camStatus != CameraStatus.Unauthorized && this.camStatus != CameraStatus.Unavailable);
45.     }

 

When the Imgcodecs.imwrite() method was added to VideoWriterExample, the video and images were saved without problems.
Windows10
Unity 2021.1.0f1
OpenCVForUnity 2.5.3

Imgcodecs.imwrite(Application.persistentDataPath + "/VideoWriterExample_image"+frameCount+".png", recordingFrameRgbMat);

https://github.com/EnoxSoftware/Ope...WriterExample/VideoWriterExample.cs#L148-L175

Code (CSharp):

1.         void OnPostRender()
2.         {
3.             if (isRecording)
4.             {
5.                 if (frameCount >= maxframeCount ||
6.                     recordingFrameRgbMat.width() != Screen.width || recordingFrameRgbMat.height() != Screen.height)
7.                 {
8.                     Debug.LogError("Please fix the screen ratio of the Game View to recognize the recording area.");
9.                     OnRecButtonClick();
10.                     return;
11.                 }
12.  
13.                 frameCount++;
14.  
15.                 // Take screen shot.
16.                 screenCapture.ReadPixels(new UnityEngine.Rect(0, 0, Screen.width, Screen.height), 0, 0);
17.                 screenCapture.Apply();
18.  
19.                 Utils.texture2DToMat(screenCapture, recordingFrameRgbMat);
20.                 Imgproc.cvtColor(recordingFrameRgbMat, recordingFrameRgbMat, Imgproc.COLOR_RGB2BGR);
21.  
22.                 Imgproc.putText(recordingFrameRgbMat, frameCount.ToString(), new Point(recordingFrameRgbMat.cols() - 70, 30), Imgproc.FONT_HERSHEY_SIMPLEX, 1.0, new Scalar(255, 255, 255), 2, Imgproc.LINE_AA, false);
23.                 Imgproc.putText(recordingFrameRgbMat, "SavePath:", new Point(5, recordingFrameRgbMat.rows() - 30), Imgproc.FONT_HERSHEY_SIMPLEX, 0.8, new Scalar(0, 0, 255), 2, Imgproc.LINE_AA, false);
24.                 Imgproc.putText(recordingFrameRgbMat, savePath, new Point(5, recordingFrameRgbMat.rows() - 8), Imgproc.FONT_HERSHEY_SIMPLEX, 0.5, new Scalar(255, 255, 255), 0, Imgproc.LINE_AA, false);
25.  
26.                 writer.write(recordingFrameRgbMat);
27.  
28.                 Imgcodecs.imwrite(Application.persistentDataPath + "/VideoWriterExample_image"+frameCount+".png", recordingFrameRgbMat);
29.             }
30.         }

 

Hi, is it possible to use human segmentation as a virtual background like in Zoom?

i using

Core.bitwise_not(rgbaMat, rgbaMat); 

but it turned out to be like the picture I attached below. How do I remove the white masking?

 

If you are interested in doing a Zoom-like screen composite, the following KeyFrameGreenScreenExample code may be helpful.
How to do image compositing using a mask image:
https://github.com/EnoxSoftware/Ope...5bd4fcc6227ec/Assets/OpenCVForUnity/Examples/ Advanced/KeyFrameGreenScreenExample/KeyFrameGreenScreenExample.cs#L228-L232

 

OpenCV for Unity is not working on my M1 Mac. It is giving this error:

DllNotFoundException: opencvforunity assembly:<unknown assembly> type:<unknown type> membernull)
OpenCVForUnity.CoreModule.Mat..ctor (System.Int32 rows, System.Int32 cols, System.Int32 type) (at Assets/OpenCVForUnity/org/opencv/core/Mat.cs:516)
CamApp.Update () (at Assets/CamApp.cs:23)

Can someone help me out?

 

M1Mac ( Apple Sillicon ) is supported with OpenCVForUnity 2.4.7 or higher.

 

Hi, Is there some plan to support Windows arm64 platform (not UWP)? Seeing this error when ran on Windows 11 arm64

DllNotFoundException: Unable to load DLL 'opencvforunity'. Tried the load the following dynamic libraries: Unable to load dynamic library 'opencvforunity' because of 'Failed to open the requested dynamic library (0x06000000) - The specified module could not be found. (WinError:0000007e)
at OpenCVForUnity.CoreModule.Mat..cto

Thank you! Great plugin BTW

 

Thank you for replying. I still haven't found a solution on how to create a virtual background without using a greenscreen. woul you give me some sample

 

I understand that UnityEditor itself does not currently support the Windows 11 ARM64 platform. Is this error displayed when you run a pre-built app?
https://docs.unity3d.com/2023.2/Documentation/Manual/system-requirements.html

 

We have modified HumanSegmentationExample to add a background composite function, please try it out.

 

I used x86 PC to compile for arm64. Then ran on arm64. That's where the error shows up. App runs fine otherwise, just the OpenCV part throws this error.

https://docs.unity3d.com/2023.2/Documentation/Manual/system-requirements.html#desktop

In the compiled app opencvforunity.dll is placed under ..\Plugins\x86_64 (which is the same as x86). I tried changing dll's CPU to Arm64 in the Inspector but when the compile completes it goes back to x64.

 

I did not know that windows ARM64 is supported in Unity 2023.2. Unfortunately, OpenCVForUnity does not currently support windows ARM64.

 

Hello and good day to you.
I'm using OpenCV and Yolov4 for real-time object detection with my custom dataset, and I need to add a UI button above each 3D item detected. I attempted to tweak the drawPred method to meet my requirements but became perplexed. I'm aware of an older question that is similar to mine and has been answered here: https://forum.unity.com/threads/released-opencv-for-unity.277080/page-52#post-6745243
However, when I attempted to use the attached file for that answer, I received the following error:

Code (CSharp):

1. CvException: An error occurred on the C++ side, causing the class initialization to fail.Enclose the point where the error occurs in Utils.setDebugMode() method, and the error on the C++ side will be displayed on the console.
2. OpenCVForUnity.DisposableObject.ThrowIfNullIntPtr (System.IntPtr ptr) (at Assets/OpenCVForUnity/org/opencv/DisposableObject.cs:58)
3. OpenCVForUnity.ObjdetectModule.CascadeClassifier..ctor (System.String filename) (at Assets/OpenCVForUnity/org/opencv/objdetect/CascadeClassifier.cs:76)
4. OpenCVForUnitySample.DetectFace2DTo3DExample.Start () (at Assets/OpenCVForUnity/Examples/DetectFace2DTo3DExample/DetectFace2DTo3DExample.cs:31)

Could you perhaps assist me with placing a UI button above every 3D item detected by the webcam?

 

Hi, H_v_H
It seems that the error was caused by a recent change in the placement path of the StreamingAssets file in OpenCVForUnity.
Please try the corrected example.

 

thank you very much

 

Hi all!

I'm developing on Unity 2021.3.10f1, on a Mac M1 laptop.

When running the examples, I'm finding I regularly get a "OnWebCamTextureToMatHelperErrorOccurred TIMEOUT" error and the webcam view doesn't load. It seems to work on first load, and then if I load a different example without stopping and restarting in Play mode, it has a timeout?

When I've separated one example into its own scene and opened the scene directly, it seems to be timing out too.

Any advice on managing the timeouts better? Is this a common issue? I haven't spotted too many mentions of it in the thread here.

Thank you for your help!
Patrick

 

Thank you very much for reporting.
I have not received any reports of similar problems.
Does changing the value of the "TImeout Frame Count" property change the behavior?

 

Hi, is this something that you are would be supporting in future? Really interested in Win ARM64 support.

Thank you!

 

Yes, I plan to add support for WindowsARM64 to OpenCVForUnity in the future. However, there seems to be the following error now.
https://issuetracker.unity3d.com/issues/nativeplugins-fail-to-build-when-building-for-windows-arm64

 

So far increasing that seems to have solved it... I'll keep an eye on it and report back if I find it reoccurring - but that Timeout Frame Count field looks to be the solution! Must be taking longer than usual to get access to the webcam on my laptop?

 

After executing

Mat opencvmaterial=Imgcodecs.imread(@"exrfile.exr", Imgcodecs.IMREAD_COLOR );

I get

Code (CSharp):

1. CvException: imgcodecs::imread_10() : OpenCV(4.7.0-dev) C:\Users\satoo\Desktop\opencv\modules\imgcodecs\src\grfmt_exr.cpp:103: error: (-213:The function/feature is not implemented) imgcodecs: OpenEXR codec is disabled. You can enable it via 'OPENCV_IO_ENABLE_OPENEXR' option. Refer for details and cautions here: https://github.com/opencv/opencv/issues/21326 in function 'cv::initOpenEXR'

Is this a compilation oversight or is exr support unavailable?
Exr is marked as supported in documentation.

 

Please enable OPENCV_IO_ENABLE_OPENEXR as per the error message.
https://github.com/opencv/opencv/issues/21326

https://support.unity.com/hc/en-us/...environment-variable-s-set-how-can-I-do-that-
Variable name OPENCV_IO_ENABLE_OPENEXR
Variable value 1

 

Apologies, after trying the script preprocessor defines I assumed it was a compilation parameter.

Adding

System.Environment.SetEnvironmentVariable("OPENCV_IO_ENABLE_OPENEXR", "1", EnvironmentVariableTarget.Process);

before imread call seems to work on android builds, but for windows, after a long while debugging, I found I had to modify the build script to not preload the dll;

pluginImporter.isPreloaded = false;

Something to consider.

 

Hello, may I ask why OpenCVForUnity can only use the CascadeClassifier detector for face detection on hololens2, but not Dnn deep learning algorithm? The performance of Dnn deep learning algorithm is better than the CascadeClassifier detector. How can I use Dnn deep learning algorithm (e.g. Dnn.readNet) on hololens2 ? Thank you very much!

 

Hi, just wanted to update that the issue has been resolved in 2023.1.0b19. Thanks!

 

The OpenCV Dnn module is not supported by the UWP platform of OpenCV C++, so there is no plan to be implemented by OpenCVforUnity for now.
https://github.com/opencv/opencv/blob/master/modules/dnn/CMakeLists.txt#L1-L3
https://github.com/opencv/opencv/issues/9177

 

Asset validation failed (90085)

When I run Distribute App from Xcode, I get an error and can't upload to App Store Connect.

Asset validation failed No architectures in the binary. Lipo failed to detect any architectures in the bundle executable. (ID: ****-****-****-****-****)

How can I resolve this error?
The development environment is as follows.

macOS Version : Monterey 12.6.3
Xcode Version : Version 14.2 (14C18)
Unity Version : 2019.3.2.f1
Framework used: OpenCVforUnity2.3.3

Thank you.

 

I'm trying to improve color correction on face swapper and face masking. It seems that the function has limited tolerance of light and color difference. Face color can easily go to infidelity turning green or purple or too bright or too dark. I've found that specifyHistogram in facechanger codes doing all the tricks but is there any way I can do with it to improve the color fidelity?

 

Older versions may cause problems.
Could you try the latest version of OpenCVforUnity?

 

Unfortunately, we could not find a better color correction algorithm. It has been some years since we created the code, though, so we may find a new way.