[RELEASED] OpenCV for Unity

Hi,why display such alarms:
Scene 'Texture2DToMatExample' couldn't be loaded because it has not been added to the build settings or the AssetBundle has not been loaded.
To add a scene to the build settings use the menu File->Build Settings...
UnityEngine.SceneManagement.SceneManager:LoadScene(String)
OpenCVForUnityExample.OpenCVForUnityExample:OnTexture2DToMatExampleButtonClick() (at Assets/OpenCVForUnity/Examples/OpenCVForUnityExample.cs:47)
UnityEngine.EventSystems.EventSystem:Update()

 

Hello. Can this plugin allow us to do the hand tracking like kinect one ? Thank you very much!

 

hi i want to ask, is webgl face mask compatible for mobile browser?

 

I received your email, but when I try to reply to your email address, I get an error and cannot reply. Could you please resend it to me using a different email address?

 

Could you add all of the “***.unity” in the “OpenCVForUnity/Examples” folder to [Build Settings] – [Scene In Build]?

 

For now, OpenPoseExample takes more than 2000 ms to estimate human pose. Perhaps real-time processing is difficult.

 

UnityWebGL does not support mobile browsers at this time.
https://docs.unity3d.com/2021.2/Documentation/Manual/webgl-browsercompatibility.html

 

Hi, is it possible to use VideoCapture to stream RTSP camera? I can use openCV in py or C++ to stream RTSP camera, but I failed by using openCVForUnity.
My code is as follows, when i change the object on videocapture from '0' to the rtsp link, it failed. So if it can work rtsp camera by videocapture, please teach me how to fix it. Thank you!

 

Hello, is it possible to use the package to run equivalent of numpy masked array operations, e.g.

zeros

,

ones

,

around

,

where

? Thanks!

 

To play streaming file, ffmpeg.dll is required.
1)Download "OpenCV for Windows Version 4.5.3"(https://sourceforge.net/projects/opencvlibrary/).
2)Copy "opencv_videoio_ffmpeg453_64.dll" to the root directory of the Project.

3)Edit User Variables.
https://stackoverflow.com/questions...reply-when-cv2-videocapture-rtsp-onvif-camera


I succeeded in playing this file.

Code (CSharp):

1.             //capture.open(Utils.getFilePath(VIDEO_FILENAME));
2.             capture.open("rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mov", Videoio.CAP_FFMPEG);

 

OpenCVForUnity is a clone of OpenCV Java, you are able to use the same API as OpenCV Java 4.5.3.
https://enoxsoftware.github.io/OpenCVForUnity/3.0.0/doc/html/index.html

In opencv, the "Mat.ones()" and "Mat.zeros()" methods exist, but the numpy-specific "around" and "where" methods do not.
If you want to perform complex operations on each element of Mat, you need to extract the data array (or use pointer access).
https://github.com/EnoxSoftware/Ope...mple/MatBasicProcessingExample.cs#L2086-L2164

 

Well, I know it can play file.But I prefer to ask to stream the rtsp "camera" because I need to use the live image by rtsp camera. Thank you~~

 

Hi EnoxSoftware.

What should I do to detect age and gender in the ResnetSSDFaceDetectionExample scene? I downloaded the necessary files. I will be happy if you help.

 

When I delivered the camera stream with VLC media player using the rtsp protocol, I was able to successfully receive the stream with OpenCVForUnity.


capture.open("rtsp://127.0.0.1:8554/", Videoio.CAP_FFMPEG);

 

@EnoxSoftware I saw you update the package again 3 days ago, for the issue described in https://forum.unity.com/threads/fai...tive-plugin-files-bc.1137496/#post-7381241are we waiting for an official fix now, or do you plan to use the workaround

For example, use the emar tool to build the archive
emar r NativePluginExample.a NativePluginExample.bc
​

in the OpenCV package?

With 2021.2.0b8 the warning is still

Code (CSharp):

1. emcc2: warning: EXTRA_EXPORTED_RUNTIME_METHODS is deprecated, please use EXPORTED_RUNTIME_METHODS instead [-Wdeprecated]
2. emcc2: error: /var/folders/qb/xtrq8wsn0lz_s_b08h7n_s2w0000gn/T/tmp_02jtg1kopencvforunity.a: File has a suffix of a static library ('.a',), but instead is an LLVM bitcode file! When linking LLVM bitcode files use .bc or .o.

 

Could you send me an email using the contact form below?
https://enoxsoftware.com/opencvforunity/contact/technical-inquiry/

 

Check out the hand tracking package in NatML.

 

hello I use your package I want to connect my webcam via a new VideoCatpure (0)

Code (CSharp):

1.                 videoCapture = new VideoCapture(0);
2.                 Debug.Log("State 1");
3.                 videoCapture.set(Videoio.CAP_PROP_CONVERT_RGB, 1);
4.                 Debug.Log("State 2");
5.                 videoCapture.set(Videoio.CAP_PROP_FRAME_WIDTH, requestedWidth);
6.                 Debug.Log("State 3");
7.                 videoCapture.set(Videoio.CAP_PROP_FRAME_HEIGHT, requestedHeight);
8.                 Debug.Log("State 4");
9.                 videoCapture.set(Videoio.CAP_PROP_FPS, (int)requestedFPS);

the problem is that the functions seem very slow to be executed, I tried to pass it through couroutine so that it does not block my scene while it is loading but it still blocks, do you have any idea?

 

I've tested the "Examples/MainModules/videoio/VideoCaptureCameraInputExample" included in the package on Windows Editor, and I haven't encountered any delay in the function. In my experience, the time it takes for the camera image to appear is about the same as when using Unity's WebCamTexture.

 

Hi. I have a question about Aruco.drawMarker. As I understand correctly, it draws aruco marker image into provided mat but I'm unable to get the correct result. Every draw attempt even with the same parameters gives me incorrect image with pixels spread all around the texture.



My thought that I was using incorrect mat and texture2d formats but I've tried several CvType and TextureFormat combinations with no luck.

Here is the sample code:

Code (CSharp):

1. Dictionary dict = Aruco.getPredefinedDictionary(Aruco.DICT_4X4_1000);
2. var markerId = 0;
3. var markerSize = 6;
4.  
5. using (Mat m = new Mat(markerSize, markerSize, CvType.CV_8UC4))
6. {
7.     Aruco.drawMarker(dict, markerId, 0, m);
8.  
9.     var t = new Texture2D(m.width(), m.height(), TextureFormat.RGBA32, false);
10.     t.filterMode = FilterMode.Point;
11.     t.Apply();
12.  
13.     Utils.matToTexture2D(m, t, false);
14. }

My task is to generate aruco marker image and prepare it for printing in runtime.

 

Good morning,
I am a student and my professor has purchased the OpenCV for Unity asset. I imported the asset to my computer and it works. The problem arises when I close the program and reopen it. I attach screenshots of the error messages.
By deleting the code that uses the asset it opens without any problems. Could you tell me what the problem is?
Thank you very much

 

You need to input the markerSize as the third argument of the Aruco.drawMarker method.
I have modified the code as follows.

Code (CSharp):

1.      
2.            Dictionary dict = Aruco.getPredefinedDictionary(Aruco.DICT_4X4_1000);
3.             var markerId = 0;
4.             var markerSize = 6;
5.  
6.             var texture = new Texture2D(markerSize, markerSize, TextureFormat.RGB24, false);
7.             texture.filterMode = FilterMode.Point;
8.             texture.Apply();
9.  
10.             using (Mat m = new Mat(markerSize, markerSize, CvType.CV_8UC4))
11.             {
12.                 Aruco.drawMarker(dict, markerId, markerSize, m);
13.  
14.                 Utils.matToTexture2D(m, texture);
15.             }
16.  

See also the examples in Examples/ContribModules/aruco/ArUcoExample/ArUcoCreateMarkerExample.

 

Hi, I am going to purchase this asset before that I was checking the free trial version of OpenCV for Unity in that I need some help :

Can anyone please share some of the references for the Openpose example using web camera.

Thank you.

 

Could you tell me the environment you tested?
OpenCV for Unity version :
Unity version :

 

OpenCV for Unity version : 2.4.5
Unity version : 2020.3.8f1

 

The same version worked fine in my environment. Do you have the same problem on other PCs?

 

Hi, I am working on an OpenPose example using the free trail version. In that, I am able to get the skeleton tracking for a single person using the web camera.

I need some help for the following scenario :

Can you please share a procedure or some of the references for the skeleton tracking for multiple persons and also how can I generate a 3d coordinates of body parts.

Thank you.

 

With my professor's PC it works. Is it possible that one license cannot be used (or be active) with two different computers?

 

There are no such restrictions.
Could you send me an email using the contact form below? I will send you the project I have set up.
https://enoxsoftware.com/opencvforunity/contact/technical-inquiry/

 

Hello, yes of course! Thanks for the availability!

 

Hello, @EnoxSoftware,

I am trying to build a project for WebGL platform. In my last test I tried to build a new project with OpenCV for Unity 2.4.5 on Unity 2019.1.0, but the build failed. My steps were the following:

1) Make a new empty project on Unity 2019.1.0
2) Import OpenCV for Unity 2.4.5 (all checked)
3) Make a development build for WebGL platform
4) The build fails resulting in the error below:

Failed running "C:\Program Files\Unity\Hub\Editor\2019.1.0f2\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten_Win\python\2.7.5.3_64bit\python.exe" -E "C:\Program Files\Unity\Hub\Editor\2019.1.0f2\Editor\Data\PlaybackEngines\WebGLSupport\BuildTools\Emscripten\emcc" @"C:\Users\User\Documents\Projects\New Unity Project\Assets\..\Temp\emcc_arguments.resp"

Any advice?

Thanks

 

@EnoxSoftware
I'd like to test the trial version but I'm not sure how to import it into my project. The zip file contains a ton of subfolders, do I just drag the parent folder into the project?

 

Hello, @EnoxSoftware,

I am using OpenCV for my Unity project in that I was working on the OpenPose. I am able to get the OpenPose for a single person. I need help for the OpenPose for multiple persons.

Can you please help me or share some of the references to get the OpenPose for multiple persons using Unity.

Thanks in advance.

 

Is there enough space left on the HDD? Build errors may occur if there is not enough space left on the HDD.

 

Could you unzip OpenCVForUnity2.4.5TrialVersion.zip and import OpenCVForUnity2.4.5TrialVersion.unitypackage into UnityEditor?

 

Thanks for the reply.
Yes, there is fair enough space left on the HDD.

 

Unfortunately, I don't have an OpenPose example that detects multiple persons.
The following page is a good reference for detecting multiple people.
https://learnopencv.com/multi-person-pose-estimation-in-opencv-using-openpose/

 

@EnoxSoftware

OK. I have figured it out.

The problem was that the OpenCVForUnity/Plugins/WebGL/2019.1/opencvforunity.bc file was not configured for the WebGL platform. Afterwards, you have to delete the one in the 2018.2 folder.

And that's all. Build succeeded.

 

I tried to use COLORMAP_HOT but it's not working. But if I try to use cvtColor then that piece of code is working.

This piece of code is not working with COLORMAP

Code (csharp):

1.  
2. if (webCamTexture.isPlaying && webCamTexture.didUpdateThisFrame)
3.             {
4.                 Utils.webCamTextureToMat(webCamTexture, rgbaMat);
5.                 croppedMat = new Mat(rgbaMat, cropRect);
6.                 Mat effectMat = croppedMat.clone();
7.                 Imgproc.applyColorMap(croppedMat, effectMat, Imgproc.COLORMAP_HOT);
8.                 Utils.matToTexture2D(effectMat, texture);
9.                 cameraFrame.texture = texture;
10.             }
11.  
12.  

If I use cvtColor this piece of code is working.

Code (csharp):

1.  
2. if (webCamTexture.isPlaying && webCamTexture.didUpdateThisFrame)
3.             {
4.                 Utils.webCamTextureToMat(webCamTexture, rgbaMat);
5.                 croppedMat = new Mat(rgbaMat, cropRect);
6.                 Mat effectMat = croppedMat.clone();
7.                 Imgproc.cvtColor(croppedMat, effectMat, Imgproc.COLOR_RGB2GRAY, 3);
8.                 Utils.matToTexture2D(effectMat, texture);
9.                 cameraFrame.texture = texture;
10.             }
11.  

Can anyone please suggest what I am doing wrong?

I tried all COLORMAP but non of them works.

 

Enclose the point where the error occurs in Utils.setDebugMode() method, and the error on the C++ side will be displayed on the console.

Code (CSharp):

1. Utils.setDebugMode (true);
2. Imgproc.applyColorMap(croppedMat, effectMat, Imgproc.COLORMAP_HOT);
3. Utils.setDebugMode (false);

Code (CSharp):

1. colormap.cpp:736: error: (-5:Bad argument) cv::ColorMap only supports source images of type CV_8UC1 or CV_8UC3 in function 'cv::colormap::ColorMap::eek:perator ()'

The error is probably due to the fact that the croppedMat type is 8UC4.

 

Hi, is aruco detection supported on Android ? It crashes on the mobile app Opencvforunityexample.

Edit: did some testing :
it crashes on my Redmi note 8 pro but not on my Lenovo tab M10.
Only Aruco WebcamTexture Example crash, but that's what I need.
The crashing device seems to use arm64 v8, and the other v7.

 

Could you tell me the environment you tested?
OpenCV for Unity version :
Unity version :

Also,
Does this crash also occur with ArUcoExample? If the crash only occurs with the ArucoWebcamTextureExample, using the latest version of OpenCVForUnity may solve the problem.

 

Thank you for the reply.

I was able to convert Mat to 8UC3 by using COLOR_BGRA2BGR

Code (csharp):

1.  
2. Mat dst = new Mat();
3.             Imgproc.cvtColor(_image, dst, Imgproc.COLOR_BGRA2BGR);
4.             Imgproc.applyColorMap(dst, dst, Imgproc.COLORMAP_CIVIDIS);
5.  
6.  

 

Hi, thx for the reply, I've not yet bought the asset, I wanted to see if it was working fine before, so I'm using the app from the playstore.
Looking at the app system info it's unity 2019.3.10f1 and opencvforunity 2.4.1.
ArUcoExample is working indeed, that's weird...

 

Hi @EnoxSoftware

How can I create this in opencvforunity?

Code (csharp):

1.  
2. cv::Mat kernel =
3.     (cv::Mat_<float>(3, 3)
4.         <<
5.         0.272, 0.534, 0.131,
6.         0.349, 0.686, 0.168,
7.         0.393, 0.769, 0.189);
8. cv::transform(input_img, output_img, kernel);
9.  

I can use Core.transform but how to create that kernel matrix?

 

Hey devs. Thank you for the amazing asset this has helped me so much on my project!

I'm currently having an issue getting the position transform of my ARGameObject to match it's real-space position relative to the camera. I have the camera position 0.4m away from the marker in "test setup" picture below, but the z value in the position transform is reading out as 0.21m. When I move the camera back 0.8m, it reads ~0.4m for the z value. This leads me to believe that some parameter is off by a factor of 2 or 0.5. Is there a resource that describes how the variables like: chArUcoBoradSquareLength, chArUcoBoradMarkerLength (ArUcoCameraCalibrationExamples.cs), markerLength (ArUcoWebCamTextureExample.cs) are measured off the charuco board and aruco markers. edit: The ARGameObject is positioned correctly visually over the aruco marker. It looks bigger because it's showing the top face of the cube, just to clarify.



The second and third pictures show how I am measuring them currently.





Am I doing something wrong or has anyone encountered this problem before? When I calibrate I am taking 40 pictures and they are taken at a variety of depths, positions, and rotations. Are there other parameters I am not setting that I didn't mention?

Thanks!

 

At least in my tests, I haven't seen any major errors. The error is within a few centimeters of the actual measured value.
In the ArUcoWebCamTextureExample scene, the "ARGameObject" is positioned relative to the "ARCamera" world coordinate position when the AR marker is detected.
Note that in our example, the initial Z-position of the "ARCamera" is 0.3.
To observe the detection values that are not affected by "ARCamera", look at the ARM matrix calculated in line 709 of ArUcoWebCamTextureExample.cs. (The value of ARM.m23 is position-Z)
As for the meaning of the values of chArUcoBoradSquareLength, chArUcoBoradMarkerLength, and markerLength, I think your measurement location is correct.

 

You can download the latest version of the demo app at the link below.
https://github.com/EnoxSoftware/OpenCVForUnity/releases/download/2.4.5/OpenCVForUnityExample.apk

 

Thanks, this version is working, I'm gonna buy it soon

 

Code (CSharp):

1.             Mat kernel = new Mat(3, 3, CvType.CV_32FC1);
2.             kernel.put(0, 0,
3.                 0.272, 0.534, 0.131,
4.                 0.349, 0.686, 0.168,
5.                 0.393, 0.769, 0.189);
6.  
7.             Mat output_img = new Mat();
8.  
9.             Core.transform(input_img, output_img, kernel);