OpenCVSharp for Unity

Expect a small update in 2-3 days, mostly fool-proofing for demo scenes and a WebCamTexture bug fix.

Changes for 1.4.4:

• Address the Unity bug with WebCamTexture mentioned at the top of this page (where the resolution of camera is falsely reported as 16x16 before camera returns the first frame) and will foolproof the asset to some extent since I do get a lot of mail and reviews about some things.
• User will be warned to check README if StreamingAssets folder was not copied to root.
• Cameras have been added to all scenes to prevent questions and bug reports about "No cameras rendering" (and there have been quite a few and even one bad review because of this). This is not a bug and if the scene is composed solely of screen space canvas camera is not needed. Message can be disabled through right clicking the "Game" tab and un-checking the option.
• Added warning when no physical camera devices are plugged in / present on the device.
• Added more exclamation marks and underscores to the RADME.txt (not sure this one is important enough to be on the list but hopefully it makes somebody actually read it )

 

Why not support webgl?

 

How does this compare to OpenCVForUnity?

Do you have any benchmarks?

 

Hi @NWHCoding ,

does this package has facemark lbf support? i need to get face landmark on my application

Thank..

 

Yes it does. Check namespace OpenCvSharp.Face.

 

I do not have a direct comparison but I can guarantee that OpenCVSharp for Unity has near best performance you can get. That is because the wrapper (which is in C#) is calling native, vanilla OpenCV C++ functions which means that only performance overhead you have are those calls. Here is an example of calling Cv2.Threshold() function on FHD image:



As you can see - total execution time was 0.25ms of which 0.22ms was actual native OpenCV doing its things, while the rest (0.03ms) is the overhead. So add about 10% to the native execution time of OpenCV function (as if you were running a C++ script) and you get the performance.

The other important thing is conversion between formats (Unity <-> OpenCV) and here the asset is quite fast too. All the conversions are done through memory copy and therefore there are no iterative methods (no for loops, etc.).

Hope this answers your question somewhat.

 

Developing for WebGL - which uses totally different native function calls from all other major platforms - would take a lot of time and I do not see many people running OpenCV in WebGL. I could be wrong. I approximated that porting it over would take a few weeks most likely and a lot of features of Unity are not available there so I decided to not support it. It would also slow down update times as each update is essentially a complete rebuild of all native libraries.

 

Do you have similar package support as OpenCVForUnity - or at least the common ones, such as face masks and orienting a unity game object to a face anchor, etc.

 

I am not familiar with package support from OpenCVForUnity so I can not comment on that. This asset includes all the features of OpenCvSharp plus the conversion functions as listed on the asset page. Also, the examples from the demo scene are all included. You can use any OpenCvSharp example found on the web too.

 

I'm a little confused - is "OpenCVSharp for Unity" a wrapper around OpenCV? Or a wrapper/set of extension methods around OpenCvSharp (which, in itself, is a wrapper around OpenCV)?

I've already tried using OpenCVSharp, but the performance for any non-trivial functionality was significantly worse than when implementing the same functions directly in OpenCV. To then implement an additional set of conversions/dependencies to use these from within Unity, I think would result in unacceptable performance to me, so I've started just implementing my own wrapper methods directly calling into the OpenCV native library. I'm very pleased with the performance, but it will take me some time to implement all the functions I want - can you clarify whether your asset would help at all, or whether that would just reduce me to OpenCvSharp?

 

Hello,
as the title implies this is OpenCvSharp made to work with Unity on multiple platforms. Also, it includes very fast common conversion functions between Unity and OpenCV formats (most importantly Mat and Texture2D). Those conversions use memory copy and therefore are as fast as it gets (except for converting to Texture2D natively from your code but that can be hairy if you want to support multiple platforms).
Of course that running everything on the C++ side will be faster, there is no question about it. But that also requires you to rebuild each time you make a change and also it will not work with Win, Mac, Linux, Android and iOS without writing a lot of platform-specific code. As you might know webcam capture from OpenCV does not work properly on mobile platforms in most cases.
The overhead is question is quite small - check the post #106 - for the flexibility you gain. Of course, there are different kinds of requirements for different projects so in the end it all boils down to what you need.

 

Thanks for the clarification. I have no interest in building for mobile - I'm only targetting Win32/64, and it only takes a handful of lines of code for me to implement Marshal.Copy for the interfaces between OpenCV# and OpenCV structures (Webcam -> Mat, Mat -> Texture2D etc.), so it sounds like I'll stick with my current approach. Cheers.

 

hi @NWHCoding

I bought you your asset openCV for unity. Since years now I use unity for a variety of apps with c#. But I'm new to OpenCV and after 2 days of trial and error I've to say, that's it's really hard to get into it. I seems, that every code uses different syntax, conversions, etc. - hard to learn from examples, which don't work or they just don't fit your needs. OpenCV has so many possibilities.

My target would be a document scanner, like this: Youtube / Github

Besides many other problems, currently I'm stucked at the line:

Code (CSharp):

1. Mat srcPointsMat = Converters.vector_Point_to_Mat(approxPt.toList(), CvType.CV_32F);

I could not find the converter "vector_Point_to_Mat" in your openCV integration. Or is there another approach? Would be glad, if you could help.

Maybe this "simple" document scanner example would be also interesting for your asset.

 

Here is my result for today.

...after trying to get a better idea from all these mat/matOfPoints/PointArrays/inputArrays/outputArrays/multidemsional/whatever-Arrays!
For developers, who normally care about vector3, particles, animations, tweens, loading, etc... this is kind of alien magic!

My testimage.jpg (loaded from streaming asset folder) and the script are attached. It works in a clean project with just OpenCV# + canvas with 2 raw images. The rectangle around the note is recognized correct.

Now it would be great, if some more experienced people could check, if it's done to complicated or inefficient.

two other things have to be improved. help would be great.
1. line 60: there should be a better way, to be sure, if this is a rectangle. some calculations about the angles
2. line 80: the texture inside the found contour should be croped, warped and correct aspect caluclated for usage as a texture.

 

Hello. I am glad you got it working. Now, to clear things up: OpenCV# for Unity is, as the name suggests, a port/build of OpenCvSharp by Shimat made to work out of the box with Unity and multiple build platforms. Besides that, it is also a collection of fast conversion functions between Unity and OpenCvSharp. So essentially customers are buying this asset to save themselves time and hair pulling needed to get the mentioned done.
Now on to the syntax: OpenCvSharp is a wrapper and as things go C++ and C# are not language compatible. C++ uses pointers amongst other things, C# does not. List goes on. Therefore the syntax can not be the same. OpenCvSharp and Emgu CV (another OpenCV C# wrapper) have fairly similar syntax and there are quite a lot of examples around the net for each of those so in most cases you can find something that at least to some extent matches your use. I have included a few, but many more are available such as these two sites:

• https://github.com/VahidN/OpenCVSharp-Samples
• https://csharp.hotexamples.com/examples/OpenCvSharp/Mat/-/php-mat-class-examples.html

Hope this clears things up and helps some. Good luck with your project.

 

I still have no idea how to get the Mat from my list of points... like as in my first post described, many example (like here) use a conversion from point like: Converters.vector_Point_to_Mat(approxPt.toList(), CvType.CV_32F). And I also find this method in some docs (Here), but I find no way to use it with your plugin. Could you clearify this?

 

I'm investigating this library for doing fast blob detection on mobile. But for initial evaluation, I downloaded the MacOS sample project from the asset store page to see how well that works.

But either it doesn't work, or I don't understand how to use it. As soon as I select any test, the camera light comes on, and the window is filled with a giant white box:


My face is definitely in view of the camera here, but as you can see, nothing but white. The other tests (e.g. blob detection) produce the same thing (though I'm not sure what color blobs it might be looking for anyway).

Am I doing something wrong, or is it really not working?

 

I have tested the asset with MacOS and a webcam before publishing, but I will run the demo again and check if the demo might be corrupt in some way. Does it say "OpenCV loaded" on the menu screen (bottom left)? To me it looks like the demo might be the problem and not the OpenCV itself since it would most likely crash immediately if OpenCV was the culprit.

 

Yes, it did say OpenCV loaded before I selected a demo.

 

Yes, I am just waiting for it to mature a bit since there are 126 open issues for the wrapper at the moment on git. It takes quite a long time (a few days) to put together an update, and for a single bug a complete rebuild is required - and with the current number of open bugs that is an issue.

 

Really good product and appreciate it a lot Thanks for all the hard work.

A question; Although there are many other sources as you've suggested before as well, do you have any pointers, example scenes or demos for image/panorama stitching to use with your plugin? Or does the asset include an example for that as well? Did not purchase it yet but definitely will do for a certain project.

 

Is it possible to run a tensorflow model with this plugin? I read some comments saying they try to use this plugin with tensorflow. However i wonder how that would be possible (and if its possible on mobile). Can you give me some information?

 

opencvforunity has been upgraded to opencv 4!

 

How to load pretrained CNN model (bvlc_googlenet.caffemodel)on android cell phone?
when i use "readNetFromCaffe(modelTxt, modelBin)"to load model,it is ok for windows,but it can't find when i run this program on android cell phone.

 

Dear developer, before purchasing the asset could you please help me to understand
will it be possible to add an "OpenPose deep learning library" to the project (https://github.com/CMU-Perceptual-Computing-Lab/openpose)
and then get something like this on ios device https://github.com/sarweshshah/gait_analysis/blob/master/results/pose trail.gif
Thank you in advance

 

This is not directly an answer to your question however you can also use arkit if you work with ios. Since arkit 3 there is realtime pose estimation build in, optimized for the hardware. You probably wont get good performance with opencv only.

Demo of body tracking with arkit

 

Hi, I've just bought the OpenCVSharp for Unity from the Asset store and have it imported into an example project. However, there is this compiler error
Assets/Scripts/newscr.cs(13,30): error CS0117: 'Net' does not contain a definition for 'ReadNetFromONNX' that is not going away.

I am using on a MacBook Pro with OS 10.15.1, Unity3D 2018.4.6f1.

Please help!

Sample CS script:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using OpenCvSharp;
5. using OpenCvSharp.Dnn;
6.  
7. public class newscr : MonoBehaviour
8. {
9.     private Net Onnx;
10.     // Start is called before the first frame update
11.     void Start()
12.     {
13.         Onnx = OpenCvSharp.Dnn.Net.ReadNetFromONNX(Application.dataPath + @"/MobileNet3D2.onnx");
14.     }
15.  
16.     // Update is called once per frame
17.     void Update()
18.     {
19.        
20.     }
21. }
22.  

 

The website and documentation links you have are all now 404's, is this no longer supported?

 

Seems like this project is now dead. No replies from author and no update for opencv 4 which has been out for over a year.

 

Sorry about that. I will be removing the asset from the asset store as the interest has been fairly low from the start and I have been focusing on other assets. Update to OpenCV4 might happen in the future with an upgrade path from the current version but not in the next few months.

 

Hi NWH Coding... I want to download this package for opencv# for unity, but the link is not found or not available. please send me a work link. :')

 

Hello. The package has been pulled from sale for the time being.
If you have already purchased it previously you can download in from within your Unity Editor (Asset Store tab > My Assets).

 

I dont have it. So can you give me a link for download it. The package same like this thread. :')

 

@edwinerdiyana

^ see this is an important part

but 'in a few months' means more likely 'never' than not, so don't count on it

 

How can I use Texture2DToMat function?
I tried to write codes like below:
texture2DToMat(texture, frame);
Texture2DToMat(texture, frame);
OpenCvSharp.Texture2DToMat(texture, frame);
OpenCvSharp.CvConvert.Texture2DToMat(texture, frame);
All of them does not work well. Any Idea for this? Thank you!

 

Hello, dear colleagues and developers!

I amaze by the great asset OpenCvSharp in general, but it does not work correctly in some cases. For example, it cannot open image via Mat image = new Mat( "imagename.png" ) correctly - I receive always empty image matrix.

But more important for me, that the asset cannot stitch images using Stitching module - Unity crashes when I try to stitch any two images every time.

Please can you help me, how can I overcome that issues?

My best regards!

 

Hello,
the asset has been deprecated about two years ago and the last version uses now outdated OpenCVSharp version.
OpenCVSharp now has builds for most platforms so I would suggest downloading it directly from the repo here: https://github.com/shimat/opencvsharp. You might still be able to use the provided Unity utilities from my asset with the newer version.

Regards,
Aron

 

Hello everyone.
I have an image taken from user's webcam and since quality can be more or less good (due also to room lighting), I'd like to improve it (I'm using that image to create a texture for UMA avatar). So is there a way to do an auto image enhance with OpenCVSharp?

 

OpenCVSharp for Unity has since been deprecated due to the official multiplatform support: https://github.com/shimat/opencvsharp

 

So the official OpenCVSharp supports compilation and usage on Android platform?
For quite some time it wasn't the case.

 

In our case, we moved over to this version quite a while ago though, due to the issues in this project:
https://assetstore.unity.com/packages/tools/integration/opencv-for-unity-21088

I do want to add that the git link the author provided here doesn't help with Unity or Android support. It is just a general C# implementation of OpenCV that you would then need to go through the process of getting into Unity, converting over code, creating Unity objects for everything, etc.

 

Author here. The OpenCV# and the asset above are different. The asset OpenCVSharp for Unity was based on OpenCvSharp (git link above) and provided builds for different platforms and utilities for converting between data formats. At the time I made the asset OpenCvSharp only supported Windows. At this point it supports most platforms (except mobile) and as such I decided to stop selling OpenCvSharp for Unity. Anyone who purchased the asset can still get it from the Package Manager and all the conversion utilities are still valid.

 

hey @Lloyd_RedironLabs
there is a fork of official opencvplus unity asset that went opensource from now on, there is a slight chance that somebody can port the opencvsharp 4 to it.
https://github.com/CollegiumXR/OpenCV-plus-Unity

 

I am the developer actually and thank you for your feedback. I have been on the store for about 5 years now and really try to refrain from this but the situation with OpenCV# for Unity was a bit specific. Still, if anyone needs the utility scripts that were included with the asset I can PM them for free.
The main selling point of the asset was providing the builds of OpenCvSharp for different platforms and at this point all the platforms that shipped with OpenCV# for Unity are available from the git, except Android and iOS. That said with a bit of cross compilation these two can be compiled quite easily too, especially Android one. So at this point I really did not feel that it was justified to continue charging customers for something that was not an issue any more.

 

Not intending to come across harsh, but be that as it may be... the end result is people gave you money for a product that you stopped supporting shortly after - and left them to use a less functional alternative that someone else open sourced for free.

That said, I will drop the topic. I do want to give you credit for maintaining communication here and the thread. Most fly by night assets don't do that. And it does show promise for temporary project needs.