[RELEASED] OpenCV for Unity

Ok, but how do I create a usable mask from an image?
In the segmentedword example, it uses the scenetext_segmented_word03_mask.png to extract the found words. But I think it is pretty vital how one can create one of those masks from a token image.

I also tried to create some of my own masks, but it feels pretty random if it works or not.

/Best regards
Daniel

 

Sure,

Windows: 10 Pro x64
Unity: 2017.1.1f1 (Personal)
OpenCV for Unity: I've tested on 2.1.3 and 1.2.3 both are the same result. Thank you in advance.

 

Currently, with reference to https://github.com/opencv/opencv_contrib/tree/master/modules/text/samples, I am investigating how to create a textline mask.
After trial and error i succeeded in creating a simple example, please try.
https://www.dropbox.com/s/xvrjjwm5si94209/TextRecognition.unitypackage?dl=0

screenshot

 

Is opencvforunity.dll set as follows?

“OpenCVForUnity/Plugins/x86_64/opencvforunity.dll” – Select platform Editor,Standalone and CPU x86_64 and OS Windows in Inspector.

 

Yes, also I set it for android(in other project), downloaded and copied opencv dlls to disk C, rebooted unity after the imported cvforunity plugins restarted computer tried for 2D and 3D project but no way..

 

Wow, it is almost working. I tried it on the sign jpg and it got most of it. I tried it on the Freephone private hire jpg and it saw nothing. most probibly to do with the font and colors.

Is tesseract doing this in opencv or is opencv doing it alone?
I heard tesseract is in version 3 but I see no tesseract action in playmaker.

 

I just tried this with a snippet jpg of a portion of a word doc and it worked very well.

Just how could the spaces be detected as it outputs without spaces between the words?

 

The recognition rate of the text image which is not black and white does not seem to be high.
In the case of such an image, it seems that it is necessary to perform additional image processing in order to create a mask image.

Also,Tesseract OCR is not supported. OpenCV is doing it alone.
https://enoxsoftware.com/opencvforunity/documentation/support-modules/

 

It seems that OpenCV's OCRDecoder cannot detect space.

 

Hi! When I'm using landscape webcam orientation - face detection works ok (I'm using facemask additional example without dlib - but it should be a general case).
When I'm rotating the webcam 90 degrees (for portrait orientation) - face detection stops working. What can I do about it? Thanks!

 

Is there any chroma key example for removing background with green screen? Thanks

 

Thanks for clearing that one up with OpenCV doing it all alone and Tesseract is not the worker.

I would however love some suggestions on how to learn OpenCV in Unity.

I have looked around a lot seeing Python, c++, Java and so on but what would be the best place for me to start and actually learn OpenCV to use in Unity?

Any books suggestions, Video tutorials or just good websites that will get me started in CV.

Thanks for any suggestions here.

 

Hello! Before purchase! I would like to know, can i create something like this?if yes which way do i dig?

 

hello..
can i combine ARHead & webcam texture facemask?
i already tried combining the script from example and it doesnt work.
thanks

 

Currently there is no such example.

 

Since this asset is a clone of OpenCV Java, you are able to use the same API as OpenCV Java.http://enoxsoftware.github.io/OpenCVForUnity/3.0.0/doc/html/index.html
If there is implementation example using "OpenCV Java", I think that it can be implemented even using "OpenCV for Unity".

The following page is helpful for me.
http://answers.opencv.org/questions/scope:all/sort:activity-desc/page:1/query:java/
https://github.com/tanaka0079/java/tree/master/opencv
http://opencv-java-tutorials.readthedocs.io/en/latest/

 

Ignore?

 

@EnoxSoftware
Hey still same.. Would you mind help me ? no? :/

 

Nice links, Thank you. Please also take a look at how we could get spaces between the recognised words of your example. That would be amazing.

 

Sorry for my late reply.

Could you setup again in the next procedure?

1. Create a new Unity project using latest Unity version.
2. Import "OpenCV for Unity version 2.2.1" Asset.
   
3. Select MenuItem[Tools/OpenCV for Unity/Set Plugin Import Settings].
4. Move the “OpenCVForUnity/StreamingAssets/” folder to the “Assets/” folder.
   
5. Run OpenCVForUnity\Examples\Basic\Texture2DToMatExample.unity.

 

I cannot judge whether it is possible or not.
Since this asset is a clone of OpenCV Java, you are able to use the same API as OpenCV Java.http://enoxsoftware.github.io/OpenCVForUnity/3.0.0/doc/html/index.html
If there is implementation example using "OpenCV Java", I think that it can be implemented even using "OpenCV for Unity".

 

Yes, doesn't work but I just changed a lot of lines of code in the source (Editor versions). Now it's working! Thank you so much for your support.

 

Hi! Is there any example on shadow removal? Thanks

 

how can i set up bundle identifier for android in the player settings?

 

Hi. how can i combine ARhead example & Facemask example?
please help

 

Firstly you have to learn "How can I search on google".

 

Currently there is no such example.

I think these pages will be helpful.
http://answers.opencv.org/question/...dow-of-a-object/?answer=121221#post-id-121221

http://opencv-python-tutroals.readt...ideo/py_bg_subtraction/py_bg_subtraction.html
Result of BackgroundSubtractorMOG2
Gray color region shows shadow region.
http://opencv-python-tutroals.readthedocs.io/en/latest/_images/resmog2.jpg

 

Please try this example.

https://www.dropbox.com/s/9eovslvwt5mmy1u/ARHeadWithFaceMaskExample.unitypackage?dl=0

 

Hi, I seem to have some trouble with setting the correct size of the texture. Any suggestion how to fix that?

ArgumentException: The output Mat object has to be of the same size
OpenCVForUnity.Utils.webCamTextureToMat (UnityEngine.WebCamTexture webCamTexture, OpenCVForUnity.Mat mat, UnityEngine.Color32[] bufferColors) (at Assets/OpenCVForUnity/org/opencv/unity/Utils.cs:673)
​

 

Hello again, I want to ask three question about marker detection and wrap perspective method. In the sample project we can detect markers in a texture(arUco). I little bit confused about position of the detected marker.I can't find it, the mat class doesn't give me any value of position. My goal is detect marker (aruco or not it doesn't matter) and find its rotation/position then wrap perspective&transform to marker's straight position. I know there is a sample for wrapperspective but no documentation for that. So, I can't understand what are the parameters in this method,

Code (CSharp):

1. src_mat.put (0, 0, 0.0, 0.0, inputMat.rows (), 0.0, 0.0, inputMat.cols (), inputMat.rows,inputMat.cols());
2. dst_mat.put (0, 0, 0.0, 0.0, inputMat.rows (), 200.0, 0.0, inputMat.cols (), inputMat.rows (), inputMat.cols () - 200.0);

Why we write these parameters and how it works ? Is it turning dst to src ? or src to dst ? How can I know the parameters for "source" image.

And last question is; Let's say I have a picture and I know its height width etc. It is source image, there are 4 marks on the picture. And I printed it, took its photo from different angles and I want to straight the photo what should be my steps, can you please explain?

Thank you for your time.

 

Could you tell me about your test environment?
OS :
Unity version :
OpenCV for Unity version :

 

Please refer to OpenCV official document(http://docs.opencv.org/3.3.0/index.html) for the details of the argument of the method.
http://docs.opencv.org/3.2.0/da/d54...sform.html#ga8c1ae0e3589a9d77fffc962c49b22043

Since this asset is a clone of OpenCV Java, you are able to use the same API as OpenCV Java.http://enoxsoftware.github.io/OpenCVForUnity/3.0.0/doc/html/index.html
If there is implementation example using "OpenCV Java", I think that it can be implemented even using "OpenCV for Unity".

http://nobotta.dazoo.ne.jp/blog/?p=479
http://docs.opencv.org/3.3.0/da/d6e/tutorial_py_geometric_transformations.html
http://bretahajek.com/2017/01/scanning-documents-photos-opencv/

 

Dam! Well that's it then. No OCR from sentences.

 

Thank you for your answer/suggestions. I examined all of website which you posted to me. I've learned a lot of things. But unfortunately I still did not find solution for marker's position/rotation. I can't convert corners(Mat object) to Point or Rectangle or Vector or something gives me x,y coordinates. This is a big problem for me. And I don't understand, really can't understand why everyone write these parameters manual like;

Code (CSharp):

1. img = cv2.imread('sudoku.png')
2. rows,cols,ch = img.shape
3. pts1 = np.float32([[56,65],[368,52],[28,387],[389,390]])
4. pts2 = np.float32([[0,0],[300,0],[0,300],[300,300]])

And this;

Code (CSharp):

1. float srcPoint[] = new float[]{52, 38, 18, 296, 474, 293, 446, 58};
2. Mat srcPointMat = new Mat(4,2,CvType.CV_32F);
3. srcPointMat.put(0, 0,srcPoint );
4. float dstPoint[] = new float[]{18, 38, 18, 296, 474, 296, 474, 38 };
5. Mat dstPointMat = new Mat(4,2,CvType.CV_32F);
6. dstPointMat.put(0, 0,dstPoint);

Yes, I can transform when I write parameters manually, trying to find the good result. But it is not good idea because I won't know what should be the parameters for taken every different photo, picture or whatever.

Or should I think that this library can not calculate these parameters automatically with findHomography or etc. ?

Actually questions are very simple, if I should be more clearly with my words;
1. How to detect arUco marker's position and rotation ?
2. How can I calculate the parameters of getperspective (with automarker's properties or not) ?

Finally, I don't want to write code for me, I just want to learn that how process these tasks in opencv for unity library but if you can show me a sample I really appreciate it.

Thank you for your time, I'm looking forward for your reply.

 

Hi,
I am adopting DLIB with face mask example. In addition, I added some face props attaching on players. After integration, the system speed is low if there is more than two players. I guess this plugin mainly runs at CPU. I just wonder if this plugin support GPU too. My project is running out of time and hope get your reply soon. Many thanks.

 

This example is helpful if you want to learn the procedure to detect markers.https://github.com/EnoxSoftware/MarkerBasedARExample
In particular, I think that this code will be helpful.
https://github.com/EnoxSoftware/Mar...asedARExample/MarkerBasedAR/MarkerDetector.cs

• Convert the image to grayscale
• Detect contours
• Find closed contours that can be approximated with 4 points
• Find is them are markers
• Calculate their poses

 

Unfortunately, Dlib's ObjectDetecter and ShapePredictor do not support GPU.

 

i try to combine facemask and facerecognition example but didnt work.
can you help me

 

I think that it is difficult to combine FaceMaskExample and FaceRecognitionExample.
You need to edit the script drastically.

 

Anyone try using this in 2017.2? I tried using the HoloLens example and it crashes the editor on any scene when I hit the play button.

 

Hi All,
I am new to OpenCV and new to the plugin.I wanted to know if there is a way to convert a point detected from an image to Vector3 in Unity.For example if I am using Hough Transform to detect a line from an image,I want to plot that line in Unity 3d world coordinates.I have attached an image for reference.

 

Could you tell me your test environment?
Unity version
HoloToolKit version
OpenCV for Unity version

 

Hey, I'm trying to average webcam images sequences into a single image.

Code (CSharp):

1. IEnumerator BackgroundAccumulation(){
2.         isBackgroundReady = false;
3.  
4.         bgAvgMat_raw = Mat.zeros(wt.height, wt.width, CvType.CV_64FC3);
5.         bgAvgMat_warped = new Mat(wt.height, wt.width, CvType.CV_8UC3);
6.         int i=0;
7.         while(i<numBGs){
8.             CaptureCurrentFrame();
9.             yield return new WaitForEndOfFrame();
10.             Imgproc.accumulate(inputMat, bgAvgMat_raw);
11.             i++;
12.         }
13.         //bgAvgMat_raw /= numBGs;    //HACK blue,green channels are missing!! only red channel shows
14.         bgAvgMat_raw.convertTo(bgAvgMat_raw, CvType.CV_8UC3);
15.  
16.         Imgproc.warpPerspective(bgAvgMat_raw, bgAvgMat_warped, perspMatrix, bgAvgMat_warped.size());
17.         Utils.matToTexture2D(bgAvgMat_raw, bgAvgTex_raw);
18.         Utils.matToTexture2D(bgAvgMat_warped, bgAvgTex_warped);
19.  
20.         isBackgroundReady = true;
21.     }

The `Imgproc.accumulate()` works fine, but the averaging part (bgAvgMat_raw /= numBGs messes the image.
After the division, my converted(to CV_8UC3) image becomes red image. The blue and green channels are missing.
If the division part is commented, the converted image shows the normal 3 channels of 'burnt'(over exposed which is correct without averaging) image.


Above is an image without the averaging. The image is 'over exposed'.


And above is an image with the averaging. Averaging works fine, but on the red channel only. The blue and green channels are missing.


I'm not sure why this is happening. If I've done anything wrong, please guide me to the right direction.

Thanks in advance!

 

https://forum.unity.com/threads/released-opencv-for-unity.277080/page-24#post-3136157

 

Please change from "bgAvgMat_raw /= numBGs;" to this code.

Code (CSharp):

1. Core.divide (bgAvgMat_raw, new Scalar (numBGs, numBGs, numBGs), bgAvgMat_raw);

https://enoxsoftware.com/opencvforunity/way-to-translation-of-mat-class-operators-defined-in-cpp/
bgAvgMat_raw /= numBGs;
This line internally calls Core.divide (bgAvgMat_raw, new Scalar (numBGs), bgAvgMat_raw);

 

wow, thank you!! That was tricky though, not intuitive. I’ll try and see if it helps.

 

Hello,

While using FaceMaskExample, How can I set Web Camera to Portrait mode (1080x1920). I did changed unity resolution from player setting, but Web camera is still showing output in landscape.

 

Which platform did you test?

 

Hello, I am pretty new to OpenCV and image processing. However, I am trying to recognize Alphabets from real time video, for example webcamTexture. Can I do it with OpenCV? I used vuforia, but it cant detect single letters. Can you give me a direction? Thanks

 

Windows 10