[RELEASED] OpenCV for Unity

Hey there,
I'm having a problem similar to some, but I'm not seeing a good fix for it. I'll post some code, and a screenshot of the errors, and I hope someone can give me some insight. I am getting a CvExcpetion: Native object address is NULL
I assumed this was for the image(a Mat object) the error does not happen with each iteration(the code is in a loop).
The Mat which is image is always initialized, and you can see that the beginning if statement never executes from the picture of the log. Any ideas?
if (image.getNativeObjAddr() == null)
{
Debug.Log("image was null");
}
Debug.Log("1");
Mat smat = new Mat();
smat = image.submat(y, y + intAggregateHeight, x, x + intAggregateWidth);
Debug.Log("2");

if (smat == null)
{
Debug.Log("smat is null");
}
Scalar avg = Core.mean(smat);
Debug.Log("3");

 

strange behavior?
Could you take a screenshot?

 

Unfortunately, There is no such function now.

 

Hello, Im upgrade to lastest version, and now, the VideoWriter's open method always return false on my pc, it return true only when i build to android( though I'm not yet test on IOS). Is this is unity error or my wrong. This code snippet is here :

Code (CSharp):

1. writer = new VideoWriter();
2. bool a = writer.open(filePath,0, codec, FPS, size);
3. Debug.LogFormat("is writer open ?? {0}", a);     // -> is writer open ?? false
4. Debug.LogFormat("is writer open ? {0}", writer.isOpened());  // is writer open ?? false
5.  

 

In order to display the native opencv's error code, please enclose the code in Utils.setDebugMode(true) and Utils.setDebugMode(false).
Errors are displayed in the console?

Code (CSharp):

1. Utils.setDebugMode(true);
2.  
3. ------------
4.  
5. Utils.setDebugMode(false);

 

I should also point out that this code was directly transferred to to Unity from an Android application using OpenCV, and it works fine there.
With debug mode on the error looks like:

 

I already saw that video and script, but it didn't help. We can't understand th way it works the code, and we couldn't get working the multi object tracking with CamShift. Sorry to bother you, but I don't know how to make that.

 

Hello,

While using WebCamTextureFaceMaskExample,
how can I change mask when a user exits (or when there is no active detection/no user) and new user comes ?

 

The argument of submat() method seems to be incorrect.

image.submat(y, y + intAggregateHeight, x, x + intAggregateWidth);

0 <= y
y <= y + intAggregateHeight
y + intAggregateHeight <= image.rows()

 

VideoWriterExample work fine?
https://github.com/EnoxSoftware/Ope...WriterExample/VideoWriterExample.cs#L185-L186
I have tested VideoWriter.fourcc ('M', 'J', 'P', 'G') on all platforms.

 

Hello, where can I find the example and explanation to use with Tensorflow - DNN to recognize objects?

Thanks.

 

I tested that yesterday. You need to load the .pb and .txt file from the zip whose link is shown in Debug-log when you try to run the scene.

But as to using TensorFlow 1.3.0 to re-train inception-model and then use it in the plugin... I got the Unity Editor (5.6.3f1) to crash when I tried that. The model could be tested in TensorFlow without problems. So can anybody answer what version of TensorFlow trained models are compatible with Open CV ?

 

Apparently there are several different versions of the Inception model. Which one does the example-code use ? Maybe I trained with too new Inception model and trying to load that crashed the Unity Editor.

 

Hi! Is there any example how to remove background from webcam feed when the user is standing in front of green screen? Thanks

 

Could you send me the code to reproduce the problem?
store@enoxsoftware.com

 

Hey this is probably a dumb question but I'm working with the marker based webcam AR example, but building for Android, and the web cam texture is not fullscreen (it's like a movie where the quad for the camera texture has bars on the top and bottom since it's a perfect square), so how does one get fullscreen video on the Quad that displays the camera?

 

OpenCVForUnity2.2.1 is based on this commit.
https://github.com/opencv/opencv/tree/3.3.0/modules/dnn/misc/tensorflow
https://github.com/opencv/opencv/issues?utf8=✓&q=is:issue tensorflow layer is:open
It may not correspond to the latest version of tensorflow.

 

Thank you, i figure out my mistake, the path of the video contain accent character like "hình ảnh". If i change the path to contain non-accent only the VideoWriter work fine.

 

Hi,
I bought your asset and I am having some real problems to understand how to get what text has been discovered???
I mean I get the text module running fine and it shows that there IS text on an image but I can't for the life of me understand how to get what text has actually been found?

Also, is there a way to scan a texture instead of a saved image? Would be so much better if I didn't have to save the image each time I run the text scan?

/Best regards
Daniel

 

I also bought this today and it is all looking very interesting. I however have the same question as Daniel above. How to get the strings from recognized text and I bought this one because of Playmaker, so please also show a text example using PlayMaker.

 

I can help you with creating a playmaker action if you need it.

 

That would be very helpful thank you.

 

After importing the playmaker actions for OpenCV my action browser is extremely slow and hardly scrolls. Searching takes ages, scrolling through the actions is painful and typing in the search bar takes 10’s of seconds. This is not acceptable and would be impossible to use PlayMaker with OpenCV.

 

https://forum.unity.com/threads/released-opencv-for-unity.277080/page-23#post-3086194

 

So, anyone on this forum know how to just get the returned results from the text example? The very basic text example.
It's not looking good on this forum as answers on the simplest stuff is just far and wide not here.
It's getting very frustrating now.

 

You can use the value of landmarkPoints.Count to judge whether a person is detected.
if (landmarkPoints.Count <= 0) {
Debug.Log("no user");
}
Also,
You can change the face mask by calling OnChangeFaceMaskButtonClick ().

 

Please try GreenScreenExample.
https://github.com/EnoxSoftware/Ope...nced/GreenScreenExample/GreenScreenExample.cs

 

let us try this question again as it seams to be getting overlooked.
Anyone on this forum know how to just get the returned results from the text example? The very basic text example.I want the Text, that is the whole point of Text recognitions is it not?

 

Sorry for my late reply.
Currently, only TextDetectionExample is included with OpenCVForUnity.
https://github.com/EnoxSoftware/Ope...ules/text/TextExample/TextDetectionExample.cs

But, I plan to add text recognition examples in future updates.
This code is translated from https://github.com/opencv/opencv_co...s/text/samples/segmented_word_recognition.cpp .

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using System.Collections.Generic;
4.  
5. #if UNITY_5_3 || UNITY_5_3_OR_NEWER
6. using UnityEngine.SceneManagement;
7. #endif
8. using OpenCVForUnity;
9.  
10. namespace OpenCVForUnityExample
11. {
12.     /// <summary>
13.     /// https://github.com/opencv/opencv_co...s/text/samples/segmented_word_recognition.cpp
14.     /// </summary>
15.     public class SegmentedWordRecognitionExample : MonoBehaviour
16.     {
17.         // Use this for initialization
18.         void Start ()
19.         {
20.             //if true, The error log of the Native side OpenCV will be displayed on the Unity Editor Console.
21.             Utils.setDebugMode (true);
22.  
23.             Mat image = Imgcodecs.imread (Utils.getFilePath ("text/scenetext_segmented_word03.jpg"));
24.             Mat mask = Imgcodecs.imread (Utils.getFilePath ("text/scenetext_segmented_word03_mask.png"), Imgcodecs.IMREAD_GRAYSCALE);
25.          
26.  
27.             // character recognition vocabulary
28.             string voc = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
29.             // Emission probabilities for the HMM language model (identity matrix by default)
30.             Mat emissionProbabilities = Mat.eye (voc.Length, voc.Length, CvType.CV_64FC1);
31.             // Bigram transition probabilities for the HMM language model
32.             Mat transitionProbabilities;
33.  
34.             List<string> lexicon = new List<string> ();
35.             lexicon.Add ("Private");
36.             lexicon.Add ("Hire");
37.             transitionProbabilities = Text.createOCRHMMTransitionsTable (voc, lexicon);
38.  
39.  
40.            
41.             OCRHMMDecoder ocrNM = OCRHMMDecoder.create (
42.                                       Utils.getFilePath ("text/OCRHMM_knn_model_data.xml.gz"),
43.                                       voc, transitionProbabilities, emissionProbabilities);
44.  
45.  
46.             string output = ocrNM.run (mask, 0);
47.             Debug.Log ("ocrNM " + output);
48.  
49.  
50.             Texture2D texture = new Texture2D (image.cols (), image.rows (), TextureFormat.RGBA32, false);
51.  
52.             Utils.matToTexture2D (image, texture);
53.  
54.             gameObject.GetComponent<Renderer> ().material.mainTexture = texture;
55.  
56.  
57.             Utils.setDebugMode (false);
58.         }
59.    
60.         // Update is called once per frame
61.         void Update ()
62.         {
63.  
64.         }
65.  
66.         /// <summary>
67.         /// Raises the back button click event.
68.         /// </summary>
69.         public void OnBackButtonClick ()
70.         {
71.             #if UNITY_5_3 || UNITY_5_3_OR_NEWER
72.             SceneManager.LoadScene ("OpenCVForUnityExample");
73.             #else
74.             Application.LoadLevel ("OpenCVForUnityExample");
75.             #endif
76.         }
77.     }
78. }

 

Could you tell me about your test environment?
OpenCV for Unity version :
PlayMaker version :

 

Thank you.

Version from all assets are all the very newest.
in the project is only PlayMaker , OpenCV and OpenCV actions.

 

No worries.
I got the example project to work after some hiccups.
But in any case, do you plan to make this kind of example project with your asset?
The reason I am asking this is the code you provide in the example uses lot more work with channels etc, while the example code that finds the text has lot less code.

Also, I asked before if there was a way to get your asset to read from say a texture that streams from the camera? To prevent the app from saving an image that is just going to be thrown away 100s of times a day?

 

Thank you very much for reporting.
Could you provide additional information?
OS :
Unity version :

 

Camera Stream means WebCamTexture?
I think it is possible to combine WebCamTextureToMatHelperExample and text recognition.
I plan to add text recognition examples(based on https://github.com/opencv/opencv_contrib/blob/master/modules/text/samples/webcam_demo.cpp) in future updates.

 

Playmaker Webcam Face detection

Working with

Windows 10 pro 64Bit

Unity 2017.1.0f3

PM 1.8.5

OpenCV 2.2.1

OpenCV actions 1.0.1

Playmaker action Browser working here fine only a bit sluggish when you open the whole OpenCV actions list. It works however.



Here not working and killing Unity

Newer versions of Unity or Playmaker breaks something as Unity crashes all the time when trying the Playmaker webcam face detect demo

Can you possibly separate the actions out a little more in the action browser, as everything is under opencvForPlaymaker and it is a lot of stuff? That is most probably the cause of the action browser hanging or being extremely sluggish.

 

Could you please explain how you got the script working in the text example?
So where did you get the scenetext_segmented_word03 from?

 

Bah I accidentally used my other login to post this reply, its still "ch0arrim"

I googled the filename and found that it exists a github project with those files being used.
https://github.com/opencv/opencv_contrib/tree/master/modules/text/samples
I downloaded all 3 files being listed in his example code above and put them in StreamingAssets/text/ folder
Try with that first and holler if you need more help

 

Thanks, I did that added the SegmentedWordRecognitionExample.cs from further above here and added the 3 files to streaming assets/Text. I then added SegmentedWordRecognitionExample.cd to the cube in the text example and removed the
TextDetectionExample.cs example script.
It just crashes Unity Imediatly.
If I just use TextDetectionExample.cs I get the marked sign Post.
Any ideas?
Do you actually get text now to the Console from just that script?

 

Hi again, sorry for asking so much but it feels I am soo close to succeed.

I took the example code above and pasted it to WebCamTextureToMatExample making my start method look like this:

Code (csharp):

1.  
2. string voc = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
3.  
4.            
5.             // Emission probabilities for the HMM language model (identity matrix by default)
6.             Mat emissionProbabilities = Mat.eye(voc.Length, voc.Length, CvType.CV_64FC1);
7.             //Mat emissionProbabilities = Mat.eye(voc.Length, voc.Length, CvType.CV_8UC4);
8.             // Bigram transition probabilities for the HMM language model
9.             Mat transitionProbabilities;
10.  
11.             List<string> lexicon = new List<string>();
12.             lexicon.Add("Enklare");
13.             lexicon.Add("jag");
14.             transitionProbabilities = Text.createOCRHMMTransitionsTable(voc, lexicon);
15.  
16.  
17.  
18.             ocrNM = OCRHMMDecoder.create(
19. Utils.getFilePath("text/OCRHMM_knn_model_data.xml.gz"),
20. voc, transitionProbabilities, emissionProbabilities);
21.  

Then I call the method at regular intervals trying to find the text.
Now I can get it to work with the scenetext_segmented_word03 images and I guess it because the scenetext_segmented_word03_mask is a sort of inverted image with the text being white and the rest is black.

But I still can't get it to work with a stream image.
What do I need to do to make the mask?
Or is it possible to use your TextDetectionExample to instead of pointing out that there is text here and there, it gives the characters found?
I would deeply appreciate if you could help me with this! Its so frustrating that I am sooo close to finally understand how to do this!

/Best regards
Daniel

 

Hello developper, please post a working example of Text Recognition and not just detection.
I bought this asset for exactly seeing how this is done. You advertise as such doing also recognition. I do not want to spend my whole time just trying to build it myself. This is why we buy assets to get a good head start.

I know you are trying to help, but if you look all people want to do is now get the text either from a pic or stream. Just please add a working example of this as soon as you can and don't forget a Playmaker example as that was the reason I bought this asset and not the other.
Thank you in advance

 

TensorFlowWebCamTextureExample also not working either. I have to download extra stuff to get a scene of yours working.

Why not include the files in the package? I expect scenes to work or where is the point?

Please in the future make a doc actually explaining how to use your examples and stuff we should have to download to get them working.

If I sound irritated, then yes it is very irritating that nothing works out of the box and I have to guess what is going on and spending my time in Google and forums.

 

This is what I would like to do.

1. Have a camera snapshot a street sign as the pic in your demo.(I can do that in Playmaker or Bolt already using natcam)

2. Then detect where the text is. (This is working)

3. Then recognize the text and set it to a string (Not included in your asset).

4. Then I can do with that string as I like.


I use Playmaker and know basics up to intermediate of c#. This is what I want to do and should be easy and included. This is the 21st century and not the 19th or 20th for that matter.

I should not have to study the entire Assembly to just have this function working.


If I want to be able to drive a car, I do not wish to know first how I have to build one. The basic steps of car driving would get me to a lot of places.

That is why we pay for learning to drive, to make things easier.


With this asset, as it is at the moment. I feel as if I would have to first build the car and then get somebody else to drive it.

 

eagleeyez, I understand that you are frustrated not being able to proceed with your project, I am too. But I don't think it's fair that you post all these opinions about what should be included etc.
We don't even know if this is their full time job. They might be working on this in their spare time, have family's and other responsibilities.
I agree that it should come with some sort of text recognition example and I believe if we are patient they will help us with this since it seems as if they are pretty active in answering people's questions. But I don't think you will get your answers faster by posting that

When they can assist us with the code to get a text recognition working, I can help you create a simple playmaker action for it.

/Best regards
Daniel

 

You are right, but this is going to be a game changer for OCR in Unity. There are a lot of people wanting this without having to learn Python, C++, Java, Tesseract and so on just to get some text recognized and also use in Android. I was just disappointed as, as it reads on their website you get detection and recognition.

I know, I have to be a little more patient but I just wanted to have that bit done and get on with the rest. So I will be looking forward to an update with recognition examples.

 

Could you try new version?
OpenCVForUnityPlayMakerActions1.0.2
Fixed the scroll delay of the Action Browser.
https://www.dropbox.com/s/up7t1gslsv1ogfw/OpenCVForUnityPlayMakerActions1.0.2.unitypackage?dl=0

Is lbpcascade_frontalface.xml located in "Assets\StreamingAssets\"?

 

This unitypackage contains all the files.Please import SegmentedWordRecognitionExample.unitypackage.
https://www.dropbox.com/s/kwzduuhq5fn46me/SegmentedWordRecognitionExample.unitypackage?dl=0

 

Thank you, that fixed it in Playmaker, now searching and scrolling the action browser is smooth. And you sectioned it out a bit in the action browser, much better.

Yes I have all in streaming assets. I will have to have another look at that.

And thanks for the segmented text package. I will take a look at that now and get back to you.

 

Thanks for this, but is it actually recognizing text? OCR

As the Log gives out the words Private Hire which is in the picture in the middle but in the script you are also adding the words Private and Hire to a lexicon.

I was expecting it to get the text out of the picture and DebugLog the following

Customer Freeplhone Private Hiere.

Maybe I am doing something wrong.
Thanks for all your help here.

 

Hello I've imported opencvforunity package and followed steps which is showed in the official setup video. I just click the play button and app has crashed and the exception is;

But the galaxy tablet is running without any problems. However, unity build is taking too much time for external device so I need to run it in unity editor can you help me please ?

 

Could you tell me about your test environment?
OS :
Unity version :
OpenCV for Unity version :