Search Unity

[RELEASED] OpenCV for Unity

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Oct 30, 2014.

  1. zerill

    zerill

    Joined:
    Sep 8, 2017
    Posts:
    3
    Hey there,
    I'm having a problem similar to some, but I'm not seeing a good fix for it. I'll post some code, and a screenshot of the errors, and I hope someone can give me some insight. I am getting a CvExcpetion: Native object address is NULL
    I assumed this was for the image(a Mat object) the error does not happen with each iteration(the code is in a loop).
    The Mat which is image is always initialized, and you can see that the beginning if statement never executes from the picture of the log. Any ideas?
    if (image.getNativeObjAddr() == null)
    {
    Debug.Log("image was null");
    }
    Debug.Log("1");
    Mat smat = new Mat();
    smat = image.submat(y, y + intAggregateHeight, x, x + intAggregateWidth);
    Debug.Log("2");

    if (smat == null)
    {
    Debug.Log("smat is null");
    }
    Scalar avg = Core.mean(smat);
    Debug.Log("3");


    opencverror.PNG opencverror2.PNG
     
  2. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    strange behavior?
    Could you take a screenshot?
     
  3. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    Unfortunately, There is no such function now.
     
  4. minhplhl

    minhplhl

    Joined:
    Sep 11, 2017
    Posts:
    4
    Hello, Im upgrade to lastest version, and now, the VideoWriter's open method always return false on my pc, it return true only when i build to android( though I'm not yet test on IOS). Is this is unity error or my wrong. This code snippet is here :
    Code (CSharp):
    1. writer = new VideoWriter();
    2. bool a = writer.open(filePath,0, codec, FPS, size);
    3. Debug.LogFormat("is writer open ?? {0}", a);     // -> is writer open ?? false
    4. Debug.LogFormat("is writer open ? {0}", writer.isOpened());  // is writer open ?? false
    5.  
     
  5. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    In order to display the native opencv's error code, please enclose the code in Utils.setDebugMode(true) and Utils.setDebugMode(false).
    Errors are displayed in the console?

    Code (CSharp):
    1. Utils.setDebugMode(true);
    2.  
    3. ------------
    4.  
    5. Utils.setDebugMode(false);
     
  6. zerill

    zerill

    Joined:
    Sep 8, 2017
    Posts:
    3
    I should also point out that this code was directly transferred to to Unity from an Android application using OpenCV, and it works fine there. opencverror3.PNG
    With debug mode on the error looks like:
     
  7. HNKMaster

    HNKMaster

    Joined:
    Aug 31, 2012
    Posts:
    17
    I already saw that video and script, but it didn't help. We can't understand th way it works the code, and we couldn't get working the multi object tracking with CamShift. Sorry to bother you, but I don't know how to make that.
     
  8. sjmtechs

    sjmtechs

    Joined:
    Jun 20, 2017
    Posts:
    11
    Hello,

    While using WebCamTextureFaceMaskExample,
    how can I change mask when a user exits (or when there is no active detection/no user) and new user comes ?
     
  9. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    The argument of submat() method seems to be incorrect.

    image.submat(y, y + intAggregateHeight, x, x + intAggregateWidth);

    0 <= y
    y <= y + intAggregateHeight
    y + intAggregateHeight <= image.rows()
     
  10. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    VideoWriterExample work fine?
    https://github.com/EnoxSoftware/Ope...WriterExample/VideoWriterExample.cs#L185-L186
    I have tested VideoWriter.fourcc ('M', 'J', 'P', 'G') on all platforms.
     
  11. RubenB

    RubenB

    Joined:
    Jun 3, 2015
    Posts:
    1
    Hello, where can I find the example and explanation to use with Tensorflow - DNN to recognize objects?

    Thanks.
     
  12. esoinila

    esoinila

    Joined:
    Apr 27, 2017
    Posts:
    7
    I tested that yesterday. You need to load the .pb and .txt file from the zip whose link is shown in Debug-log when you try to run the scene.

    But as to using TensorFlow 1.3.0 to re-train inception-model and then use it in the plugin... I got the Unity Editor (5.6.3f1) to crash when I tried that. The model could be tested in TensorFlow without problems. So can anybody answer what version of TensorFlow trained models are compatible with Open CV ?
     
  13. esoinila

    esoinila

    Joined:
    Apr 27, 2017
    Posts:
    7
    Apparently there are several different versions of the Inception model. Which one does the example-code use ? Maybe I trained with too new Inception model and trying to load that crashed the Unity Editor.
     
  14. slava_pankratov

    slava_pankratov

    Joined:
    Jan 17, 2015
    Posts:
    42
    Hi! Is there any example how to remove background from webcam feed when the user is standing in front of green screen? Thanks
     
  15. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    Could you send me the code to reproduce the problem?
    store@enoxsoftware.com
     
  16. seansteezy

    seansteezy

    Joined:
    Nov 28, 2013
    Posts:
    114
    Hey this is probably a dumb question but I'm working with the marker based webcam AR example, but building for Android, and the web cam texture is not fullscreen (it's like a movie where the quad for the camera texture has bars on the top and bottom since it's a perfect square), so how does one get fullscreen video on the Quad that displays the camera?
     
  17. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    OpenCVForUnity2.2.1 is based on this commit.
    https://github.com/opencv/opencv/tree/3.3.0/modules/dnn/misc/tensorflow
    https://github.com/opencv/opencv/issues?utf8=✓&q=is:issue tensorflow layer is:open
    It may not correspond to the latest version of tensorflow.
     
  18. minhplhl

    minhplhl

    Joined:
    Sep 11, 2017
    Posts:
    4
  19. ch0arrim

    ch0arrim

    Joined:
    Jul 18, 2012
    Posts:
    10
    Hi,
    I bought your asset and I am having some real problems to understand how to get what text has been discovered???
    I mean I get the text module running fine and it shows that there IS text on an image but I can't for the life of me understand how to get what text has actually been found?

    Also, is there a way to scan a texture instead of a saved image? Would be so much better if I didn't have to save the image each time I run the text scan?

    /Best regards
    Daniel
     
  20. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    I also bought this today and it is all looking very interesting. I however have the same question as Daniel above. How to get the strings from recognized text and I bought this one because of Playmaker, so please also show a text example using PlayMaker.
     
  21. ch0arrim

    ch0arrim

    Joined:
    Jul 18, 2012
    Posts:
    10
    I can help you with creating a playmaker action if you need it.
     
  22. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    That would be very helpful thank you.
     
  23. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    After importing the playmaker actions for OpenCV my action browser is extremely slow and hardly scrolls. Searching takes ages, scrolling through the actions is painful and typing in the search bar takes 10’s of seconds. This is not acceptable and would be impossible to use PlayMaker with OpenCV.
     
  24. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    https://forum.unity.com/threads/released-opencv-for-unity.277080/page-23#post-3086194
     
  25. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    So, anyone on this forum know how to just get the returned results from the text example? The very basic text example.
    It's not looking good on this forum as answers on the simplest stuff is just far and wide not here.
    It's getting very frustrating now.
     
  26. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    You can use the value of landmarkPoints.Count to judge whether a person is detected.
    if (landmarkPoints.Count <= 0) {
    Debug.Log("no user");
    }
    Also,
    You can change the face mask by calling OnChangeFaceMaskButtonClick ().
     
  27. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
  28. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    let us try this question again as it seams to be getting overlooked.
    Anyone on this forum know how to just get the returned results from the text example? The very basic text example.I want the Text, that is the whole point of Text recognitions is it not?
     
  29. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    Sorry for my late reply.
    Currently, only TextDetectionExample is included with OpenCVForUnity.
    https://github.com/EnoxSoftware/Ope...ules/text/TextExample/TextDetectionExample.cs

    But, I plan to add text recognition examples in future updates.
    This code is translated from https://github.com/opencv/opencv_co...s/text/samples/segmented_word_recognition.cpp .
    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. using System.Collections.Generic;
    4.  
    5. #if UNITY_5_3 || UNITY_5_3_OR_NEWER
    6. using UnityEngine.SceneManagement;
    7. #endif
    8. using OpenCVForUnity;
    9.  
    10. namespace OpenCVForUnityExample
    11. {
    12.     /// <summary>
    13.     /// https://github.com/opencv/opencv_co...s/text/samples/segmented_word_recognition.cpp
    14.     /// </summary>
    15.     public class SegmentedWordRecognitionExample : MonoBehaviour
    16.     {
    17.         // Use this for initialization
    18.         void Start ()
    19.         {
    20.             //if true, The error log of the Native side OpenCV will be displayed on the Unity Editor Console.
    21.             Utils.setDebugMode (true);
    22.  
    23.             Mat image = Imgcodecs.imread (Utils.getFilePath ("text/scenetext_segmented_word03.jpg"));
    24.             Mat mask = Imgcodecs.imread (Utils.getFilePath ("text/scenetext_segmented_word03_mask.png"), Imgcodecs.IMREAD_GRAYSCALE);
    25.          
    26.  
    27.             // character recognition vocabulary
    28.             string voc = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
    29.             // Emission probabilities for the HMM language model (identity matrix by default)
    30.             Mat emissionProbabilities = Mat.eye (voc.Length, voc.Length, CvType.CV_64FC1);
    31.             // Bigram transition probabilities for the HMM language model
    32.             Mat transitionProbabilities;
    33.  
    34.             List<string> lexicon = new List<string> ();
    35.             lexicon.Add ("Private");
    36.             lexicon.Add ("Hire");
    37.             transitionProbabilities = Text.createOCRHMMTransitionsTable (voc, lexicon);
    38.  
    39.  
    40.            
    41.             OCRHMMDecoder ocrNM = OCRHMMDecoder.create (
    42.                                       Utils.getFilePath ("text/OCRHMM_knn_model_data.xml.gz"),
    43.                                       voc, transitionProbabilities, emissionProbabilities);
    44.  
    45.  
    46.             string output = ocrNM.run (mask, 0);
    47.             Debug.Log ("ocrNM " + output);
    48.  
    49.  
    50.             Texture2D texture = new Texture2D (image.cols (), image.rows (), TextureFormat.RGBA32, false);
    51.  
    52.             Utils.matToTexture2D (image, texture);
    53.  
    54.             gameObject.GetComponent<Renderer> ().material.mainTexture = texture;
    55.  
    56.  
    57.             Utils.setDebugMode (false);
    58.         }
    59.    
    60.         // Update is called once per frame
    61.         void Update ()
    62.         {
    63.  
    64.         }
    65.  
    66.         /// <summary>
    67.         /// Raises the back button click event.
    68.         /// </summary>
    69.         public void OnBackButtonClick ()
    70.         {
    71.             #if UNITY_5_3 || UNITY_5_3_OR_NEWER
    72.             SceneManager.LoadScene ("OpenCVForUnityExample");
    73.             #else
    74.             Application.LoadLevel ("OpenCVForUnityExample");
    75.             #endif
    76.         }
    77.     }
    78. }
     
  30. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    Could you tell me about your test environment?
    OpenCV for Unity version :
    PlayMaker version :
     
  31. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    Thank you.

    Version from all assets are all the very newest.
    in the project is only PlayMaker , OpenCV and OpenCV actions.
     
  32. ch0arrim

    ch0arrim

    Joined:
    Jul 18, 2012
    Posts:
    10
    No worries.
    I got the example project to work after some hiccups.
    But in any case, do you plan to make this kind of example project with your asset?
    The reason I am asking this is the code you provide in the example uses lot more work with channels etc, while the example code that finds the text has lot less code.

    Also, I asked before if there was a way to get your asset to read from say a texture that streams from the camera? To prevent the app from saving an image that is just going to be thrown away 100s of times a day?
     
    Last edited: Sep 17, 2017
  33. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    Thank you very much for reporting.
    Could you provide additional information?
    OS :
    Unity version :
     
  34. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    Camera Stream means WebCamTexture?
    I think it is possible to combine WebCamTextureToMatHelperExample and text recognition.
    I plan to add text recognition examples(based on https://github.com/opencv/opencv_contrib/blob/master/modules/text/samples/webcam_demo.cpp) in future updates.
     
  35. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    Playmaker Webcam Face detection

    Working with

    Windows 10 pro 64Bit

    Unity 2017.1.0f3

    PM 1.8.5

    OpenCV 2.2.1

    OpenCV actions 1.0.1

    Playmaker action Browser working here fine only a bit sluggish when you open the whole OpenCV actions list. It works however.



    Here not working and killing Unity

    Newer versions of Unity or Playmaker breaks something as Unity crashes all the time when trying the Playmaker webcam face detect demo

    Can you possibly separate the actions out a little more in the action browser, as everything is under opencvForPlaymaker and it is a lot of stuff? That is most probably the cause of the action browser hanging or being extremely sluggish.
     
  36. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    Could you please explain how you got the script working in the text example?
    So where did you get the scenetext_segmented_word03 from?
     
  37. Softgear

    Softgear

    Joined:
    Feb 21, 2016
    Posts:
    3
    Bah I accidentally used my other login to post this reply, its still "ch0arrim" :)

    I googled the filename and found that it exists a github project with those files being used.
    https://github.com/opencv/opencv_contrib/tree/master/modules/text/samples
    I downloaded all 3 files being listed in his example code above and put them in StreamingAssets/text/ folder
    Try with that first and holler if you need more help :)
     
  38. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351

    Thanks, I did that added the SegmentedWordRecognitionExample.cs from further above here and added the 3 files to streaming assets/Text. I then added SegmentedWordRecognitionExample.cd to the cube in the text example and removed the
    TextDetectionExample.cs example script.
    It just crashes Unity Imediatly.
    If I just use TextDetectionExample.cs I get the marked sign Post.
    Any ideas?
    Do you actually get text now to the Console from just that script?
     
  39. Softgear

    Softgear

    Joined:
    Feb 21, 2016
    Posts:
    3
    Hi again, sorry for asking so much but it feels I am soo close to succeed.

    I took the example code above and pasted it to WebCamTextureToMatExample making my start method look like this:
    Code (csharp):
    1.  
    2. string voc = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
    3.  
    4.            
    5.             // Emission probabilities for the HMM language model (identity matrix by default)
    6.             Mat emissionProbabilities = Mat.eye(voc.Length, voc.Length, CvType.CV_64FC1);
    7.             //Mat emissionProbabilities = Mat.eye(voc.Length, voc.Length, CvType.CV_8UC4);
    8.             // Bigram transition probabilities for the HMM language model
    9.             Mat transitionProbabilities;
    10.  
    11.             List<string> lexicon = new List<string>();
    12.             lexicon.Add("Enklare");
    13.             lexicon.Add("jag");
    14.             transitionProbabilities = Text.createOCRHMMTransitionsTable(voc, lexicon);
    15.  
    16.  
    17.  
    18.             ocrNM = OCRHMMDecoder.create(
    19. Utils.getFilePath("text/OCRHMM_knn_model_data.xml.gz"),
    20. voc, transitionProbabilities, emissionProbabilities);
    21.  
    Then I call the method at regular intervals trying to find the text.
    Now I can get it to work with the scenetext_segmented_word03 images and I guess it because the scenetext_segmented_word03_mask is a sort of inverted image with the text being white and the rest is black.

    But I still can't get it to work with a stream image.
    What do I need to do to make the mask?
    Or is it possible to use your TextDetectionExample to instead of pointing out that there is text here and there, it gives the characters found?
    I would deeply appreciate if you could help me with this! Its so frustrating that I am sooo close to finally understand how to do this!

    /Best regards
    Daniel
     
  40. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    Hello developper, please post a working example of Text Recognition and not just detection.
    I bought this asset for exactly seeing how this is done. You advertise as such doing also recognition. I do not want to spend my whole time just trying to build it myself. This is why we buy assets to get a good head start.

    I know you are trying to help, but if you look all people want to do is now get the text either from a pic or stream. Just please add a working example of this as soon as you can and don't forget a Playmaker example as that was the reason I bought this asset and not the other.
    Thank you in advance
     
  41. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    TensorFlowWebCamTextureExample also not working either. I have to download extra stuff to get a scene of yours working.

    Why not include the files in the package? I expect scenes to work or where is the point?

    Please in the future make a doc actually explaining how to use your examples and stuff we should have to download to get them working.

    If I sound irritated, then yes it is very irritating that nothing works out of the box and I have to guess what is going on and spending my time in Google and forums.
     
  42. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351
    This is what I would like to do.

    1. Have a camera snapshot a street sign as the pic in your demo.(I can do that in Playmaker or Bolt already using natcam)

    2. Then detect where the text is. (This is working)

    3. Then recognize the text and set it to a string (Not included in your asset).

    4. Then I can do with that string as I like.


    I use Playmaker and know basics up to intermediate of c#. This is what I want to do and should be easy and included. This is the 21st century and not the 19th or 20th for that matter.

    I should not have to study the entire Assembly to just have this function working.


    If I want to be able to drive a car, I do not wish to know first how I have to build one. The basic steps of car driving would get me to a lot of places.

    That is why we pay for learning to drive, to make things easier.


    With this asset, as it is at the moment. I feel as if I would have to first build the car and then get somebody else to drive it.
     
  43. ch0arrim

    ch0arrim

    Joined:
    Jul 18, 2012
    Posts:
    10
    eagleeyez, I understand that you are frustrated not being able to proceed with your project, I am too. But I don't think it's fair that you post all these opinions about what should be included etc.
    We don't even know if this is their full time job. They might be working on this in their spare time, have family's and other responsibilities.
    I agree that it should come with some sort of text recognition example and I believe if we are patient they will help us with this since it seems as if they are pretty active in answering people's questions. But I don't think you will get your answers faster by posting that :)

    When they can assist us with the code to get a text recognition working, I can help you create a simple playmaker action for it.

    /Best regards
    Daniel
     
  44. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351

    You are right, but this is going to be a game changer for OCR in Unity. There are a lot of people wanting this without having to learn Python, C++, Java, Tesseract and so on just to get some text recognized and also use in Android. I was just disappointed as, as it reads on their website you get detection and recognition.

    I know, I have to be a little more patient but I just wanted to have that bit done and get on with the rest. So I will be looking forward to an update with recognition examples.
     
  45. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    Could you try new version?
    OpenCVForUnityPlayMakerActions1.0.2
    Fixed the scroll delay of the Action Browser.
    https://www.dropbox.com/s/up7t1gslsv1ogfw/OpenCVForUnityPlayMakerActions1.0.2.unitypackage?dl=0

    Is lbpcascade_frontalface.xml located in "Assets\StreamingAssets\"?
     
  46. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
  47. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351

    Thank you, that fixed it in Playmaker, now searching and scrolling the action browser is smooth. And you sectioned it out a bit in the action browser, much better.

    Yes I have all in streaming assets. I will have to have another look at that.

    And thanks for the segmented text package. I will take a look at that now and get back to you.
     
  48. eagleeyez

    eagleeyez

    Joined:
    Mar 21, 2013
    Posts:
    351

    Thanks for this, but is it actually recognizing text? OCR

    As the Log gives out the words Private Hire which is in the picture in the middle but in the script you are also adding the words Private and Hire to a lexicon.

    I was expecting it to get the text out of the picture and DebugLog the following

    Customer Freeplhone Private Hiere.

    Maybe I am doing something wrong.
    Thanks for all your help here.
     
  49. ikasapoglu

    ikasapoglu

    Joined:
    Jul 17, 2017
    Posts:
    14
    Hello I've imported opencvforunity package and followed steps which is showed in the official setup video. I just click the play button and app has crashed and the exception is;
    But the galaxy tablet is running without any problems. However, unity build is taking too much time for external device so I need to run it in unity editor can you help me please ?
     
  50. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,123
    Could you tell me about your test environment?
    OS :
    Unity version :
    OpenCV for Unity version :