Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

[RELEASED] OpenCV for Unity

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Oct 30, 2014.

  1. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    I've been trying to merge FaceMask and Props for a couple weeks now. I keep getting close, then the ARProps stops scaling appropriately. I got your new combined demo to work, from your drop box, in my environment, but now the Human Head, ie. the ARProps holder, is rotating correctly but it is not mapping to the detected face. The scale is off. Any thoughts? I'm running out of time on my deadline. Anything would be helpful. Attached is my file. I changed the namespace and object name to fit in our app.
     

    Attached Files:

    • Masks.cs
      File size:
      36.8 KB
      Views:
      960
  2. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    Well, I got your Masks and Props example to work. I just used your scene and readded my custom objects and scripts. Thank you. Which leads me to my next question.

    Why are my dynamically added props, not adhering to the ARObjects measurement?

    I see the new gameobject but it's not in the right position and scale, but I can see it in the right position when I pause and look at in screen mode. Do I have to reset something when I add the new prop? Or do I need to add them before the MasksProps script fires up?

    We need to dynamically load the prop from an api, so we can update the props without requiring the user to download an app update.

    Attached is my latest script. Note: AddProp function on Line 969 is where I am adding the prop, via a json data node.

    Any suggestions would be most appreciated.
     

    Attached Files:

  3. natwales

    natwales

    Joined:
    Dec 5, 2012
    Posts:
    10
    I've been using OpenCV for unity and I've had no issues with Xcode. However, I just installed Firebase into my project and it uses cocoapods. Now when I launch the .workspace file and try to build. I'm getting the following build error.

    Undefined symbols for architecture arm64:
    "_OBJC_CLASS_$_ALAssetsLibrary", referenced from:
    objc-class-ref in opencv2(cap_ios_video_camera.o)
    ld: symbol(s) not found for architecture arm64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)

    Any suggestions on how to address this error or how to use openCVForUnity with other plugins that rely on Cocoapods?
     
  4. natwales

    natwales

    Joined:
    Dec 5, 2012
    Posts:
    10
    resolved the issue. Just needed to add AssetsLibrary.framework to buildPhases->LinkBinaryWithLibraries.
     
    EnoxSoftware likes this.
  5. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    Have you set the layer of the new game object to "UI"?
    facetracker_ar_2.PNG
     
    Last edited: Nov 18, 2017
  6. sjmtechs

    sjmtechs

    Joined:
    Jun 20, 2017
    Posts:
    11
    Hello,

    While using FaceMaskExample, my mask shakes, what can i do to make it stable ?
     
  7. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
  8. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    Turned out that I needed to make props a prefab, rather than building and adding game objects with code. For the record Developers, everything needs to be a prefab. Had a dynamically created UI button lose scope on the data passed to it. Even though the prefab doesn’t do anything other than hold the C# script. Maybe I am missing something, but it seems pretty stupid that you can’t build things with code and assets. Whatever.

    Next Question:
    We want to record the face mask and prop experience to either a single image or a 10 second video. Do you have an example for this?

    Any suggested 3rd party that would work on IOS and Android?

    I have the render texture from both the mask and AR cameras rendering to a cropped texture, but it’s crashing the app. Will have to crop the video during the save process, not during.

    Of bigger concern, the face mask is much darker in the rendered texture than it is in the actual experience. I read somewhere that it’s maybe a gamma issue. Haven’t figured that one out yet. Have you seen this before? If so, how did you fix it?

    Another weird bug I’ve been battling. We are loading the face mask and prop scene into a main scene for our app, the orthographicSize of the main camera will not change. Seems like the Camera.main gets confused and doesn’t set all the way. Even when I am setting it with code on Update.

    MainCameraAR.orthographicSize = (width * (float)Screen.height / (float)Screen.width) / 2;

    It only works when I toggle the camera setting from orthographic to perspective and then back to orthographic. Then it sizes to the width of the app. Seems like a Unity specific bug. Hence why we went with the render texture display. Which we had to anyway for a potential save photo and video solution.

    Attached is my Render class. It’s functional, but it has a bunch of testing code and debug logs. Please don’t judge me. Thank you for your help. Your combined face mask and prop solution is testing out great.
     

    Attached Files:

  9. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    Have you tried VideoWriterExample already?
    https://github.com/EnoxSoftware/Ope...eoio/VideoWriterExample/VideoWriterExample.cs
     
  10. reza1313

    reza1313

    Joined:
    Jul 14, 2017
    Posts:
    7
    hi dear, I'm a beginner in OpenCV.I want to load an image and click on that interactive but I don't know how.for example, interactive grab cut I see examples but I can't find anything related to interactive.Thanks a lot
     
  11. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
  12. reza1313

    reza1313

    Joined:
    Jul 14, 2017
    Posts:
    7
  13. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    https://github.com/EnoxSoftware/Ope...cking/TrackingExample/TrackingExample.cs#L229
    It is possible to convert from screen point to Mat point using convertScreenPointToMatPoint() method.
     
  14. reza1313

    reza1313

    Joined:
    Jul 14, 2017
    Posts:
    7
    thank you I found this example but I get this error
    Screen position out of view frustum (screen pos 351.000000, 133.000000) (Camera rect 0 0 1160 326)
    UnityEngine.SendMouseEvents:DoSendMouseEvents(Int32)
    and it seems does not work
     
  15. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    Things are moving along on our project, thank your for your help.

    Question: When I lean back from the camera, the ARObjects begin to get cut off by what looks like the quad. Facemasks still work. I tried creating different layers for each camera to render, still the props get clipped. Attached is a screen grab. Thoughts?
     

    Attached Files:

  16. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    Ha! 2 seconds after posting this I found that increasing the Clipping Planes Far from 1000 to 10000 did the trick.
     

    Attached Files:

    EnoxSoftware likes this.
  17. itouhirohumi

    itouhirohumi

    Joined:
    Dec 31, 2015
    Posts:
    2
    I haven't built my project with iOS 11. Could you tell me build secceed secret.
     
  18. PSpiroz

    PSpiroz

    Joined:
    Feb 15, 2015
    Posts:
    21
    Hi there,
    I am trying to make a 2D-game where video input (webcamTexture) is being threshold to black and white,
    take the black areas (not the whites', maybe by swapping colours, or taking threshold frame's negative picture , I don't really know how I can do this), as contours
    and make 2D-sprites and eventually 2D-colliders. (Very specifically I want to use the shadows on a wall as gameObjects by capturing them with webCamera ).

    For all the above, I have to mention that I am not an experienced programmer - but I understand coding, so I need help to figure out a strategy in making a method-coding-processing order, and what methods I will need, to make this happened.

    I think the trickiest thing is creating a 2D-sprite or whatever convenient object (even only the colliders by making point-by-point a polygon collider) out of a contour and choosing the right contours from the family tree.

    Right now I have very little idea of openCV but I am not willing to learn all openCV coding, if it is not necessary..
    So any thoughts / help of what I should do?
     
  19. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    itouhirohumi likes this.
  20. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    Has anyone experienced a dark WebCamTexture on LG v20 phones? The facial recognition is working for both mask and props, but the image is super dark. Not happening on the device camera mode. Attached is a side by side image. Same thing on Google Pixel.
     

    Attached Files:

    Last edited: Dec 1, 2017
  21. itouhirohumi

    itouhirohumi

    Joined:
    Dec 31, 2015
    Posts:
    2
    Thank you for replying.I solved my problem. My problem caused to Xcode 9.1 configuration.
    I had to change Unity-phone->General->Deployment info->Deployment Target 7.0 to Deployment Target 11.0.
    I apologize to you for a fuss over
     
    EnoxSoftware likes this.
  22. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    I think that this tutorial will be helpful.
    https://www.packtpub.com/mapt/book/application_development/9781783287376/7
    Chapter7: CREATING A PHYSICS SIMULATION BASED ON A PEN AND PAPER SKETCH
    https://github.com/PacktPublishing/OpenCV-for-Secret-Agents
     
    PSpiroz likes this.
  23. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    Checking in on the webcamtexture darkness. This seems to be wider than just one device. Major blocker for our app. I found a thread dealing with Vuforia here. Any thoughts? https://developer.vuforia.com/forum/android/dark-camera-feed-android
     
    Last edited: Dec 3, 2017
  24. HernanC

    HernanC

    Joined:
    Dec 1, 2012
    Posts:
    2
    Hi,
    I have an issue when I trying to upload the application (iOS Platform) to the AppStore. The issues are the following:

    iTunes Store Operation Failed
    ERROR ITMS-90087: "Unsupported Architectures. The executable for [MyApp]/Frameworks/opencv2.framework/opencv2 contains unsupported architectures '[x86_64, i386]'

    iTunes Store Operation Failed
    ERROR ITMS-90209: "Invalid Segment Alignment. The app binary at '[MyApp]/Frameworks/opencv2.framework/opencv2' does not have proper segment alignment. Try rebuilding the app with the latest Xcode version.

    iTunes Store Operation Failed
    ERROR ITMS-90125: "The binary is invalid. The encryption info in the LC_ENCRYPTION_INFO load command is either missing or invalid, or the binary is already encrypted. This binary does not seem to have built with Apple's linker"

    Editor version Unity 2017.2.0f3
    Editor plataform Mac
    Build Target platform: iOS
    Asset Version: OpenCV for Unity 2.2.3
    XCode 9.1

    This problem appears only when I try to publish to the app store. Locally I am able to deploy the app in my iPad without any problem.

    Any suggestion?

    Thank you.
     
    Last edited: Dec 4, 2017
    manu_unity388 and ina like this.
  25. HernanC

    HernanC

    Joined:
    Dec 1, 2012
    Posts:
    2
  26. DHein

    DHein

    Joined:
    Jan 26, 2016
    Posts:
    38
    I just tried to import the plugin into a 2017.2 project and get 131 exceptions.
    All of them seem to be caused by missing types or namespaces from the Utils namespace.

    For instance:
    How can I fix this?
    screenshot.png
     
    Last edited: Dec 4, 2017
  27. DHein

    DHein

    Joined:
    Jan 26, 2016
    Posts:
    38
    The problem is caused by conflicting Utils namespaces when using ARKit & Opencv for unity in the same project.
     
  28. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    Does this problem occur only when WebCamTexture class and OpenCVForUnity are combined?
     
  29. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    From my research it appears that you have to set an exposure setting for Android phones. The default value established by the camera manufacturer is set too low. I found the following.
    http://joshua-p-r-pan.blogspot.tw/2015/07/how-to-access-android-camera-in-unity.html
    https://medium.com/@olokobayusuf/natcam-tutorial-series-1-starting-off-dc3990f5dab6

    Is this something that should be built into the WebCamTextureHelper?
     
  30. sticklezz

    sticklezz

    Joined:
    Oct 27, 2015
    Posts:
    33
    This would be very helpful for me to auto-exposure lock or something, segmentation gets messed up with auto--exposure
     
  31. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
    After a bunch of searching I found the following in your integration page on your site.
    http://docs.natcam.io/Members/NatCam_PreviewMatrix.html

    What is the best way to integrate NatCam Pro with OpenCV for Unity? Do I have to replace the materials, colors and textures in the Example Facemask / Props cs file with NatCam.Play?

    Is there a performance bump on mobile using NatCam versus WebCamTexture?
     
  32. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    Unfortunately, I am not familiar with NatCam.
    Please contact NatCam's publisher.
     
  33. zerill

    zerill

    Joined:
    Sep 8, 2017
    Posts:
    3
    Hey, I'm having an error when I transferred a project I'm working on to another computer/unity installation. Here's a screenshot, these errors are all on opencv files and the project runs on my home computer. unity error.png
     
  34. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    There seems to be a compile error in the RobotController script.This error may be the cause of errors in the OpenCV file.
     
  35. upressplay

    upressplay

    Joined:
    Aug 8, 2017
    Posts:
    24
  36. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    There is no such method in OpenCVForUnity. However, optimization is enabled by default.
     
  37. trakloz

    trakloz

    Joined:
    Jul 23, 2017
    Posts:
    14
    Hi, I've been working with the grid markers, which are great but I'm trying to move to custom textures and the answer I found in the Q&A documentation of the marker based sample (could you update it?) didn't help me to reach the desired result. Is it even possible or should I bow to Vuforia and the like? Thanks.
     

    Attached Files:

  38. Ghus86

    Ghus86

    Joined:
    Oct 24, 2013
    Posts:
    15
    Hi

    I am having a problem with the tensorflow example I have downloaded the required files and placed them in the dnn folder that is inside the streaming assets folder.

    But then i run the example it this message appears

    model file is not loaded.The model and class names list can be downloaded here

    I tried both in unity 5.6.3 and 2017.2 in a windows 10computer
    in the project settings I tried both pc and android platforms

    I am using the latest version of the opencv for unity.

    Also I would like to confirm that it is to run this sample in both android and ios.

    Thanks in advance
     
  39. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    Sorry, the screenshot of the document is the screen of the old version.
    Please change the Inspector of MarkerSettings.
    MarkerSettings.PNG
     
  40. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    Additional Setup for TensorFlowWebCamTextureExample :
    Downlod and unzip https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip. Copy tensorflow_inception_graph.pb and imagenet_comp_graph_label_strings.txt to “Assets/StreamingAssets/dnn/” folder.
    TensorFlowExample.PNG
     
  41. Ghus86

    Ghus86

    Joined:
    Oct 24, 2013
    Posts:
    15

    Attached Files:

  42. Ghus86

    Ghus86

    Joined:
    Oct 24, 2013
    Posts:
    15
    Ok got it the streaming assets folder was inside the opencv for unity in needs to be outside
     
    EnoxSoftware likes this.
  43. trakloz

    trakloz

    Joined:
    Jul 23, 2017
    Posts:
    14
    Thanks for getting back! I'm not sure we're on the same page though. I'm trying to upload a custom 2d texture (called 'maze' on the screenshot I sent) but I cannot get it to trigger. Is there a chance to disable grid marker design in favour of a .png pattern or a photo? Sorry if I'm missing something obvious here...
     
  44. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    Please try MarkerLessARExample.
    "MarkerBased AR Example" is the Augmented Reality Example that detect and recognize only N x N block markers.
    but, "MarkerLess AR Example" is the Augmented Reality Example that detect and recognize natural 2d images.
     
  45. Vtrain

    Vtrain

    Joined:
    Oct 4, 2016
    Posts:
    1
    Hi. Im new with Open CV and Im trying to do an AR app using it. For the moment I want to change the marker to a custom made one, such like a landscape or a portrait. Anyone knows how to do that or point me in the right direction on how to do that please.

    Thanks a lot
     
  46. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    Have you tried "MarkerLess AR Example" already?
    "MarkerBased AR Example" is the Augmented Reality Example that detect and recognize only N x N block markers.
    but, "MarkerLess AR Example" is the Augmented Reality Example that detect and recognize natural 2d images.
     
  47. MFKJ

    MFKJ

    Joined:
    May 13, 2015
    Posts:
    264
    I found your github project and you have also implemented Yolo, this is cool. But there are some question regarding your YoloObjectDetectionWebCamTextureExample script.

    1. It seems that its detecting all the matched object continuously (on Update). Is there are facility available that it work on mouse input. Like if i click on screen or texture then, the script detect the object under mouse coordinates or highlight the rectangle and tell what kind of object it is.

    2. What is the performance , i mean is video running smoothly

    3. You given only examples probably, one about texture and the second is webcam texture. Is this can work with a video? prerecorded video in any format.?

    Thanks
     
  48. matrix211v1

    matrix211v1

    Joined:
    Jan 20, 2009
    Posts:
    193
    I am using the MarkerLess AR Example, but I need to make a library of images to match against and return the ID of the image it matched against.

    An example would be that I have 128 images I need to test against. If the WebCam finds a match, it reports the ID of the image it found.

    At the moment, I am making an array and using "findPattern" of 20 GameObjects but I will need to have over 300 patterns to match against each fame and at the moment, the FPS is dropping below 3.

    Any suggestions would be great.

    Thanks.
     
    Last edited: Dec 25, 2017
  49. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,567
    1. There is currently no such function.
    2. 5fps (YoloObjectDetectionWebCamTextureExample MacbookAir 2013, Mat size is 640x480.) Unfortunately, OpenCVforUnity don't support GPU.
    3. Of course. Please refer to VideoCaptureExample, HOGDescriptorExample. VideoCapture class supports mjepg format on all platform.
     
  50. MFKJ

    MFKJ

    Joined:
    May 13, 2015
    Posts:
    264
    Thanks for reply, hope that I am not disturbing in your holidays:) Some more queires

    1. Do you have any future plan to implment mouse input based video object detection?
    2. 5 fps is very low and not implementable in real applications. is there any way available to improve it? It seems that if i replace video with webcam texture then, FPS will down more which is not desirable. And whats your plan about Gpu version?
    3. Is this video also supportable with yolo? and mouse input based object tracking in these classes?



    Thanks