[RELEASED] OpenCV for Unity

I've been trying to merge FaceMask and Props for a couple weeks now. I keep getting close, then the ARProps stops scaling appropriately. I got your new combined demo to work, from your drop box, in my environment, but now the Human Head, ie. the ARProps holder, is rotating correctly but it is not mapping to the detected face. The scale is off. Any thoughts? I'm running out of time on my deadline. Anything would be helpful. Attached is my file. I changed the namespace and object name to fit in our app.

 

Well, I got your Masks and Props example to work. I just used your scene and readded my custom objects and scripts. Thank you. Which leads me to my next question.

Why are my dynamically added props, not adhering to the ARObjects measurement?

I see the new gameobject but it's not in the right position and scale, but I can see it in the right position when I pause and look at in screen mode. Do I have to reset something when I add the new prop? Or do I need to add them before the MasksProps script fires up?

We need to dynamically load the prop from an api, so we can update the props without requiring the user to download an app update.

Attached is my latest script. Note: AddProp function on Line 969 is where I am adding the prop, via a json data node.

Any suggestions would be most appreciated.

 

I've been using OpenCV for unity and I've had no issues with Xcode. However, I just installed Firebase into my project and it uses cocoapods. Now when I launch the .workspace file and try to build. I'm getting the following build error.

Undefined symbols for architecture arm64:
"_OBJC_CLASS_$_ALAssetsLibrary", referenced from:
objc-class-ref in opencv2(cap_ios_video_camera.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Any suggestions on how to address this error or how to use openCVForUnity with other plugins that rely on Cocoapods?

 

resolved the issue. Just needed to add AssetsLibrary.framework to buildPhases->LinkBinaryWithLibraries.

 

Have you set the layer of the new game object to "UI"?

 

Hello,

While using FaceMaskExample, my mask shakes, what can i do to make it stable ?

 

https://forum.unity.com/threads/released-dlib-facelandmark-detector.408958/page-4#post-3281274
Also,
I have not tried it yet, but adapting a low pass filter to the acquired face landmark points may reduce the noise.

 

Turned out that I needed to make props a prefab, rather than building and adding game objects with code. For the record Developers, everything needs to be a prefab. Had a dynamically created UI button lose scope on the data passed to it. Even though the prefab doesn’t do anything other than hold the C# script. Maybe I am missing something, but it seems pretty stupid that you can’t build things with code and assets. Whatever.

Next Question:
We want to record the face mask and prop experience to either a single image or a 10 second video. Do you have an example for this?

Any suggested 3rd party that would work on IOS and Android?

I have the render texture from both the mask and AR cameras rendering to a cropped texture, but it’s crashing the app. Will have to crop the video during the save process, not during.

Of bigger concern, the face mask is much darker in the rendered texture than it is in the actual experience. I read somewhere that it’s maybe a gamma issue. Haven’t figured that one out yet. Have you seen this before? If so, how did you fix it?

Another weird bug I’ve been battling. We are loading the face mask and prop scene into a main scene for our app, the orthographicSize of the main camera will not change. Seems like the Camera.main gets confused and doesn’t set all the way. Even when I am setting it with code on Update.

MainCameraAR.orthographicSize = (width * (float)Screen.height / (float)Screen.width) / 2;

It only works when I toggle the camera setting from orthographic to perspective and then back to orthographic. Then it sizes to the width of the app. Seems like a Unity specific bug. Hence why we went with the render texture display. Which we had to anyway for a potential save photo and video solution.

Attached is my Render class. It’s functional, but it has a bunch of testing code and debug logs. Please don’t judge me. Thank you for your help. Your combined face mask and prop solution is testing out great.

 

Have you tried VideoWriterExample already?
https://github.com/EnoxSoftware/Ope...eoio/VideoWriterExample/VideoWriterExample.cs

 

hi dear, I'm a beginner in OpenCV.I want to load an image and click on that interactive but I don't know how.for example, interactive grab cut I see examples but I can't find anything related to interactive.Thanks a lot

 

interactive?
Have you tried GrabCutExample?
https://github.com/EnoxSoftware/Ope...y/Examples/MainModules/imgproc/GrabCutExample

 

Yes I tried it but more but I need interactive grab cut
like this example https://docs.opencv.org/trunk/d8/d83/tutorial_py_grabcut.html
the usual processing is awesome with your package but Interactive processing I need get some points from the user and then start processing

 

https://github.com/EnoxSoftware/Ope...cking/TrackingExample/TrackingExample.cs#L229
It is possible to convert from screen point to Mat point using convertScreenPointToMatPoint() method.

 

thank you I found this example but I get this error
Screen position out of view frustum (screen pos 351.000000, 133.000000) (Camera rect 0 0 1160 326)
UnityEngine.SendMouseEventsoSendMouseEvents(Int32)
and it seems does not work

 

Things are moving along on our project, thank your for your help.

Question: When I lean back from the camera, the ARObjects begin to get cut off by what looks like the quad. Facemasks still work. I tried creating different layers for each camera to render, still the props get clipped. Attached is a screen grab. Thoughts?

 

Ha! 2 seconds after posting this I found that increasing the Clipping Planes Far from 1000 to 10000 did the trick.

 

I haven't built my project with iOS 11. Could you tell me build secceed secret.

 

Hi there,
I am trying to make a 2D-game where video input (webcamTexture) is being threshold to black and white,
take the black areas (not the whites', maybe by swapping colours, or taking threshold frame's negative picture , I don't really know how I can do this), as contours
and make 2D-sprites and eventually 2D-colliders. (Very specifically I want to use the shadows on a wall as gameObjects by capturing them with webCamera ).

For all the above, I have to mention that I am not an experienced programmer - but I understand coding, so I need help to figure out a strategy in making a method-coding-processing order, and what methods I will need, to make this happened.

I think the trickiest thing is creating a 2D-sprite or whatever convenient object (even only the colliders by making point-by-point a polygon collider) out of a contour and choosing the right contours from the family tree.

Right now I have very little idea of openCV but I am not willing to learn all openCV coding, if it is not necessary..
So any thoughts / help of what I should do?

 

Could you send an error log?
https://enoxsoftware.com/opencvforunity/contact/technical-inquiry/

 

Has anyone experienced a dark WebCamTexture on LG v20 phones? The facial recognition is working for both mask and props, but the image is super dark. Not happening on the device camera mode. Attached is a side by side image. Same thing on Google Pixel.

 

Thank you for replying.I solved my problem. My problem caused to Xcode 9.1 configuration.
I had to change Unity-phone->General->Deployment info->Deployment Target 7.0 to Deployment Target 11.0.
I apologize to you for a fuss over

 

I think that this tutorial will be helpful.
https://www.packtpub.com/mapt/book/application_development/9781783287376/7
Chapter7: CREATING A PHYSICS SIMULATION BASED ON A PEN AND PAPER SKETCH
https://github.com/PacktPublishing/OpenCV-for-Secret-Agents

 

Checking in on the webcamtexture darkness. This seems to be wider than just one device. Major blocker for our app. I found a thread dealing with Vuforia here. Any thoughts? https://developer.vuforia.com/forum/android/dark-camera-feed-android

 

Hi,
I have an issue when I trying to upload the application (iOS Platform) to the AppStore. The issues are the following:

iTunes Store Operation Failed
ERROR ITMS-90087: "Unsupported Architectures. The executable for [MyApp]/Frameworks/opencv2.framework/opencv2 contains unsupported architectures '[x86_64, i386]'

iTunes Store Operation Failed
ERROR ITMS-90209: "Invalid Segment Alignment. The app binary at '[MyApp]/Frameworks/opencv2.framework/opencv2' does not have proper segment alignment. Try rebuilding the app with the latest Xcode version.

iTunes Store Operation Failed
ERROR ITMS-90125: "The binary is invalid. The encryption info in the LC_ENCRYPTION_INFO load command is either missing or invalid, or the binary is already encrypted. This binary does not seem to have built with Apple's linker"

Editor version Unity 2017.2.0f3
Editor plataform Mac
Build Target platform: iOS
Asset Version: OpenCV for Unity 2.2.3
XCode 9.1

This problem appears only when I try to publish to the app store. Locally I am able to deploy the app in my iPad without any problem.

Any suggestion?

Thank you.

 

I just realized how to solve this.
Is explained in this post.

http://ikennd.ac/blog/2015/02/stripping-unwanted-architectures-from-dynamic-libraries-in-xcode/

The validation worked and I sent the new build to the app store. Now I'm waiting Apple releases the build, but this is other story.

I hope this help anybody with the same issue.

 

I just tried to import the plugin into a 2017.2 project and get 131 exceptions.
All of them seem to be caused by missing types or namespaces from the Utils namespace.

For instance:

How can I fix this?

 

The problem is caused by conflicting Utils namespaces when using ARKit & Opencv for unity in the same project.

 

Does this problem occur only when WebCamTexture class and OpenCVForUnity are combined?

 

From my research it appears that you have to set an exposure setting for Android phones. The default value established by the camera manufacturer is set too low. I found the following.
http://joshua-p-r-pan.blogspot.tw/2015/07/how-to-access-android-camera-in-unity.html
https://medium.com/@olokobayusuf/natcam-tutorial-series-1-starting-off-dc3990f5dab6

Is this something that should be built into the WebCamTextureHelper?

 

This would be very helpful for me to auto-exposure lock or something, segmentation gets messed up with auto--exposure

 

After a bunch of searching I found the following in your integration page on your site.
http://docs.natcam.io/Members/NatCam_PreviewMatrix.html

What is the best way to integrate NatCam Pro with OpenCV for Unity? Do I have to replace the materials, colors and textures in the Example Facemask / Props cs file with NatCam.Play?

Is there a performance bump on mobile using NatCam versus WebCamTexture?

 

Unfortunately, I am not familiar with NatCam.
Please contact NatCam's publisher.

 

Hey, I'm having an error when I transferred a project I'm working on to another computer/unity installation. Here's a screenshot, these errors are all on opencv files and the project runs on my home computer.

 

There seems to be a compile error in the RobotController script.This error may be the cause of errors in the OpenCV file.

 

We have been working on performance issues on many different devices. And noticed in the documentation a cv2.setUseOptimized().

https://docs.opencv.org/3.3.0/dc/d71/tutorial_py_optimization.html

Does OpenCV for Unity have a setting for this? If so, is it on by default? If not, how can we turn this on?

 

There is no such method in OpenCVForUnity. However, optimization is enabled by default.

 

Hi, I've been working with the grid markers, which are great but I'm trying to move to custom textures and the answer I found in the Q&A documentation of the marker based sample (could you update it?) didn't help me to reach the desired result. Is it even possible or should I bow to Vuforia and the like? Thanks.

 

Hi

I am having a problem with the tensorflow example I have downloaded the required files and placed them in the dnn folder that is inside the streaming assets folder.

But then i run the example it this message appears

model file is not loaded.The model and class names list can be downloaded here

I tried both in unity 5.6.3 and 2017.2 in a windows 10computer
in the project settings I tried both pc and android platforms

I am using the latest version of the opencv for unity.

Also I would like to confirm that it is to run this sample in both android and ios.

Thanks in advance

 

Sorry, the screenshot of the document is the screen of the old version.
Please change the Inspector of MarkerSettings.

 

Additional Setup for TensorFlowWebCamTextureExample :
Downlod and unzip https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip. Copy tensorflow_inception_graph.pb and imagenet_comp_graph_label_strings.txt to “Assets/StreamingAssets/dnn/” folder.

 

Thanks for replying I have done that before but it didnt work
any other tip

 

Ok got it the streaming assets folder was inside the opencv for unity in needs to be outside

 

Thanks for getting back! I'm not sure we're on the same page though. I'm trying to upload a custom 2d texture (called 'maze' on the screenshot I sent) but I cannot get it to trigger. Is there a chance to disable grid marker design in favour of a .png pattern or a photo? Sorry if I'm missing something obvious here...

 

Please try MarkerLessARExample.
"MarkerBased AR Example" is the Augmented Reality Example that detect and recognize only N x N block markers.
but, "MarkerLess AR Example" is the Augmented Reality Example that detect and recognize natural 2d images.

 

Hi. Im new with Open CV and Im trying to do an AR app using it. For the moment I want to change the marker to a custom made one, such like a landscape or a portrait. Anyone knows how to do that or point me in the right direction on how to do that please.

Thanks a lot

 

Have you tried "MarkerLess AR Example" already?
"MarkerBased AR Example" is the Augmented Reality Example that detect and recognize only N x N block markers.
but, "MarkerLess AR Example" is the Augmented Reality Example that detect and recognize natural 2d images.

 

I found your github project and you have also implemented Yolo, this is cool. But there are some question regarding your YoloObjectDetectionWebCamTextureExample script.

1. It seems that its detecting all the matched object continuously (on Update). Is there are facility available that it work on mouse input. Like if i click on screen or texture then, the script detect the object under mouse coordinates or highlight the rectangle and tell what kind of object it is.

2. What is the performance , i mean is video running smoothly

3. You given only examples probably, one about texture and the second is webcam texture. Is this can work with a video? prerecorded video in any format.?

Thanks

 

I am using the MarkerLess AR Example, but I need to make a library of images to match against and return the ID of the image it matched against.

An example would be that I have 128 images I need to test against. If the WebCam finds a match, it reports the ID of the image it found.

At the moment, I am making an array and using "findPattern" of 20 GameObjects but I will need to have over 300 patterns to match against each fame and at the moment, the FPS is dropping below 3.

Any suggestions would be great.

Thanks.

 

1. There is currently no such function.
2. 5fps (YoloObjectDetectionWebCamTextureExample MacbookAir 2013, Mat size is 640x480.) Unfortunately, OpenCVforUnity don't support GPU.
3. Of course. Please refer to VideoCaptureExample, HOGDescriptorExample. VideoCapture class supports mjepg format on all platform.

 

Thanks for reply, hope that I am not disturbing in your holidays Some more queires

1. Do you have any future plan to implment mouse input based video object detection?
2. 5 fps is very low and not implementable in real applications. is there any way available to improve it? It seems that if i replace video with webcam texture then, FPS will down more which is not desirable. And whats your plan about Gpu version?
3. Is this video also supportable with yolo? and mouse input based object tracking in these classes?



Thanks