Search Unity

  1. The Unity Pro & Visual Studio Professional Bundle gives you the tools you need to develop faster & collaborate more efficiently. Learn more.
    Dismiss Notice
  2. Want more efficiency in your development work? Sign up to receive weekly tech and creative know-how from Unity experts.
    Dismiss Notice
  3. Participate with students all over the world and build projects to teach people. Join now!
    Dismiss Notice
  4. Build games and experiences that can load instantly and without install. Explore the Project Tiny Preview today!
    Dismiss Notice
  5. Improve your Unity skills with a certified instructor in a private, interactive classroom. Watch the overview now.
    Dismiss Notice
  6. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice

[RELEASED] OpenCV for Unity

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Oct 30, 2014.

  1. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    OpenCV for Unity
    http://u3d.as/content/enox-software/open-cv-for-unity





    screenshot3.png
    opencv3.3.1_features.png
    scrrenshot5.png

    Requires Unity 5.6.6 or higher.

    Works with Unity Cloud Build

    iOS & Android support
    Windows10 UWP support
    WebGL support
    Win & Mac & Linux Standalone support
    Preview support in the Editor


    OpenCV for Unity is an Assets Plugin for using OpenCV 3.4.2 from within Unity.

    Official Site | SampleCode | Android Demo WebGL Demo | Tutorial & Demo Video | Forum | API Reference | Support Modules


    Features:
    • Since this package is a clone of OpenCV Java, you are able to use the same API as OpenCV Java 3.4.2 (link).
    • You can image processing in real-time by using the WebCamTexture capabilities of Unity.(real-time face detection works smoothly in iPhone 5)
    • Provides a method to interconversion of Unity's Texture2D and OpenCV's Mat.
    • IDisposable is implemented in many classes.You can manage the resources with the “using” statement.
    • PlayMakerActions for OpenCVforUnity is available.

    ExampleCode using OpenCV for Unity is available.
    MarkerBased AR Example
    MarkerLess AR Example
    FaceTracker Example
    FaceSwapper Example
    FaceMask Example
    RealTime FaceRecognition Example
    Vuforia with OpenCV for Unity Example
    Kinect with OpenCV for Unity Example
    GoogleVR With OpenCV for Unity Example
    AVPro with OpenCV for Unity Example
    HoloLens with OpenCV for Unity Example


    OpenCV for Unity uses OpenCV under 3-clause BSD License; see Third-Party Notices.txt file in package for details.


    System Requirements:
    Build Win Standalone & Preview Editor : Windows7 or later
    Build Mac Standalone & Preview Editor : OSX 10.9 or later


    More Info >>


    Release Notes:
    2.3.1

    [Common]Updated to OpenCV3.4.2.
    [Android,UWP]Fixed Utils.setDebugMode() method on the IL2CPP backend.
    [Common]Added DnnObjectDetectionExample and DnnObjectDetectionWebCamTextureExample.
    2.3.0
    [iOS]Added a function to automatically remove the simulator architecture(i386,x86_64) at build time.
    [Common] Improved OpenCVForUnityMenuItem.setPluginImportSettings() method.
    2.2.9
    [Linux]Simplified the Linux platform setup procedure.
    [Common]Added support for Utils. setDebugMode() method on all platforms.
    [Common]Updated to WebCamTextureToMatHelper.cs v1.0.9.
    [Common]Added MatToTextureInRenderThreadExample and AlphaBlendingExample.
    2.2.8
    [Common]Updated to WebCamTextureToMatHelper.cs v1.0.7.
    [Common]Added MatBasicProcessingExample.
    [Common]Fixed WebCamTextureToMatExample, WebCamTextureToMatHelperExample, ArUcoExample.
    [Common]Added flip flag to Utils.fastMatToTexture2D() method and Utils.fastTexture2DToMat() method.
    [Common]Added throwException flag to Utils.setDebugMode() method.
    2.2.7
    [Common]Updated to OpenCV3.4.1.
    [Common]Added OpenPoseExample(The model file is not included in this asset.), KalmanFilterExample, ArUcoCameraCalibrationExample.
    [Common]Fixed VideoWriterExample, VideoCaptureExample, ImwriteScreenCaptureExample, CamShiftExample, TrackingExample, HandPoseEstimationExample, ArUcoCreateMarkerExample, ArUcoExample, ArUcoWebCamTextureExample.
    [Common] Updated to WebCamTextureToMatHelper.cs v1.0.6.
    2.2.6
    [Android]Added arm64-v8a Architecture.
    [Common]Added ImwriteScreenCaptureExample.
    2.2.5
    [Common] Updated to WebCamTextureToMatHelper.cs v1.0.4.
    [Common] Fixed MobileNetSSDExample and MobileNetSSDWebCamTextureExample.
    2.2.4
    [Common]Updated to OpenCV3.3.1.
    [Common]Added ResnetSSDFaceDetectionExample, YoloObjectDetectionExample, YoloObjectDetectionWebCamTextureExample.
    2.2.3
    [Common]Updated to WebCamTextureToMatHelper.cs v1.0.3.
    [iOS]opencv2.framework is changed from static framework to embededd framework. (Target minimum iOS Version must be set to 8.0 or higher.)
    2.2.5
    [Common] Updated to WebCamTextureToMatHelper.cs v1.0.4.
    [Common] Fixed MobileNetSSDExample and MobileNetSSDWebCamTextureExample.
    2.2.4
    [Common]Updated to OpenCV3.3.1.
    [Common]Added ResnetSSDFaceDetectionExample, YoloObjectDetectionExample, YoloObjectDetectionWebCamTextureExample.
    2.2.3
    [Common]Updated to WebCamTextureToMatHelper.cs v1.0.3.
    [iOS] opencv2.framework is changed from static framework to embededd framework. (Target minimum iOS Version must be set to 8.0 or higher.)
    2.2.2
    [Common]Added TextRecognitionExample.
    2.2.1
    [Common]Updated to OpenCV3.3.0.
    [Common]Added dnn module.(win,mac,ios,android platform)
    [Common]Added img_hash, reg, text module.(all platform)
    [Common]Added MobileNetSSDExample, MobileNetSSDWebCamTextureExample, TensorFlowWebCamTextureExample, ThinPlateSplineShapeTransformerExample, TextDetectionExample, VideoWriterExample.
    [Common]WindowsStoreApp8.1 & WindowsPhone8.1 support have been deprecated.
    2.0.0
    [Common]Updated WebCamTextureToMatHelper.cs v1.0.2
    [Common]Improved Utils.getFilePathAsync().
    2.1.9
    [WebGL]Fixed Utils.getFilePathAsync() method.
    2.1.8
    [Common]Added PCAExample.
    [Common]Updated WebCamTextureToMatHelper.cs and OptimizationWebCamTextureToMatHelper.cs(Changed several method names.).
    2.1.7
    [Common]Improved Utils.getFilePath() and Utils.getFilePathAsync().
    [Common]Improved WebCamTextureAsyncDetectFaceExample.cs.
    [Common] Fixed the const value of Calib3d class.
    2.1.6
    [Common]Fixed fastMatToTexture2D() method.
    2.1.5
    [Common]Updated to OpenCV3.2.0.
    [Common]Added fuzzy, phase_unwrapping, saliency, shape, tracking module.
    [Common]Added TrackingSample.
    [iOS]Added ios_exclude_contrib.zip for build size reduction.
    [Android]Added android_exclude_contrib.zip for build size reduction.
    2.1.4
    [Common]Changed the scene name.("Sample" to "Example")
    [Common]Fixed ArUcoTexture2DExample and ArUcoWebCamTextureExample.
    [Common]Added ConnectedComponentsExample.
    [Common]Added GreenScreenExample.
    2.1.3
    [UWP]Added OpenCVForUnityUWP_Beta3.zip
    2.1.2
    [Common]Fixed WebCamTextureToMatHelper.cs.(flipVertical and flipHorizontal flag)
    2.1.1
    [Common]Fixed OpenCVForUnityMenuItem.cs.(No valid name for platform: 11 Error)
    [Common]Added Utils.textureToTexture2D() method. [Common]Added Mat class operators. [Common]Added PolygonFilterSample.
    2.1.0
    [Common]Fixed WebCamTextureToMatHelper class.
    [Common]Added Utils.getVersion().
    [Common]Fixed Utils.getFilePathAsync().
    2.0.9
    [WebGL]Added WebGL(beta) support.(Unity5.3 or later)
    2.0.8
    [Common]Improved WebCamTextureHelper class.
    [Common]Fixed ArUcoSample.
    2.0.7
    [Common]Added aruco, structured_light, xfeatures2d module.
    [Common]Added ArUcoSample, GrabCutSample, InpaintSample, MatchShapesSample, MSERSample.
    2.0.6
    [WSA]Fixed an issue where Windows App Certification Kit fails.
    2.0.5
    [Common]Added HOGDescriptorSample.
    2.0.4
    [Android]Added Support for Split Application Binary (.OBB)
    [Android]Removed opencvforunity.jar.
    2.0.3
    [Common]Added SVMSample.
    [Common]Fixed VideoCaptureSample and WebCamTextureAsyncDetectFaceSample.
    [UWP]Added OpenCVForUnityUWP_Beta2.zip.
    2.0.2
    [Common]Fixed CS0618 warnings: `UnityEngine.Application.LoadLevel(string)' is obsolete: `Use SceneManager.LoadScene'.
    2.0.1
    [OSX]Fixed SIGILL Exception.
    [Common]Added Utils.setDebugMode() method.
    [Common]Added MatchTemplateSample, StereoBMSample, SeamlessCloneSample and WebCamTextureDetectCirclesSample.
    [Common]Added flipVertical flag, flapHorizontal flag and GetWebCamDevice() method to WebCamTextureToMatHelper.cs.
    [Common]Excluded Old Version based on “OpenCV 2.4.11”.(In order to reduce the package size)
    2.0.0
    [Common]Updated to OpenCV3.1.0.
    [Common]Included Old Version based on “OpenCV2.4.11”.
    [Common]Included Beta Version of Windows10 UWP Support.(This is beta version based on OpenCV3.0.0. opencv_contrib modules is not supported.)
    1.3.2
    [iOS]Fixed libopencvforunity.a Bitcode Setting.
    [Common]Added Beta16 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.
    1.3.1
    [Common]Fixed WebCamTextureToMatHelper.cs.(Add didUpdateThisFrame () method)
    [Common]Added Beta15 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.
    1.3.0
    [Common]Fixed WebCamTextureToMatHelper.cs.( Bug of rotation convertion from WebCamTexture to Mat in Win,Mac StandAlone Build)
    [Common] Added Beta14 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.
    1.2.9
    [Common]Added fastTexture2DToMat() and fastMatToTexture2D().
    [Common] Renewed the samples using WebCamTextureToMatHelper.(Supports all screen orientation.)
    [Common] Added Beta13 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.
    1.2.8
    [iOS]Fixed malloc_error that occurs in Unity5.3.1p2.
    [Common]Added Beta12 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.
    1.2.7
    [Common] Added Beta11 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”(Added mjpeg format support in VideoCapture class).
    1.2.6
    [iOS]Enabled Bitcode.
    [Common] Added Beta10 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.
    1.2.5
    [Common] Added Beta9 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.( Added support for Windows10 UWP.This is a test version. opencv_contrib modules is not supported.)
    1.2.4
    [Common]Fix FaceRecognizerSample.
    [Common] Delete the method using Default parameter specifiers.
    1.2.3
    [Common] Add Beta7Version of “OpenCV for Untiy” based on “OpenCV3.0.0”.
    [Common]Add WrapPerspectiveSample,HandPoseEstimationSample,MultiObjectTrackingBasedOnColorSample.
    1.2.2
    [iOS]Move “OpenCVForUnity/ iOSforXcode/opencv2.framework” to “OpenCVForUnity/Plugins/iOS/”folder.
    [iOS]Fix WebCamTexture bug of SampleScene in Unity5.2.
    1.2.1
    [Common] Add Beta5 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”(Add Linux support).
    [Common]Rewrite SampleScene.
    1.2.0
    [Common]Add Utils. getGraphicsDeviceType().
    [Common]Add SampleScene Setup Tutorial Video for Unity5.
    1.1.9
    [Common]Add CamShiftSample.(Object Tracking)
    [Common]Add OpenCVForUnityMenuItem.cs.( This script set plugin import settings automatically from MenuItem.)
    1.1.8
    [iOS] Fix problem when working with Metaio(UnityAppController problem).
    [iOS] Change file name from “OpenCVForUnity/Plugins/iOS/MyAppController.mm” to “OpenCVForUnity/Plugins/iOS/OpenCVForUnityAppController.mm”.
    [Common] Add [System.Serializable] to basic class.
    [iOS]Move “OpenCVForUnity/ iOSforXcode/iOS_BuildPostprocessor.cs” to “OpenCVForUnity/Editor”folder.
    [Common] Add Beta2 Version of “OpenCV for Untiy” based on “OpenCV3.0.0”(support Unity5).
    1.1.7
    [Common] Update to OpenCV2.4.11.
    [Common] Add Beta Version of “OpenCV for Untiy” based on “OpenCV3.0.0”(support Unity5).
    1.1.6
    [Common]Fix FaceRecognizer.cs(Compile Error had happened only in the case of Unity4 Non Pro License).
    1.1.5
    [Common]Add Beta Version of “OpenCV for Untiy” based on “OpenCV3.0.0-rc1”(support Unity5).
    [Android]Fix Utils.getFilePath().
    [Common]Add WebCamTextureAsyncDetectFaceSample.
    [iOS]Change folder name from “OpenCVForUnity/iOS for Xcode/” to “OpenCVForUnity/iOSforXcode/”.Add iOS_BuildPostprocessor.cs.
    1.1.4
    [Common]Add FaceRecognizer subclass.
    [Common]Add FaceRecognizerSampleScene.
    [Common]Fix SampleScene.
    1.1.3
    [Common]Fix SampleScene.
    [Common]Change Property of Platform Dependent Compilation from UNITY_IPHONE to UNITY_IOS.
    1.1.2
    [Common]Fix the direction of rotation of the mat that is converted from WebCamTexture.
    1.1.1
    [Common]Add OpticalFlowSampleScene.
    [Common]Fix SampleScene.
    [Common]Fix function name of CvANN_MLP_TrainParams class.
    1.1.0
    [Common]Divide asset for Unity4 and Unity5.
    1.0.9
    [Common]Support for Unity5.
    1.0.8
    [Common]Update to OpenCV2.4.10
    1.0.7
    [iOS]Support for arm64 build target.(Unity 4.6.1p3 or higher)
    [Common]Add Constructor VideoCapture(string filename).(Support for Win,Mac,iOS)
    [Common]Add Method copyToMat(),copyFromMat().
    1.0.6
    [Android]Support for x86 build target.(Unity 4.6 or higher)
    1.0.5
    [Common] Bug fixes SampleScene.
    1.0.4
    [Common]Add Method matToTexture2D( Mat mat, Texture2D texture2D, Color32[] bufferColors = null).
    1.0.3
    [Common]Support for preview in the Editor.(Pro only)
    [Common]Support for Win & Mac Standalone.(Pro only)
    [Android]Change of location of the file that you want to use for Utils.getFilePath().Changed to use“Aseets/StreamingAssets/” folder.
    [iOS] Add the file that you want to use for Utils.getFilePath() to Xcode project is no longer required.Changed to use“Aseets/StreamingAssets/” folder.
     

    Attached Files:

    Last edited: Aug 20, 2018
    MODev likes this.
  2. Subartik

    Subartik

    Joined:
    Nov 12, 2014
    Posts:
    7
    is this using native openCV on android or through Java JNI
     
  3. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    JNI is only being used to copy the files in the Unity of ”StreamingAssets” folder from ”assets” folder of Android to "/ data / data / ****** / files /" folder.
    Other processing is all native openCV.
     
  4. Subartik

    Subartik

    Joined:
    Nov 12, 2014
    Posts:
    7
    Thanks. What version of openCV? Does it rely on the opencv manager?
     
  5. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    The current version of OpenCV is 2.4.9.
    OpenCV Manager does not use. "OpenCV for Unity" uses the OpenCV to static.
     
  6. sysper

    sysper

    Joined:
    Oct 11, 2014
    Posts:
    18
    Hi,
    I would like to draw chin line by myself.
    So, can I get chin line or face line with "OpenCV for Unity".
     
  7. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    There is no API to directly detect the face of parts to ”OpenCV for Unity”.
    (API Reference OpenCV Java 2.4.9)
    You be able to detect the position of the face using the detectMultiScale ().
     
  8. Mr.Stein

    Mr.Stein

    Joined:
    Dec 4, 2013
    Posts:
    169
    Hi with this assets i can make an augment reality project?
    is there some example in this assets on of how to do based on images recognition?

    thx.
     
    devdeeplocal likes this.
  9. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Released Version 1.0.4

    Version changes
    1.0.4
    [Common]Add Method matToTexture2D( Mat mat, Texture2D texture2D, Color32[] bufferColors = null).

    @ bufferColors Optional array to receive pixel data.
    * You can optionally pass in an array of Color32s to use in colors to avoid allocating new memory each frame.
    * The array needs to be initialized to a length matching width * height of the texture (http://docs.unity3d.com/ScriptReference/WebCamTexture.GetPixels32.html)
     
  10. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Only this asset is difficult, but it can be realized by combining with other asset.

    Where I've tested, "OpenCV for Unity" worked successfully together with Vuforia.

    Please move the "Plugins/x86/opencvforunity.dll" in the same folder as the "QCARWrapper.dll".

    It was successful in image processing using the "OpenCV for Unity" against Texture2D that was acquired from ReadPixels () in OnPostRender ().

    Exemple code:
    void OnPostRender ()
    {
    UnityEngine.Rect rect = new UnityEngine.Rect (0, 0, cameraTexture.width, cameraTexture.height);
    cameraTexture.ReadPixels (rect, 0, 0, true);

    Utils.texture2DToMat (cameraTexture, cameraMat);

    Core.transform(cameraMat, cameraMat, mSepiaKernel);
    Core.putText (cameraMat, "OpenCV for Unity SEPIA " + cameraTexture.width + "x" + cameraTexture.height, new Point (5, cameraTexture.height - 5), Core.FONT_HERSHEY_PLAIN, 1.0, new Scalar (255, 0, 0, 255));

    Utils.matToTexture2D (cameraMat, outputTexture, colors);
    }


    Please see the screen shot of when running in iPhone5.

    Origial Sepia Pixelize
    写真 2014-11-15 19 00 10.png 写真 2014-11-15 19 06 51.png 写真 2014-11-15 19 12 00.png
     
    Last edited: Jan 26, 2015
  11. ztstephenson

    ztstephenson

    Joined:
    Nov 18, 2014
    Posts:
    1
    By chance do you have a postbuild script to copy the framework into the xcode project automatically?
     
  12. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    You can easily automate By using ”XCodeEditor-for-Unity”.
    https://github.com/kyubuns/XCodeEditor-for-Unity

    *.projmods example
    {
    "group": "OpenCVForUnity",
    "libs": [],
    "frameworks": [],
    "headerpaths": [],
    "files": ["Editor/iOS/OpenCVForUnity/opencv2.framework"],
    "folders": [],
    "excludes": ["^.*.meta$", "^.*.mdown^", "^.*.pdf$"],
    "buildSettings": {
    }
    }
     
  13. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Released Version 1.0.5

    Version changes
    1.0.5
    [Common] Bug fixes SampleScene.
     
  14. evanchen

    evanchen

    Joined:
    Oct 5, 2014
    Posts:
    2
    Is there a way to achieve better fps on lower end android devices? I am trying to make a real time face tracking app using this.
     
    Last edited: Dec 5, 2014
  15. Subartik

    Subartik

    Joined:
    Nov 12, 2014
    Posts:
    7
    We have purchase the plugin and it works great.
    two questions,
    1. are the GPU functions supported? I could not find them
    2. Are you going to support openCV 3.0?
     
  16. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Can speed up the process by adjusting the parameters of the sample scenes.
    • by adjusting the parameters of detectMultiScale ().(minSize etc.)
    • Using a smaller size of webCamTexture.
     
  17. evanchen

    evanchen

    Joined:
    Oct 5, 2014
    Posts:
    2
    Thank you very much for the answer. Now I've encountered another problem, as it's been set up in the sample scene, opencv can only track WebCamTexture input form one direction (only work on landscape mode, if the camera is upside down or rotated 90 degrees, it wounldn't work). I know it's a common issue, but most of the examples I've found in the web don't seen to help much as they are setup in other environments and I am a beginner in opencv and programming.

    At this stage I only need it to show correct results in portrait mode.Could you please point me to the right direction to solve this problem or provide more documentations of the asset in the future?
     
  18. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    1. are the GPU functions supported? I could not find them
    GPU functions does not currently support.
    Because "OpenCV Java" does not support GPU functions, I have not decided whether to add the GPU instruction in the future.

    2. Are you going to support openCV 3.0?
    I plan to continue to sequentially version up.
     
  19. sloopidoopi

    sloopidoopi

    Joined:
    Jan 2, 2010
    Posts:
    242
    Hi,
    looks awesome.
    (Have sent you a PM.)
     
  20. z_blush

    z_blush

    Joined:
    Apr 16, 2014
    Posts:
    8
    Hi, could you clarify what is going on here ^^^. Unity docs say to use WWW to read files in StreamingAssets (because they are in jar), then it sounds like you are writing the contents elsewhere? Is this done during construction of the cascadeclassifier?

    I ask because xml's used by cascade classifiers tend to be really large and for whatever reason they are not compressed in apk. I'd like to compress them but I don't really see a way to do this and have it work with your plugin.

    Thanks
     
  21. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984

    By changing the sample code as follows, you can show correct results in portrait mode.
    Can be detected object from another direction by rotating the Mat converted from WebCamTexture.


    Edit fix the memory leak that occurs in rgbaMat.t().
    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3.  
    4. using OpenCVForUnity;
    5.  
    6. namespace OpenCVForUnitySample
    7. {
    8.     /// <summary>
    9.     /// WebCamTexture detect face sample.
    10.     /// </summary>
    11.     public class RoteteWebCamTextureDetectFaceSample : MonoBehaviour
    12.     {
    13.    
    14.         WebCamTexture webCamTexture;
    15.         Color32[] colors;
    16.         #if (UNITY_ANDROID || UNITY_IPHONE) && !UNITY_EDITOR
    17.         bool isFront = false;
    18.         #endif
    19.         int width = 640;
    20.         int height = 480;
    21.         Mat rgbaMat;
    22.         Mat RotatedRgbaMat;
    23.         Mat grayMat;
    24.         Texture2D texture;
    25.         CascadeClassifier cascade;
    26.         MatOfRect faces;
    27.         bool initDone = false;
    28.    
    29.         // Use this for initialization
    30.         void Start ()
    31.         {
    32.             // Checks how many and which cameras are available on the device
    33.             for (int cameraIndex = 0; cameraIndex < WebCamTexture.devices.Length; cameraIndex++) {
    34.            
    35.                 #if (UNITY_ANDROID || UNITY_IPHONE) && !UNITY_EDITOR
    36.                 if (WebCamTexture.devices [cameraIndex].isFrontFacing == isFront) {
    37.                     #endif
    38.                
    39.                     Debug.Log (cameraIndex + " name " + WebCamTexture.devices [cameraIndex].name + " isFrontFacing " + WebCamTexture.devices [cameraIndex].isFrontFacing);
    40.                
    41.                     //Set the appropriate fps
    42.                     webCamTexture = new WebCamTexture (WebCamTexture.devices [cameraIndex].name, width, height, 3);
    43.                
    44.                     #if (UNITY_ANDROID || UNITY_IPHONE) && !UNITY_EDITOR
    45.                     break;
    46.                 }
    47.                 #endif
    48.            
    49.             }
    50.        
    51.             Debug.Log ("width " + webCamTexture.width + " height " + webCamTexture.height + " fps " + webCamTexture.requestedFPS);
    52.        
    53.        
    54.             // Starts the camera
    55.             webCamTexture.Play ();
    56.        
    57.        
    58.             StartCoroutine (init ());
    59.        
    60.        
    61.         }
    62.    
    63.         private IEnumerator init ()
    64.         {
    65.             while (true) {
    66.                 //If you want to use webcamTexture.width and webcamTexture.height on iOS, you have to wait until webcamTexture.didUpdateThisFrame == 1, otherwise these two values will be equal to 16. (http://forum.unity3d.com/threads/webcamtexture-and-error-0x0502.123922/)
    67.                 if (webCamTexture.didUpdateThisFrame) {
    68.                     Debug.Log ("width " + webCamTexture.width + " height " + webCamTexture.height + " fps " + webCamTexture.requestedFPS);
    69.                
    70.                     colors = new Color32[webCamTexture.width * webCamTexture.height];
    71.                
    72.                     rgbaMat = new Mat (webCamTexture.height, webCamTexture.width, CvType.CV_8UC4);
    73.  
    74.                     //replace width and height
    75.                     RotatedRgbaMat = new Mat (webCamTexture.width, webCamTexture.height, CvType.CV_8UC4);
    76.                     grayMat = new Mat (webCamTexture.width, webCamTexture.height, CvType.CV_8UC1);
    77.                
    78.                     texture = new Texture2D (webCamTexture.height, webCamTexture.width, TextureFormat.RGBA32, false);
    79.                
    80.                     gameObject.transform.eulerAngles = new Vector3 (0, 0, 0);
    81.                     gameObject.transform.localScale = new Vector3 (webCamTexture.height, webCamTexture.width, 1);
    82.                
    83.                
    84.                     cascade = new CascadeClassifier (Utils.getFilePath ("haarcascade_frontalface_alt.xml"));
    85.                     faces = new MatOfRect ();
    86.                
    87.                     gameObject.GetComponent<Renderer> ().material.mainTexture = texture;
    88.                
    89.                     initDone = true;
    90.                
    91.                     break;
    92.                 } else {
    93.                     yield return 0;
    94.                 }
    95.             }
    96.         }
    97.    
    98.         // Update is called once per frame
    99.         void Update ()
    100.         {
    101.             if (!initDone)
    102.                 return;
    103.        
    104.             if (webCamTexture.didUpdateThisFrame) {
    105.            
    106.                 Utils.WebCamTextureToMat (webCamTexture, rgbaMat, colors);
    107.  
    108. //                //rotate by -90 degrees.
    109. //               Core.flip(rgbaMat.t(),RotatedRgbaMat,1);
    110.  
    111. //fix the memory leak that occurs in rgbaMat.t().
    112.                 //rotate by -90 degrees.
    113.                 using (Mat transposeRgbaMat = rgbaMat.t()) {
    114.                       Core.flip (transposeRgbaMat, RotatedRgbaMat, 1);
    115.                 }
    116.            
    117.            
    118.                 Imgproc.cvtColor (RotatedRgbaMat, grayMat, Imgproc.COLOR_RGBA2GRAY);
    119.                 Imgproc.equalizeHist (grayMat, grayMat);
    120.            
    121.            
    122.                 if (cascade != null)
    123.                     cascade.detectMultiScale (grayMat, faces, 1.1, 2, 2, // TODO: objdetect.CV_HAAR_SCALE_IMAGE
    124.                                              new Size (50, 50), new Size ());
    125.            
    126.                 OpenCVForUnity.Rect[] rects = faces.toArray ();
    127.                 for (int i = 0; i < rects.Length; i++) {
    128.                     //                Debug.Log ("detect faces " + rects );
    129.                  
    130.                     Core.rectangle (RotatedRgbaMat, new Point (rects .x, rects .y), new Point (rects .x + rects .width, rects .y + rects .height), new Scalar (255, 0, 0, 255), 2);
    131.                 }
    132.  
    133.              
    134.                 Utils.matToTexture2D (RotatedRgbaMat, texture, colors);
    135.              
    136.              
    137.             }
    138.         }
    139.      
    140.         void OnDisable ()
    141.         {
    142.             webCamTexture.Stop ();
    143.         }
    144.      
    145.         void OnGUI ()
    146.         {
    147.             float screenScale = Screen.width / 240.0f;
    148.             Matrix4x4 scaledMatrix = Matrix4x4.Scale (new Vector3 (screenScale, screenScale, screenScale));
    149.             GUI.matrix = scaledMatrix;
    150.          
    151.          
    152.             GUILayout.BeginVertical ();
    153.             if (GUILayout.Button ("back")) {
    154.                 Application.LoadLevel ("OpenCVForUnitySample");
    155.             }
    156.          
    157.          
    158.             GUILayout.EndVertical ();
    159.         }
    160.     }
    161. }
     
    Last edited: Jan 3, 2015
    Mikael-H likes this.
  22. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    In order to read a file in Android NDK, files in the StreamingAssets folder will be copied to the "/ data / data / ****** / files /" folder.

    Copy from ”assets” folder of Android to "/ data / data / ****** / files /" folder is done when is called "Utils.getFilePath ()".
    cascade = new CascadeClassifier (Utils.getFilePath ("haarcascade_frontalface_alt.xml"));

    Copy processing procedure​
    1. Check whether File that is specified in the "Utils.getFilePath ()" is present in "/ data / data / ****** / files /" folder.
    2. if false,copy file.
    3. if true,Check whether the file has been updated. If file is updated, again to copy file.

    Though I'm not sure whether it is possible to reduce the size,
    It might be possible if decompress the compressed file that was placed in ”StreamingAssets” folder before calling "Utils.getFilePath ()".
     
  23. z_blush

    z_blush

    Joined:
    Apr 16, 2014
    Posts:
    8
    Thanks for the explanation, that helps a lot!

    "Though I'm not sure whether it is possible to reduce the size,
    It might be possible if decompress the compressed file that was placed in ”StreamingAssets” folder before calling "Utils.getFilePath ()"."

    I think the StreamAssets "folder" is read-only on the device... (I'll double check). So, I think if I wanted to compress the xmls in streamingassets I'd have to decompress straight to /data/data/***/files/ with a custom function and then *not* use Utils.getFilePath() in CascadeClassifier ctor.
     
  24. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Report Thank you.
    If you manage directly "/data/data/***/files/" without using "Utils.getFilePath ()" as you wrote, I think that it works correctly.
     
  25. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Released Version 1.0.6

    Version changes
    1.0.6
    [Android]Support for x86 build target.(Unity 4.6 or higher)
     
  26. sloopidoopi

    sloopidoopi

    Joined:
    Jan 2, 2010
    Posts:
    242
    Hi,
    i try to combine OpenCV for Unity with the Kinect v2 Example that Microsoft provides.
    In general it works but i have some performance issues as with the current API i have only the possibility to convert the data to and from OpenCV via a Texture2D.
    Microsoft's example code provides the data of the Kinect as buffers (byte[], ushort[])
    I extended the openCV4Unity API with a Method Utils.intPtrToMat(IntPtr,Mat) to create a Mat this way:
    Code (CSharp):
    1. var pColorHandle = GCHandle.Alloc(pColorBuffer,GCHandleType.Pinned);
    2. Utils.intPtrToMat(pColorHandle.AddrOfPinnedObject(),rgbMat);
    3. Do some ImgProc stuff with rgbMat ....
    4.  
    The main performance problem seems to be the re-conversion from a Mat to a buffer (byte[])
    I had two version.
    The first version works as expected but is slow:
    Code (CSharp):
    1. Utils.matToTexture2D(rgbMat,myTexture2D);
    2. var handle = GCHanlde.Alloc(myTexture2D.GetPixels32(),GCHandleType.Pinned);
    3. Marshal.Copy(handle.AddrOfPinnedObject(),pColorBuffer,0,pColorBuffer.Length);
    4. handle.Free();
    The second version is fast but the image that i get from the buffer is flipped horizontally:
    Code (CSharp):
    1. Marshal.Copy((IntPtr)rgbMat.dataAddr(),pColorBuffer,0,pColorBuffer.Length);
    It seems that the Mat.dataAddr() Method provides the data in a wrong way.

    Is there a better or faster way to convert the data with IntPtr?

    And my feature request is : Extended API to exchange data via IntPtr directly in the right way (without hacking it by myself ).

    Thanks
     
  27. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Thank information about Combining "Kinect" and "OpenCV for Unity".

    Since OpenCV Mat and Unity Texture2D are the origin of the image is different, cv::flip(flipcode = 0) has been called in OpenCVForUnity_MatToTexture() and OpenCVForUnity_TextureToMat() C++ side.

    Copy OpenCV Mat to the pixel data array IntPtr Please refer to the code below.
    Code (CSharp):
    1. /**
    2.      * Copy OpenCV Mat to the pixel data array IntPtr.
    3.      * <p>
    4.      * <br>This function copy the OpenCV Mat to the pixel data array IntPtr.The copied byte array has been up and down flip.
    5.      * <br>The input Mat object has to be of the types 'CV_8UC4' (RGBA) , 'CV_8UC3' (RGB) or 'CV_8UC1' (GRAY).
    6.      * <br>The pixel data array has to be 4byte per pixel.
    7.      * <br>The pixel data array has to be of the same size as the input Mat'.
    8.      *
    9.      * @param mat The input Mat object has to be of the types 'CV_8UC4' (RGBA) , 'CV_8UC3' (RGB) or 'CV_8UC1' (GRAY).
    10.      * @param intPtr The pixel data array has to be 4byte per pixel.The pixel data array has to be of the same size as the input Mat'.
    11.      */
    12.                 public static void matToIntPtr (Mat mat, IntPtr intPtr)
    13.                 {
    14.                         if (mat != null)
    15.                                 mat.ThrowIfDisposed ();
    16.        
    17.                         if (mat == null)
    18.                                 throw new ArgumentNullException ("mat");
    19.                         if (intPtr == null)
    20.                                 throw new ArgumentNullException ("intPtr");
    21.        
    22.        
    23.                         #if UNITY_PRO_LICENSE || ((UNITY_ANDROID || UNITY_IPHONE) && !UNITY_EDITOR)
    24.            
    25.                         OpenCVForUnity_MatToTexture(mat.nativeObj, intPtr);
    26.        
    27.                         #else
    28.                         return;
    29.                         #endif
    30.        
    31.        
    32.                 }
    33.  
    34.                 /**
    35.      * Copy the pixel data array IntPtr to OpenCV Mat.
    36.      * <p>
    37.      * This function copy the pixel data array IntPtr to the OpenCV Mat.The copied byte array has been up and down flip.
    38.      * <br>The pixel data array has to be 4byte per pixel.
    39.      * <br>The output Mat object has to be of the same size as the pixel data array'.
    40.      * The output Mat object has to be of the types 'CV_8UC4' (RGBA) , 'CV_8UC3' (RGB) or 'CV_8UC1' (GRAY).
    41.      *
    42.      * @param intPtr The pixel data array has to be 4byte per pixel.
    43.      * @param mat The output Mat object has to be of the same size as the pixel data array'.
    44.      * The output Mat object has to be of the types 'CV_8UC4' (RGBA) , 'CV_8UC3' (RGB) or 'CV_8UC1' (GRAY).
    45.      */
    46.                 public static void intPtrToMat (IntPtr intPtr, Mat mat)
    47.                 {
    48.                         if (mat != null)
    49.                                 mat.ThrowIfDisposed ();
    50.        
    51.                         if (intPtr == null)
    52.                                 throw new ArgumentNullException ("intPtr == null");
    53.                         if (mat == null)
    54.                                 throw new ArgumentNullException ("mat == null");
    55.  
    56.    
    57.                         #if UNITY_PRO_LICENSE || ((UNITY_ANDROID || UNITY_IPHONE) && !UNITY_EDITOR)
    58.        
    59.                         OpenCVForUnity_TextureToMat( intPtr, mat.nativeObj);
    60.        
    61.                         #else
    62.                         return;
    63.                         #endif
    64.        
    65.                 }

    if rgbaMat.isContinuous() is true,I think that the code below is faster than matToIntPtr().
    Code (CSharp):
    1. Core.flip (rgbaMat,rgbaMat,0);
    2. Marshal.Copy((IntPtr)rgbaMat.dataAddr(),colorBytes,0,colorBytes.Length);

    In the next version I would like to add method as described above.
     
  28. sloopidoopi

    sloopidoopi

    Joined:
    Jan 2, 2010
    Posts:
    242
    Many thanks for the info and your code!
    I made a test with both versions and it works. (As you said the direct Marshal.Copy method is a tiny bit faster in my case)
    The only thing that seems to need a change is your (IntPtr == null )check. The compiler (VS) complains about it . Should be (intPtr == IntPtr.Zero) .
     
  29. jococo

    jococo

    Joined:
    Dec 15, 2012
    Posts:
    231
    Can this be used to detect if a texture2d contains a human standing in a particular position? The research I have done makes me think a HOG algorithm to do this perhaps?
     
  30. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    By using the "OpenCV for Unity", you can detect human by using the HOGDescriptor class.
    http://docs.opencv.org/java/org/opencv/objdetect/HOGDescriptor.html
     
    jococo likes this.
  31. sloopidoopi

    sloopidoopi

    Joined:
    Jan 2, 2010
    Posts:
    242
    Hi,
    i have a nother problem with the IntPtrToMat conversion. It only works with CV_8U2 & CV_8U4 . When i try to use a Mat with type CV_U8,CV_U8C1 or CV_U8C3 the application crashes when i call Utils.intPtrToMat().

    Here my testcode:
    Code (CSharp):
    1.  
    2. const int        cDepthWidth  = 512;
    3. const int        cDepthHeight = 424;
    4. Mat testMat1 = new Mat(cDepthHeight, cDepthWidth, CvType.CV_8UC1);
    5. Mat testMat4 = new Mat(cDepthHeight, cDepthWidth, CvType.CV_8UC4);
    6.  
    7. byte[]buffer1 = Enumerable.Repeat((byte)255, cDepthWidth * cDepthHeight * 1).ToArray();
    8. byte[]buffer4 = Enumerable.Repeat((byte)255, cDepthWidth * cDepthHeight * 4).ToArray();
    9.  
    10.  var handle1 = GCHandle.Alloc(buffer1, GCHandleType.Pinned);
    11.   Utils.intPtrToMat(handle1.AddrOfPinnedObject(), testMat1);//crash
    12.  
    13. var handle4 = GCHandle.Alloc(buffer4, GCHandleType.Pinned);
    14.   Utils.intPtrToMat(handle4.AddrOfPinnedObject(), testMat4);//works
    15.  
    16.  
     
  32. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    The src pixel data array of Utils.intPtrToMat() has to be 4byte per pixel.
    In the next version I plan to improve the method.

    If one pixel is not a 4-byte you can handle with code such as the following.
    Code (CSharp):
    1. //testMat1.isContinuous = true
    2. Marshal.Copy(buffer1,0,(IntPtr)testMat1.dataAddr(),buffer1.Length);
     
  33. sloopidoopi

    sloopidoopi

    Joined:
    Jan 2, 2010
    Posts:
    242
    Many thanks for the info!

    When i use the Marshal.Copy way for Input and Output it works now with CV_8U
    Code (CSharp):
    1.  
    2. Marshal.Copy(buffer1,0, (IntPtr)testMat1.dataAddr(), buffer1.Length);// Buffer to Mat
    3. //ImgProc.....
    4. Marshal.Copy((IntPtr)testMat1.dataAddr(), buffer1,0, buffer1.Length);// Mat to Buffer
    5.  
    (The MatToIntPtr method for reading back also crashes.)
     
    EnoxSoftware likes this.
  34. roycn

    roycn

    Joined:
    Dec 24, 2013
    Posts:
    7
    1) I have one question can addon (Non-Rigid Face Tracking)?
    2) Another run in pc is ok but i mac cannot work(miss material and web cam capture)?
     
  35. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    1) A simple method to the ”Non-Rigid Face Tracking” is not implemented.

    2) "WebCamTextureToMatSample.scene" is not work?
    Please tell me the system environment.
     
    Last edited: Jan 7, 2015
  36. roycn

    roycn

    Joined:
    Dec 24, 2013
    Posts:
    7
    IMG_3315.JPG IMG_3315.JPG IMG_3315.JPG
     

    Attached Files:

  37. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Please check whether "***.unity" is added to "Scene In Build".
     
    Last edited: Jan 8, 2015
  38. roycn

    roycn

    Joined:
    Dec 24, 2013
    Posts:
    7
    I have one pc and one imac ! same step in pc is work but mac is not work,why?
     
  39. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    If look at the screenshot that you have attached,
    "Level 'Texture2DtoMatSample' (-1) could not be loaded because it has not been added to the build settings." Error Did not appear in the Console?
     
  40. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
  41. ianjgrant

    ianjgrant

    Joined:
    Mar 11, 2013
    Posts:
    28
    Hi, can I perform 'silhouette extraction' with Open CV for Unity? I'm looking to automatically create a still png texture for a sprite from an RGB video frame-grab that creates something like this (without the detail hightlights). I understand I could use the 'BackgroundSubtractor' classes or follow an algorithm using thresholds, noise reduction and contour detection as described here: http://stackoverflow.com/questions/...thm-opencv-active-contour-not-working-so-well . Can Open CV for Unity do that! Any help / suggestions would be gratefully received.

    Kind regards, Ian

     
    ina likes this.
  42. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Because I am not familiar with the image processing, I do not know whether correct advice, I think it is good to use the "BackgroundSubtractor", which is designed for video.Please look for an example of using the "OpenCV Java".

    However, Because the ”cvSnakeImage()” that are used in the link is not implemented in a clone source "OpenCV Java", it is not implemented in "OpenCV for Unity".
     
  43. sloopidoopi

    sloopidoopi

    Joined:
    Jan 2, 2010
    Posts:
    242
    Look at Imgproc.findContours() . Depending on your camera you first need to do some background substraction otherwise it is hard for OpenCV to know what is foreground and what is background. The easiest way is to use the Kinect as you get a nice user shilouette right out of the box. I made some test with Kinect 2 and OpenCV4Unity and some Triangulation :
     

    Attached Files:

  44. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    681
    do you need to utilize opencv in this case though ? kinect/2/ extracts usable silohouette anyway, doesn't it ?
    I'd say to use kinect is the easiest way to do this, but it's not always available...
     
  45. sloopidoopi

    sloopidoopi

    Joined:
    Jan 2, 2010
    Posts:
    242
    With opencv you can extract the contour data for further processing. So you can simplify it for example or use this data for triangulation. If you only need a raw binary image from your silhouette you can use the Kinect image directly.
     
  46. r618

    r618

    Joined:
    Jan 19, 2009
    Posts:
    681
    I c, that's a good way, yes, if one needs a mesh of the silhouette
     
  47. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Released Version 1.0.7

    Version changes
    1.0.7
    [iOS]Support for arm64 build target.(Unity 4.6.1p3 or higher)
    [Common]Add Constructor VideoCapture(string filename).(Support for Win,Mac,iOS)
    [Common]Add Method copyToMat(),copyFromMat().
     
    Last edited: Jan 18, 2015
  48. orAnc

    orAnc

    Joined:
    Jan 20, 2015
    Posts:
    5
    First of all,
    thx for your plugins

    please convert this code -> OpencvForUnity wrap class


    vector<vector<Point>>contours;
    vector<Vec4i>hierarchy;
    findContours(imageIn,contours,hierarchy,CV_RETR_CCOMP,CV_CHAIN_APPROX_SIMPLE);
    for(intidx=0;idx>=0;idx=hierarchy[idx][0])
    {
    Scalarcolor(rand()&255,rand()&255,rand()&255);
    drawContours(imageOut,contours,idx,color,CV_FILLED,8,hierarchy);
    }

    =================================================

    //rgbaMat => init() -> rgbaMat = new Mat (webCamTexture.height, webCamTexture.width, CvType.CV_8UC3);

    Mat hierarchy = new Mat();
    List<MatOfPoint> contours = new List<MatOfPoint>();

    Imgproc.findContours (rgbaMat, contours,hierarchy, Imgproc.RETR_CCOMP, Imgproc.CHAIN_APPROX_SIMPLE); ----> error occur (CvException: CvType.CV_32SC2 != m.type() || m.cols()!=1)

    for (int i = 0; i >= 0; i = ???) // ??? -> i wrote -> (int)hierarchy.get(i,0)[0]
    {
    Scalar color = new Scalar (Random.Range (0, 255), Random.Range (0, 255), Random.Range (0, 255));
    Imgproc.drawContours(rgbaMat,contours,i,color);
    }

    Please Save ME!
     
    Last edited: Jan 20, 2015
  49. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    984
    Mat image of Imgproc.findContours() must to be CvType.CV_8UC1.
    Since "OpenCV for Unity" is a clone of "OpenCV Java", Please look for an example of OpenCV java.

    Code (CSharp):
    1. grayMat = new Mat (webCamTexture.height, webCamTexture.width, CvType.CV_8UC1);
    2. thresholdMat = new Mat();
    3.  
    4. Imgproc.threshold (grayMat, thresholdMat, 0, 255, Imgproc.THRESH_BINARY | Imgproc.THRESH_OTSU);
    5.  
    6. Mat hierarchy = new Mat();
    7. List<MatOfPoint> contours = new List<MatOfPoint>();
    8.  
    9. Imgproc.findContours (thresholdMat, contours,hierarchy, Imgproc.RETR_CCOMP, Imgproc.CHAIN_APPROX_SIMPLE);
    10.  
    11. for( int i = 0; i< contours.Count; i++ )
    12. {
    13.     Scalar color = new Scalar (Random.Range (0, 255), Random.Range (0, 255), Random.Range (0, 255));
    14.     Imgproc.drawContours(grayMat,contours,i,color);
    15. }
     
    itouhirohumi and lab_ui_um like this.
  50. Visual_Lies

    Visual_Lies

    Joined:
    Nov 3, 2014
    Posts:
    17
    I just wanted to confirm before purchasing this asset that this would be functional with the CalibrateCamera function to estimate the camera position based off points on a planar surface as described here:

    docs.opencv.org/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html