Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

[Tutorial] Using C++ OpenCV within Unity

Discussion in 'Scripting' started by Thomas-Mountainborn, Mar 5, 2017.

  1. AjdiNNNN


    Jan 14, 2019
    Help for Android build

    Code (CSharp):
    1. CMake Error: CMake can not determine linker language for target: native-lib
    2. Failed to compute build system.
    3. CMake Error at C:\Users\A\AndroidStudioProjects\OPENCV2\app\src\main\cpp\CMakeLists.txt:13 (add_library):
    4.   Cannot find source file:
    6.     app/src/main/cpp/native-lib.cpp
    8.   Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
    9.   .hxx .in .txx
    12. CMake Error: CMake can not determine linker language for target: native-lib
    13. Failed to compute build system.
    Tried all, how to provide path for that cpp file
  2. arvind1359


    Sep 11, 2019
    Hi Thomas,
    I have downloaded your file from the unity forum. So far i have run your file and looks like every thing is fine but 2019-09-11.png I am getting a blank window. So, please help me out. More over I have added the camera.dll to plugins (C:\Users\*****\Downloads\OpenCV Sample\Assets\Plugins\camera.dll).
    Last edited: Sep 11, 2019
  3. arvind1359


    Sep 11, 2019
    Hi everyone,
    Who ever knows the solution for my issue can help me out.
    Thank you
  4. arvind1359


    Sep 11, 2019
    Hi everyone,
    Can we import Unity files to Iphone using windows PC or we need a Mac and Xcode for it.

    Thank you
  5. blindio


    May 25, 2013
    Thank you so much for this
    Thomas-Mountainborn likes this.
  6. eliaswang


    Aug 30, 2018
    perfect tutorials!! It works, thank you very much! @Thomas-Mountainborn
    btw,I'm not this man,ahhh.It's a video:)

    Attached Files:

    Thomas-Mountainborn likes this.
  7. BNJMO


    Dec 6, 2015
    Any detailed tutorial please on how to efficiently transfer the frame from C++ to C#/Unity, please?
  8. iseta


    May 5, 2016
    Hey Thomas, this is an incredibly detailed tutorial for anyone who's getting into OpenCV for Unity! Thanks a lot for your effort and the work that you put into it!

    I've managed to setup a working version of your tutorial using Unity 2020.1 and OpenCV 4.5 and created a repository on GitHub, just in case it could help anyone else too. Here's the link! (Also, if that's not apropriate please notify me and I'll take down the repository, but honestly all credit's due to you and if you have a GitHub and wanna send me your user that'd be great!)
  9. Thomas-Mountainborn


    Jun 11, 2015
    Great idea, thanks for setting that up! I'm always glad that the tutorials are still helping out three years down the road. I've linked to your repo at the end of the last part, so people can find it there. My Git handle is ThomasMountainborn, but there's nothing on there.
    iseta likes this.
  10. Jehocifer


    Oct 28, 2020
    Hi @iseta, thanks for putting together updated versions of this project! I'm new to Unity, could you explain how to import your repo to Unity?

    Edit: I just copied over the folders from a repo into a Unity project I made, that seems to work fine.
    Last edited: Nov 2, 2020
    iseta likes this.
  11. PPP_Paws


    Dec 22, 2017
    Great tutorial! Thanks Thomas.

    I got two errors in unity and finally solved them, hope this can help you guys :

    DllNotFoundException: OpenCVCPlusTest
    OpenCVFaceDetection.Start () (at Assets/OpenCVFaceDetection.cs:23)

    This is because not all the required .dll are copied to the Plugins folder, use Dependency Walker to check which .dll is missing, as Thomas suggested in his part 3 tutorial.
    In my case, in C++ project, I use $(OPENCV_DIR)\lib\Debug and $(OPENCV_DIR)\lib\Release, opencv_core310d.lib and opencv_core310.lib, opencv_highgui310d.lib and opencv_highgui310.lib at the same time, which causes the compiled .dll incorrectly depends on .dll in \OpenCV 3.1\bin\Debug folder. Try delete $(OPENCV_DIR)\lib\Debug, opencv_core310d.lib, opencv_highgui310d.lib in C++ project.

    [OpenCVFaceDetection] Failed to open camera stream.
    UnityEngine.Debug:LogWarningFormat(String, Object[])
    OpenCVFaceDetection:Start() (at Assets/OpenCVFaceDetection.cs:32)

    This is beacuse a camera hardware is requred. If you don't have a camera , Try go to C++ project ,change; 
    to read a specific video clip.
    iseta likes this.
  12. Mannion2k


    Dec 22, 2020
    Thanks to everyone for all of your contributions. This is one of the best OpenCV for Unity resources out there, especially for native Android development. I am starting work on an updated plugin using OpenCV4Android 4.5.0. hopefully I will post some updates.

    I am considering using both the Android Studio and Visual Studio/Xamarin approaches. I was wondering if anyone had any opinions on whether or not Android Studio was still the best way to go,?
  13. Mannion2k


    Dec 22, 2020
    Thankyou once again to everyone for their incredibly helpful contributions (Thomas, martejpad, inder2, mcelroy-jon, iseta and my apologies to anyone I may have have missed out). Here is a brief update including some notes that will hopefully prove useful to someone else taking the same journey. I also have soem questions that someone may be able to shine some light upon. If so I would be extremely grateful.

    1) Correct OpenCV SDK Python script build parameters.
    2) Use of OpenCV functions in Android e.g. (VideoCapture, imread, selectROI)
    3) Input Unity VideoPlayer frames into OpenCV as Texture2D. Can't get this to work or any other form of video input to OpenCV in android.

    I have made some progress over the Christmas holidays. I have managed to utilise a lot of OpenCV functionality via the Windows plugin and I have had slightly more limited success using the Android equivalent despite martejpad's excellent notes that got a brief shout out in a Thomas Hallberg video at 3:13:

    Here is my configuration:
    OpenCV: 4.5.0
    Unity: 2020.1.7f1
    Android Studio: 4.1.1

    (1) I have gone down the route of using the OpenCV 4.5.0 sdk out of the box and I haven't built the libraries from source or included the open_contrib content. I had a brief play with this going down both the CMake and python script routes. I wasn't entirely sure of the correct parameters for the Git clone GitHub\opencv\platforms\android\ python script. If anyone knows these I would appreciate any input. I did use --ndk_path and --sdk_path

    I had some trouble with getting Android Studio to build the native-lib so library reliably and found the following video very useful indeed with an overview and some useful diagrams and tips.

    I found the following two lines particularly useful:

    Code (CSharp):
    1. set(OpenCV_DIR $ENV{OPENCV_ANDROID}/sdk/native/jni)
    this sets up the OpenCV root directory used in conjunction with environment variables.


    Code (CSharp):
    1. find_package(OpenCV REQUIRED)
    really useful for automatically using the sdk to configure settings.

    Here are the settings that I use more often than not for CMakeLists:

    Code (CSharp):
    1. #opencv
    2. set(CMAKE_BUILD_TYPE Release)
    3. set(CMAKE_CXX_FLAGS "-std=c++11")
    4. set(OpenCV_STATIC on)
    5. set(OpenCV_DIR $ENV{OPENCV_ANDROID}/sdk/native/jni)
    7. message(STATUS "OpenCV library status1:")
    8. message(STATUS "OpenCV SubDir: ${OpenCV_CONFIG_SUBDIR}")
    10. message(STATUS "OpenCV library status2:")
    11. message(STATUS "    version: ${OpenCV_VERSION}")
    12. message(STATUS "    libraries: ${OpenCV_LIBS}")
    13. message(STATUS "    include path: ${OpenCV_INCLUDE_DIRS}")
    15. find_package(OpenCV REQUIRED)
    17. # Creates and names a library, sets it as either STATIC
    18. # or SHARED, and provides the relative paths to its source code.
    19. # You can define multiple libraries, and CMake builds them for you.
    20. # Gradle automatically packages shared libraries with your APK.
    21. add_library( # Sets the name of the library.
    22.         native-lib
    24.         # Sets the library as a shared library.
    25.         SHARED
    27.         # Provides a relative path to your source file(s).
    28.         opencv-utils.cpp
    29.         native-lib.cpp )
    31. # Searches for a specified prebuilt library and stores the path as a
    32. # variable. Because CMake includes system libraries in the search path by
    33. # default, you only need to specify the name of the public NDK library
    34. # you want to add. CMake verifies that the library exists before
    35. # completing its build.
    36. find_library( # Sets the name of the path variable.
    37.         log-lib
    39.         # Specifies the name of the NDK library that
    40.         # you want CMake to locate.
    41.         log )
    42. #find_library(jnigraphics-lib jnigraphics)
    45. # Specifies libraries CMake should link to your target library. You
    46. # can link multiple libraries, such as libraries you define in this
    47. # build script, prebuilt third-party libraries, or system libraries.
    48. target_link_libraries( # Specifies the target library.
    49.         native-lib
    51.         # Links the target library to the log library
    52.         # included in the NDK.
    53.         ${OpenCV_LIBS}
    54.        # ${jnigraphics-lib}
    55.         ${log-lib}
    56.         log )
    (2) I found it very difficult to use any of the OpenCV library calls for accessing media very difficult such as
    VideoCapture for videos and
    imread for images &
    selectROI for easy bounding rect UI selection of ROI
    all 3 were very straightforward in windows.
    If anyone has any information on this front I would appreciate it.

    I found the best way to achieve input via camera or video is via the Unity library and passing the data through to OpenCV via the methods largely explained above:
    For camera use the following excellent video for setup

    and then after you have your WebCamTexture backCam playing use the following with the SubmitFrame native C++ call. Code from the guys thanked above:

    Code (CSharp):
    1. Color32[] pixels = backCam.GetPixels32();
    2.                 GCHandle pixelHandle = GCHandle.Alloc(pixels, GCHandleType.Pinned);
    4.                 IntPtr results = NativeLibAdapter.SubmitFrame(backCam.width, backCam.height,
    5.                     pixelHandle.AddrOfPinnedObject());
    7.                 int bufferSize = backCam.width * backCam.height * 4;
    8.                 byte[] rawData = new byte[bufferSize];
    10.                 if (results != IntPtr.Zero)
    11.                 {
    12.                     Marshal.Copy(results, rawData, 0, bufferSize);
    13.                     outTexture.LoadRawTextureData(rawData);
    14.                     outTexture.Apply();
    15.                 }
    16.                 background.texture = outTexture;
    18.                 rawData = null;
    19.                 pixelHandle.Free();
    (3) For videos I haven't manage to get it to work properly with Unity's VideoPlayer. I changed mp4 format to enconding with H-265 without audio using VLC and imported the file to streamingassets in Unity to enable playing on an android device. The VideoPlayer input is rendered to a render texture and displayed on a RawImage within Unity. I have tried the following:
    Code (CSharp):
    2. dest = new Texture2D(1280, 720, TextureFormat.RGBA32, false);
    3. videoPlayer.Play();
    4. /////////
    5. dest.Apply(false);
    6. Graphics.CopyTexture((videoPlayer.texture as RenderTexture), dest);
    7. //
    8. Color32[] pixels = dest.GetPixels32();
    and then used the webcam approach but I am not getting any output although the IntPtr results are populated. The output data appears blank. I have tested displaying the dest texture on a separate raw image and it is rendered correctly?

    Android threshold screenshots:
    Screenshot_20210101-184623.jpg Screenshot_20210101-184716.jpg

    Attached Files:

    neginfinity and iseta like this.
  14. SavedByZero


    May 23, 2013
    Hey all, I'm late to the game here but I see a lot of good information. Has anyone been able to find a real solution to loading cascade files with the Android plugin? Copying them manually to the Android folder and using the persistentData path wouldn't do for a real world situation, but I feel like there has to be a way to do this.

    PS: anyone trying to compile an android .so library from the most recent version of Visual Studio 2019, you can ignore the weird __fp16 errors in arm_neon.h and build; it will work, despite those errors, as long as everything else is correct (and the proper ndk matches up to the version of OpenCV -- I used 4.01 with r16b)

    EDIT: Figure it out -- you have to write the file to the persistent data path with code, then recopy the name of its location.
    Code (CSharp):
    1.  string filePath = System.IO.Path.Combine(Application.streamingAssetsPath, fileName);
    2.         string result = "";
    3.         if (filePath.Contains("://") || filePath.Contains(":///"))
    4.         {
    5.             UnityEngine.Networking.UnityWebRequest www = UnityEngine.Networking.UnityWebRequest.Get(filePath);
    6.             yield return www.SendWebRequest();
    7.             result = www.downloadHandler.text;
    8.         }
    9.         else
    10.             result = System.IO.File.ReadAllText(filePath);
    11.         File.WriteAllText(Application.persistentDataPath + "/"+fileName, result);
    12.        string cascadeFileName = (Application.persistentDataPath + "/" + fileName);
    NEW QUESTION: Why does my android device, which clearly has the webcam working and showing my face, fail to find the camera stream with the code from the website tutorial (which also matches the code from "Unity 2018 Augmented Reality Projects" by Jesse Glover)?
    Last edited: Jun 9, 2021
    iseta likes this.
  15. thunderwiring


    Aug 28, 2021
    Thanks! that was a great help! i was still getting DLL not found exception, and turns out i had to add the debug dlls of opencv as well, im not sure why, since i built my dll in release mode.