[Tutorial] Using C++ OpenCV within Unity

Help for Android build

Code (CSharp):

1. CMake Error: CMake can not determine linker language for target: native-lib
2. Failed to compute build system.
3. CMake Error at C:\Users\A\AndroidStudioProjects\OPENCV2\app\src\main\cpp\CMakeLists.txt:13 (add_library):
4.   Cannot find source file:
5.  
6.     app/src/main/cpp/native-lib.cpp
7.  
8.   Tried extensions .c .C .c++ .cc .cpp .cxx .m .M .mm .h .hh .h++ .hm .hpp
9.   .hxx .in .txx
10.  
11.  
12. CMake Error: CMake can not determine linker language for target: native-lib
13. Failed to compute build system.

Tried all, how to provide path for that cpp file

 

Hi Thomas,
I have downloaded your file from the unity forum. So far i have run your file and looks like every thing is fine but I am getting a blank window. So, please help me out. More over I have added the camera.dll to plugins (C:\Users\*****\Downloads\OpenCV Sample\Assets\Plugins\camera.dll).

 

Hi everyone,
Who ever knows the solution for my issue can help me out.
Thank you

 

Hi everyone,
Can we import Unity files to Iphone using windows PC or we need a Mac and Xcode for it.

Thank you

 

Thank you so much for this

 

perfect tutorials!! It works, thank you very much! @Thomas-Mountainborn
btw,I'm not this man,ahhh.It's a video

 

Any detailed tutorial please on how to efficiently transfer the frame from C++ to C#/Unity, please?

 

Hey Thomas, this is an incredibly detailed tutorial for anyone who's getting into OpenCV for Unity! Thanks a lot for your effort and the work that you put into it!

I've managed to setup a working version of your tutorial using Unity 2020.1 and OpenCV 4.5 and created a repository on GitHub, just in case it could help anyone else too. Here's the link! (Also, if that's not apropriate please notify me and I'll take down the repository, but honestly all credit's due to you and if you have a GitHub and wanna send me your user that'd be great!)

 

Great idea, thanks for setting that up! I'm always glad that the tutorials are still helping out three years down the road. I've linked to your repo at the end of the last part, so people can find it there. My Git handle is ThomasMountainborn, but there's nothing on there.

 

Hi @iseta, thanks for putting together updated versions of this project! I'm new to Unity, could you explain how to import your repo to Unity?

Edit: I just copied over the folders from a repo into a Unity project I made, that seems to work fine.

 

Great tutorial! Thanks Thomas.


I got two errors in unity and finally solved them, hope this can help you guys :


DllNotFoundException: OpenCVCPlusTest
OpenCVFaceDetection.Start () (at Assets/OpenCVFaceDetection.cs:23)
This is because not all the required .dll are copied to the Plugins folder, use Dependency Walker to check which .dll is missing, as Thomas suggested in his part 3 tutorial.
In my case, in C++ project, I use $(OPENCV_DIR)\lib\Debug and $(OPENCV_DIR)\lib\Release, opencv_core310d.lib and opencv_core310.lib, opencv_highgui310d.lib and opencv_highgui310.lib at the same time, which causes the compiled .dll incorrectly depends on .dll in \OpenCV 3.1\bin\Debug folder. Try delete $(OPENCV_DIR)\lib\Debug, opencv_core310d.lib, opencv_highgui310d.lib in C++ project.


[OpenCVFaceDetection] Failed to open camera stream.
UnityEngine.Debug:LogWarningFormat(String, Object[])
OpenCVFaceDetection:Start() (at Assets/OpenCVFaceDetection.cs:32)
This is beacuse a camera hardware is requred. If you don't have a camera , Try go to C++ project ,change

_capture.open(0); 

to

_capture.open("C:/Users/yourAccout/Desktop/yourVideo.mp4");  

to read a specific video clip.

 

Thanks to everyone for all of your contributions. This is one of the best OpenCV for Unity resources out there, especially for native Android development. I am starting work on an updated plugin using OpenCV4Android 4.5.0. hopefully I will post some updates.

I am considering using both the Android Studio and Visual Studio/Xamarin approaches. I was wondering if anyone had any opinions on whether or not Android Studio was still the best way to go,?

 

Thankyou once again to everyone for their incredibly helpful contributions (Thomas, martejpad, inder2, mcelroy-jon, iseta and my apologies to anyone I may have have missed out). Here is a brief update including some notes that will hopefully prove useful to someone else taking the same journey. I also have soem questions that someone may be able to shine some light upon. If so I would be extremely grateful.

1) Correct OpenCV SDK Python build_sdk.py script build parameters.
2) Use of OpenCV functions in Android e.g. (VideoCapture, imread, selectROI)
3) Input Unity VideoPlayer frames into OpenCV as Texture2D. Can't get this to work or any other form of video input to OpenCV in android.

I have made some progress over the Christmas holidays. I have managed to utilise a lot of OpenCV functionality via the Windows plugin and I have had slightly more limited success using the Android equivalent despite martejpad's excellent notes that got a brief shout out in a Thomas Hallberg video at 3:13:

Here is my configuration:
OpenCV: 4.5.0
Unity: 2020.1.7f1
Android Studio: 4.1.1

(1) I have gone down the route of using the OpenCV 4.5.0 sdk out of the box and I haven't built the libraries from source or included the open_contrib content. I had a brief play with this going down both the CMake and python script routes. I wasn't entirely sure of the correct parameters for the Git clone GitHub\opencv\platforms\android\build_sdk.py python script. If anyone knows these I would appreciate any input. I did use --ndk_path and --sdk_path

I had some trouble with getting Android Studio to build the native-lib so library reliably and found the following video very useful indeed with an overview and some useful diagrams and tips.

I found the following two lines particularly useful:

Code (CSharp):

1. set(OpenCV_DIR $ENV{OPENCV_ANDROID}/sdk/native/jni)

this sets up the OpenCV root directory used in conjunction with environment variables.

and

Code (CSharp):

1. find_package(OpenCV REQUIRED)

really useful for automatically using the sdk to configure settings.

Here are the settings that I use more often than not for CMakeLists:

Code (CSharp):

1. #opencv
2. set(CMAKE_BUILD_TYPE Release)
3. set(CMAKE_CXX_FLAGS "-std=c++11")
4. set(OpenCV_STATIC on)
5. set(OpenCV_DIR $ENV{OPENCV_ANDROID}/sdk/native/jni)
6.  
7. message(STATUS "OpenCV library status1:")
8. message(STATUS "OpenCV SubDir: ${OpenCV_CONFIG_SUBDIR}")
9. message(STATUS "CMAKE CURRENT_LIST_FILE: ${CMAKE_CURRENT_LIST_FILE}")
10. message(STATUS "OpenCV library status2:")
11. message(STATUS "    version: ${OpenCV_VERSION}")
12. message(STATUS "    libraries: ${OpenCV_LIBS}")
13. message(STATUS "    include path: ${OpenCV_INCLUDE_DIRS}")
14.  
15. find_package(OpenCV REQUIRED)
16.  
17. # Creates and names a library, sets it as either STATIC
18. # or SHARED, and provides the relative paths to its source code.
19. # You can define multiple libraries, and CMake builds them for you.
20. # Gradle automatically packages shared libraries with your APK.
21. add_library( # Sets the name of the library.
22.         native-lib
23.  
24.         # Sets the library as a shared library.
25.         SHARED
26.  
27.         # Provides a relative path to your source file(s).
28.         opencv-utils.cpp
29.         native-lib.cpp )
30.  
31. # Searches for a specified prebuilt library and stores the path as a
32. # variable. Because CMake includes system libraries in the search path by
33. # default, you only need to specify the name of the public NDK library
34. # you want to add. CMake verifies that the library exists before
35. # completing its build.
36. find_library( # Sets the name of the path variable.
37.         log-lib
38.  
39.         # Specifies the name of the NDK library that
40.         # you want CMake to locate.
41.         log )
42. #find_library(jnigraphics-lib jnigraphics)
43.  
44.  
45. # Specifies libraries CMake should link to your target library. You
46. # can link multiple libraries, such as libraries you define in this
47. # build script, prebuilt third-party libraries, or system libraries.
48. target_link_libraries( # Specifies the target library.
49.         native-lib
50.  
51.         # Links the target library to the log library
52.         # included in the NDK.
53.         ${OpenCV_LIBS}
54.        # ${jnigraphics-lib}
55.         ${log-lib}
56.         log )

(2) I found it very difficult to use any of the OpenCV library calls for accessing media very difficult such as
VideoCapture for videos and
imread for images &
selectROI for easy bounding rect UI selection of ROI
all 3 were very straightforward in windows.
If anyone has any information on this front I would appreciate it.

I found the best way to achieve input via camera or video is via the Unity library and passing the data through to OpenCV via the methods largely explained above:
For camera use the following excellent video for setup

and then after you have your WebCamTexture backCam playing use the following with the SubmitFrame native C++ call. Code from the guys thanked above:

Code (CSharp):

1. Color32[] pixels = backCam.GetPixels32();
2.                 GCHandle pixelHandle = GCHandle.Alloc(pixels, GCHandleType.Pinned);
3.  
4.                 IntPtr results = NativeLibAdapter.SubmitFrame(backCam.width, backCam.height,
5.                     pixelHandle.AddrOfPinnedObject());
6.  
7.                 int bufferSize = backCam.width * backCam.height * 4;
8.                 byte[] rawData = new byte[bufferSize];
9.  
10.                 if (results != IntPtr.Zero)
11.                 {
12.                     Marshal.Copy(results, rawData, 0, bufferSize);
13.                     outTexture.LoadRawTextureData(rawData);
14.                     outTexture.Apply();
15.                 }
16.                 background.texture = outTexture;
17.  
18.                 rawData = null;
19.                 pixelHandle.Free();

(3) For videos I haven't manage to get it to work properly with Unity's VideoPlayer. I changed mp4 format to enconding with H-265 without audio using VLC and imported the file to streamingassets in Unity to enable playing on an android device. The VideoPlayer input is rendered to a render texture and displayed on a RawImage within Unity. I have tried the following:

Code (CSharp):

1.  
2. dest = new Texture2D(1280, 720, TextureFormat.RGBA32, false);
3. videoPlayer.Play();
4. /////////
5. dest.Apply(false);
6. Graphics.CopyTexture((videoPlayer.texture as RenderTexture), dest);
7. //
8. Color32[] pixels = dest.GetPixels32();

and then used the webcam approach but I am not getting any output although the IntPtr results are populated. The output data appears blank. I have tested displaying the dest texture on a separate raw image and it is rendered correctly?

Android threshold screenshots:

 

Hey all, I'm late to the game here but I see a lot of good information. Has anyone been able to find a real solution to loading cascade files with the Android plugin? Copying them manually to the Android folder and using the persistentData path wouldn't do for a real world situation, but I feel like there has to be a way to do this.

PS: anyone trying to compile an android .so library from the most recent version of Visual Studio 2019, you can ignore the weird __fp16 errors in arm_neon.h and build; it will work, despite those errors, as long as everything else is correct (and the proper ndk matches up to the version of OpenCV -- I used 4.01 with r16b)

EDIT: Figure it out -- you have to write the file to the persistent data path with code, then recopy the name of its location.

Code (CSharp):

1.  string filePath = System.IO.Path.Combine(Application.streamingAssetsPath, fileName);
2.         string result = "";
3.         if (filePath.Contains("://") || filePath.Contains(":///"))
4.         {
5.             UnityEngine.Networking.UnityWebRequest www = UnityEngine.Networking.UnityWebRequest.Get(filePath);
6.             yield return www.SendWebRequest();
7.             result = www.downloadHandler.text;
8.         }
9.         else
10.             result = System.IO.File.ReadAllText(filePath);
11.         File.WriteAllText(Application.persistentDataPath + "/"+fileName, result);
12.        string cascadeFileName = (Application.persistentDataPath + "/" + fileName);
13.  
14.  

NEW QUESTION: Why does my android device, which clearly has the webcam working and showing my face, fail to find the camera stream with the code from the website tutorial (which also matches the code from "Unity 2018 Augmented Reality Projects" by Jesse Glover)?

 

Thanks! that was a great help! i was still getting DLL not found exception, and turns out i had to add the debug dlls of opencv as well, im not sure why, since i built my dll in release mode.