[Tutorial] Using C++ OpenCV within Unity

Hi martejpad, just wondering, how did you get this to build on Mac? Could you give a walkthrough? I don't even see monodevelop show up as an option when configuring CMake before clicking generate.

 

Hey @Ggabe, so sorry I just saw your message. What do you mean exactly by "how did you get this to build on Mac?" Have you followed the instructions of this post? https://forum.unity.com/threads/tutorial-using-c-opencv-within-unity.459434/#post-3036971

Let me know

 

Hi, thank you for that tuto.
Yet, I keep having this error :
Error:A problem occurred configuring project ':app'.
> executing external native build for cmake C:\Users\NC_ESM\AndroidStudioProjects\MyApplication\app\CMakeLists.txt
I do not know why !
I have been looking around, it seems that the issues might come from the fact that I am running on 32 bit. What do you think ?
Cheers

 

Great tutorial thanks for sharing.
Has anyone been successful in getting this to work with the Hololens?

 

I face a problem when I run in the unity. The unity opencv interop sample which is used to display the streaming display black screen the whole time. It would help a ton if anyone can help me on this issue.

 

Hello MountainBorn,

Can you elaborate on how we would go about displaying video footage directly from OpenCV within Unity by passing an array of pixels? As mentioned from Part 3 in the paragraph below

"This same procedure can be used to pass an array of pixels between OpenCV and Unity without having to copy it, allowing you to display video footage from OpenCV within Unity, or passing a WebcamTexture stream to OpenCV for processing. That is beyond the scope of this part, however."


Thanks! If anyone else knows how to go about this I would greatly appreciate it.

 

I read your post but have no clue how to make it work in Mac. I installed cmake but it doesnt work with visual studio, tried to use mono but dont even know where is the installation path. I open cmake and ask me for Unix makefiles generator and as compiler doesnt work with visual studio. Also everything is in red in cmake. Also there is no opencv version for mac, i just used the source code in cmake. Could you explain me how you did it?

 

Hi @dienat, you don't need to download anything besides Android Studio, where you'll do all the development of your plugin, and OpenCV4Android, which contains all the OpenCV libraries you need to link from your Android Studio project. You don't need a specific version for Mac of OpenCV. The post I linked contains all the relevant links and step by step instructions on how to do everything using these tools. Please give it a try and let me know how it goes.

 

Hi Thomas
I was following your tutorial and I am facing an error which is :
DllNotFoundException: UnityOpenCVSample
FacedetectionScript.Start () (at Assets/Scripts/FacedetectionScript.cs:46)
can you please help

 

Thank you, this really helped me get to grips with setting up and importing a dll in general let alone Unity, I would recommend everyone who has this burning quest,to follow these links
Couple of things for any future linkers:
You will likely get it wrong a couple of times following this, maybe not - but I did, but that's to do with people and comprehension and subject specific knowledge and not the author - although It is worth mentioning that you should be building Debug and Release from ALL_BUILD from the VS project, compiled after being cmake'd.
And dont delete the orignal extract of openCV from your machine, you may (likely will) need it to build upon what has been done or reduce need for unused library files for optimization later on.
Anyway... Thanks again @Thomas-Mountainborn

 

Hi all! Crossed all my "T"s and dotted all my "I"s when following the tutorial; however my built .dll file does not seem to work with the Unity project.

Downloading @Thomas-Mountainborn 's sample project and utilizing his .dll file in my plugins/scripts works like a charm and the imshow debug window pops up; however, the .dll built from following the tutorial, the test scene starts but no imshow debug window pops up.

Has anyone had a similar issue where their built .dll does not work but the sample one does in Unity? I would sincerely appreciate a nudge in the right direction - been toying around but can't understand why this is happening.

EDIT 1: clarified issue
EDIT 2: Solved the issue! As suggested by the tutorial, I utilized Dependency Walker to analyze the .dll I built and the sample project's .dll . Turns out I was missing certain dependencies when using my .dll (which indicates I may have followed the tutorial incorrectly). The solution was to import all the release and debug dlls from the OpenCV library into my project. Doing so got the imshow debug window to pop up.

 

I tackled OpenCV+Unity to iOS based upon the guides of @Thomas-Mountainborn and @Bonfire-Boy . Sharing my process of porting the OpenCV sample project here, for posterity, and in particular for future beginners such as myself.

1) Adding OpenCV 3.1.0 to your XCode project
2) Changes to Source.cpp code and Unity code

1) Adding OpenCV 3.1.0 to your XCode project
Bonfire-Boy lays down all the guidelines for getting setup, so I'll detail the steps for those who are not as familiar.

The opencv 3.1.0 framework may be found on https://opencv.org/releases.html . Upon downloading the framework, add the framework to your XCode Project by navigating to "Build Phases -> Link Binary With Libraries". Additionally, you will need to add 5 frameworks. Simply click on the plus sign under "Link Binary With Libraries" and add the following:

• AssetsLibrary
• CoreGraphics
• CoreMedia
• CoreFoundation
• Accelerate

Having done the above - now we can add the search paths to our XCode project. This step requires 3 paths: Framework search path, Header search path, and Library search path. Navigate to your XCode project's "Build Settings -> Search Paths" to locate the appropriate fields.

1. Framework Search Paths: the path leading to your opencv2.framework file (this may be in your Downloads if you downloaded the framework from opencv's website)
   
2. Header Search Paths: the path leading to the opencv header files. Upon compiling OpenCV on my OS X machine, the path to the opencv header files was /usr/local/include
   
3. Library Search Paths: the path leading to the opencv library. Upon compiling OpenCV on my OS X machine, the path to the opencv lib was /usr/local/lib

That should do the trick! I encountered issues with my opencv2.framework download (did not support armv7 architecture) and ultimately had to compile a patched version of OpenCV 3.1.0 following these instructions:
https://github.com/opencv/opencv/pull/7266)

2) Changes to Source.cpp code and Unity code
Good work on getting this far!

In order for the Source.cpp code to register your lbpcascade/haarcascade file - we need to do two things. First, we must add the cascade file to our XCode project's Supporting Files. Then we must change our Source.cpp file to a .mm (Objective-C++) file in order to reference the cascade file location in our built project.

To add the cascade file, use your XCode Project's navigation window to find "Unity-iPhone Test -> Supporting Files". Right click on "Supporting Files" and click on "Add Files to..." Having done so, we must then go to the project's "Build Phases -> Copy Bundle Resources" tab. Add the cascade file there as well. This will add the cascade to our project when built, and allow it to be referenced within the app bundle.

Find the Source.cpp file within your XCode project, and change it's Type to "Default - Objective-C++ source". Don't fret! Objective-C++ files may contain C++ code as well. We will now add some Objective-C++ code to be able to reference the cascade file path.

// store the bundle we are working with

NSBundle * appBundle = [NSBundle mainBundle];



// constant file name

NSString * cascadeName = @"lbpcascade_frontalface";

NSString * cascadeType = @"xml";



// get file path in bundle

NSString * cascadePathInBundle = [appBundle pathForResource: cascadeName ofType: cascadeType];

Add this block of code between the "using namespace cv;" line and the "struct Circle" line in your code.
Next, prior to the ' extern "C" int Init( ' declaration in your code, we must convert the NSString variable to one which C++ can understand. Prior to the Init method declaration, add this line:

std::string cascadePath([cascadePathInBundle UTF8String]);

Now, find the following statement within the Init method:

if (!_faceCascade.load(......))

And change it to

if (!_faceCascade.load(cascadePath))

Your code will now know where the cascade file is located!
XCode may throw up errors in regards to certain namespaces, simply add "cv::" to those cropping up errors.

------

Once again, thank you to Bonfire-Boy and Thomas-Mountainborn, very grateful for their posts and I hope this one helps future readers as well.

 

Grabbed Thomas' project zip... Has anyone been able to get this to work in Unity 2018.1.0f2?

Runs without errors and loads opencv input prompt, but white screen is shown in Unity.

When I select allow unsafe in project settings, this error occurs:
NullReferenceException: Object reference not set to an instance of an object
PositionAtFaceScreenSpace.Update () (at Assets/Scripts/PositionAtFaceScreenSpace.cs:14)

 

Actually the latest version is OpenCV 3.4.1 - has anyone tried with that

 

The sample project worked for me using Unity 2017.4.1f1, that being said, the error you've encountered cropped up for me previously as well. Line 14 in PositionAtFaceScreenUpdate references the OpenCVFaceDetection script - so the latter script is actually where your error is coming from.

When you say "loads opencv input prompt", do you mean the Unity OpenCV Interop Sample window pops up? What do you see in that window?

 

Actually - can you share your iOS port/wrap

 

How did you resolve the error of nullObjectReference ?

 

@martejpad - First, thank you so much for the tutorial! I haven't been able to find anything like it. I think I found a small mistake here. According to this post jniLibs goes in app/src/main.

It also states that this method is depreciated because OpenCV uses Async initialization. I was curious if Async initialization would be a good alternative?

 

Hi

I hope someone can help me here. I followed Thomas tutorial about OpenCV and Unity because I want to use them in a project, unfortunately i can not have the same results.

When run the unity i have the following error "failed to open a camera stream" but the thing I do not get the same windows you have.

I am new to both OpenCV and Unity. Can anyone help me with the issues or guide me to the path to follow?

Thank you in advance.

 

First, @Thomas-Mountainborn, thanks for the tutorial.
But have you ever tried to use EmguCV with Unity? Already is a C# port of OpenCV, so maybe would be just a matter of import the dlls as plugins inside Unity?

I'm thinking of using OpenCV/EmguCV for hand gestures recognition with Kinect. If somebody knows a free plugin for kinect/unity let me know, please.

 

I've tried EmguCV before switching to C++ - it was far too slow for real time use.I don't know if it has progressed since then, but from my experience it's not usable.

 

@Thomas-Mountainborn, do you remebmer how did you use EmguCV with Unity? Is just to copy the dlls?
Because they coast $ 399 at asset store! I would like to try it before. (Note: I'm not experienced in image processing)

I will try Vitruvius Framework for Kinect, which is very impressive even at free versions.

But I would like to learn how to implement my own algorithms for face / hand detection and use in Kinect too, even if they are kind of slow in Unity, it's just for academic purporses (fow now at least)

 

I definitely just used the code from Emgu directly, not any asset.

 

Hello @Thomas-Mountainborn

I really want to make a game for IOS and Android. Will openCV work on these platforms?

 

Hey @Thomas-Mountainborn
First i want to say Thank You for the great and detailed Tutorial on implementing OpenCV to Unity3D.
By the way it still works out fine with the recently released OpenCV 4.0 stable version.
Mentioning 4.0 how is that fourth part doing? [ no pressure intended ]
We struggled a lot in a recent project with some of the implementations from the AssetStore .. performance issues mainly ... and your tutorial might be our saviour .. if we could get the image feed insinde Unity.

Keep up the good work!

 

Thank you Thomas-Mountainborn and martejpad for sharing the work you did to get OpenCV working with Unity. I'm taking a hybrid approach of using Thomas-Mountainborn's C# and C++ code for face detection and martejpad's explanation on how to make an Android .so library, but I'm getting a linker error when I try to build the library.

error: undefined reference to 'cv::CascadeClassifier::detectMultiScale(cv::_InputArray const&, std::__ndk1::vector<cv::Rect_<int>, std::__ndk1::allocator<cv::Rect_<int> > >&, double, int, int, cv::Size_<int>, cv::Size_<int>)'

If I comment out the call
_faceCascade.detectMultiScale(resizedGray, faces);
everything builds fine.

Since it can build just including a CascadeClassifier object, I suspect the issue may have to do with the cascade classifier .xml that Thomas-Mountainborn mentions in part 3 of his tutorial, which file I can't find. Does anyone know how to help the linker find the reference? Thanks.

 

This has been a great resource for developers who are looking at developing AR applications using OpenCV with Unity.
Current options available have commercial implications (Vuforia , OpencvForUnity from asset store etc).

Once again thanks to martejpad for not only sharing the steps but also persistently guiding individuals who have been struggling to implement.

I was able to implement the OpenCV - Unity plugin for Android as per guidelines given by martejpad , but it kept me thinking that how can I reduce my development cycle by testing my features on Unity Editor as against testing it on Android. So here I am sharing the steps I performed for creating successful plugin for MAC Osx.

Building Unity plugin for Mac Osx

How to use OpenCV functions in Unity Editor (on MAC OS) using native Plugin without using 3rd party plugin from assets store like openCVforUnity.

Prerequisite – Xcode and CMake are installed on your machine. These are trivial installation.
Xcode can be installed from App store
CMake - follow steps provided at this site.

Build and Install OpenCV

Download openCV and unzip it somewhere on your computer. Create build folder inside it.
Open CMake application
Click Browse Source and navigate to your openCV folder.
Click Browse Build and navigate to your build Folder.
Click the configure button. You will be asked how you would like to generate the files.
Choose Unix-Makefile from the Drop Down menu and Click OK.
CMake will perform some tests and return a set of red boxes appear in the CMake Window.
Click Configure again, then Click Generate.​

Goto build folder

Code (CSharp):

1. # cd build
2. # make
3. # sudo make install
4.  

This will install the opencv libraries on your computer.​

Create Plug-In using Xcode
Now we will create a native Plugin for Mac to be used in Unity as Plugin.

Please start Xcode
Choose MacOS> Framework & Library> Bundle. "Product Name" is set to "UnityPlugin".
Use packagae “com.hmi”
Change bundle identifier to “com.hmi.opencvplugin”
Select UnityPlugin in the navigation view

Go to "Build Settings" → "Header Search Path" field (enter the details without collapsing the “Header Search Paths”) and enter "/usr/local/include" (the opencv header files are stored here as part of build that we performed in earlier step)

Go to "Build Settings" → "Library Search Path" field (enter the details without collapsing the “Header Search Paths”) and enter "/usr/local/lib" (the opencv *opencv*.dyib files are stored here as part of build that we performed in earlier step)

Go to "Build Settings" → "Other Linker Flags" field, enter “-lopencv_dnn -lopencv_ml -lopencv_objdetect -lopencv_shape -lopencv_stitching -lopencv_superres -lopencv_videostab -lopencv_calib3d -lopencv_features2d -lopencv_highgui -lopencv_videoio -lopencv_imgcodecs -lopencv_video -lopencv_photo -lopencv_imgproc -lopencv_flann -lopencv_core”. These are all the linked libraries of opencv. Validate this with following command on terminal window

Code (CSharp):

1. # pkg-config --libs --cflags opencv

Next, create a new cpp file (OpenCVPlugin.cpp) Deslect “Also create header file” for use as a Native Plugin.
Now create a sample opencv function as shown below

Code (CSharp):

1. #include <stdio.h>
2. #include <string>
3.  
4. #include <opencv2/opencv.hpp>
5. #include <opencv2/core.hpp>
6. #include <opencv2/imgproc.hpp>
7.  
8. using namespace cv;
9.  
10. extern "C" {
11.    int TestFunction_Internal();
12. }
13.  
14. int TestFunction_Internal() {
15.     return 12345;
16. }

Note - For simplicity I have just added TestFunction here to show how PlugIn functions. Please feel free to add your opencv image operation features as per your need. I can share more details on how to write these functions in my separate thread.

Select "Product" → "Build" from the menu bar and build the project, UnityPlugin.bundle will be created in the Products folder.​

Installing plugin in Unity
If you have reached this stage, Congratulations !!, ..you are now few minutes away from your final test.

Please create a demo project in Unity.
Go to your project window and create a "Plugins" folder under the Asset.
Create Osx folder under assets → Plugins folder. (you will have different OS types here.)
Insert "UnityPlugin.bundle" that you had created in earlier steps into Osx folder. You can simply drag and drop directly from Xcode's Products folder to Unity. ( or go to Xcode editor and right click OpenCVPlugin.bundle and select show in folder. From there you can drag it into Unity)

Create Scripts folder under assets.
Under this Scripts folder you can create Debug and Native folders respectively. (Assets → Scripts → Debug and Assets → Scripts → Native)
Now we create new c# controller.cs script in Debug folder. This script will consist of passing info back and forth between managed C# and native C++ code.

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.UI;
3.  
4. using System.Runtime.InteropServices;
5. using System;
6.  
7. /*
8.     Controller provides examples for passing pointers for byte arrays back and forth
9.     between managed C# and native C++. When making your own classes, be sure to specify
10.     the Texture2D texture format before attempting to load raw bytes into it.
11. */
12. public class Controller : MonoBehaviour
13. {
14.     void Awake()
15.     {
16.     Debug.LogWarning("Test Value in C++ is " + NativeLibAdapter.Test());
17.     }
18.  
19.     void Update()
20.     {
21.     }
22. }

You can see that it calls Test function from Native library. Now we create new C# script NativeLibAdapter.cs in Assets → Scripts → Native folder

Code (CSharp):

1. using System.Runtime.InteropServices;
2. using System;
3.  
4. using UnityEngine;
5.  
6. /*
7.     NativeLibAdapter is an example communication layer between managed C# and native C++
8. */
9. public class NativeLibAdapter
10. {
11. #if !UNITY_EDITOR
12.     [DllImport("native-lib")]
13.     private static extern int TestFunction_Internal();
14. #elif UNITY_EDITOR
15.     [DllImport("OpenCVPlugin")]
16.     private static extern int TestFunction_Internal();
17. #endif
18.  
19.  
20.     public static int Test()
21.     {
22. #if !UNITY_EDITOR
23.         return TestFunction_Internal();
24. #elif UNITY_EDITOR
25.         return TestFunction_Internal();
26. #else
27.         return -1;
28. #endif
29.     }
30. }

You will see that the script now has fine control on which library to link based on where you are running your player. (Unity Editor or Android i.e. Non Unity editor). I am assuming that you have build the native-lib as per instruction provided by martejpad.
Create a cube 3D object under your scene and drop controller.cs script on to it.
Now if you run this project in Unity editor, you should see in your console "Test Value in C++ is 12345"

Please let me know if you are able to follow these steps. If not, I can edit these based on where you stumble upon.
Happy coding !!​

 

In case you are looking for complete source code, I have uploaded it at git repo.

 

Great tutorial @Thomas-Mountainborn! Almost 4 years after it was made and it's still relevant and working pretty well.
However, I have encountered an error. I managed to compile the dll just fine and putting them in unity (under the Plugins folder). But now every time I run it I get this error:

EntryPointNotFoundException: Init

OpenCVFaceDetection.Awake () (at Assets/OpenCVFaceDetection.cs:50)

The line it's referring to specifically is this one:

Code (CSharp):

1. int result = OpenCVInterop.Init(ref camWidth, ref camHeight);

I don't know what is causing this. It seems like it can find the dll just fine, but for some reason it cannot acces the Init?
What do you think? Im trying to implement this parallel to Vuforia, so I can use OpenCV for AR.

 

Did you also include all the other OpenCV dlls alongside your own in the Plugins folder?

 

Yes, all the ones I found in bin/Release.

 

I don't know if this has anything to do, @Thomas-Mountainborn, but if I try to build the project with the script in the Third Step, I get a bunch of LNK2001 errors. It compiles just fine, but the dll isnt created unless I first build the script without anything from this point forward.

Code (CSharp):

1. CascadeClassifier _faceCascade;
2. String _windowName = "Unity OpenCV Interop Sample";
3. VideoCapture _capture;
4. int _scale = 1;

I did set the path for the permanent OpenCV location, as well as the dependencies in the project. The only change I made was swapping 310 with 401, since my version is different from yours.

 

Hello @martejpad,

Did you solve the problem that combines Vuforia with OpenCV in unity?
I'm also studying in using Vuforia within Unity and want to use image processing by OpenCV first. But I'm really a
beginner with unity and Vuforia so that I don't know how to continue to make it come true. And if you have any idea please feel free to give your comments.
Sorry for my bad English and bothering. Thanks for your assistance.

 

I had the same linker error in Android Studio and what worked for me was changing my NDK path to the path inside my Android sdk folder. I use android-ndk-r16b.
The path can be changed in Android Studio: File -> Project Structure -> SDK Location.

 

i'm using opencv with unity.there is an object following the movements of face detection circle on x,y,z axises.and i wanna prevent y,z axises and just move only for x axis here is the code;
using UnityEngine;
public class PositionAtFaceScreenSpace : MonoBehaviour
{
private float _camDistance;
void Start()
{
_camDistance = Vector3.Distance(Camera.main.transform.position, transform.position);
}
void Update()
{
if (OpenCVFaceDetection.NormalizedFacePositions.Count == 0)
return;
transform.position = Camera.main.ViewportToWorldPoint(new Vector3( OpenCVFaceDetection.NormalizedFacePositions[0].x, OpenCVFaceDetection.NormalizedFacePositions[0].y, OpenCVFaceDetection.NormalizedFacePositions[0].z));
}


}
///////////////////////////////////////////////////(opencv)
/////////////////////////////
using System.Collections;
using System.Collections.Generic;
using System.Runtime.InteropServices;
using UnityEngine;
public class OpenCVFaceDetection : MonoBehaviour {
public static List<Vector3> NormalizedFacePositions { get; private set; }
public static Vector2 CameraResolution;
public float DepthDevider = 10f;
public float DepthAdder = 10f;
private const int DetectionDownScale = 1;
private bool _ready;
private int _maxFaceDetectCount = 5;
private CvCircle[] _faces;
void Start()
{
int camWidth = 0, camHeight = 0;
int result = OpenCVInterop.Init(ref camWidth, ref camHeight);
if (result < 0)
{
if (result == -1)
{
Debug.LogWarningFormat("[{0}] Failed to find cascades definition.", GetType());
}
else if (result == -2)
{
Debug.LogWarningFormat("[{0}] Failed to open camera stream.", GetType());
}
return;
}
CameraResolution = new Vector2(camWidth, camHeight);
_faces = new CvCircle[_maxFaceDetectCount];
NormalizedFacePositions = new List<Vector3>();
OpenCVInterop.SetScale(DetectionDownScale);
_ready = true;
}
void OnApplicationQuit()
{
if (_ready)
{
OpenCVInterop.Close();
}
}
void Update()
{
if (!_ready)
return;
int detectedFaceCount = 0;
unsafe
{
fixed (CvCircle* outFaces = _faces)
{
OpenCVInterop.Detect(outFaces, _maxFaceDetectCount, ref detectedFaceCount);
}
}
NormalizedFacePositions.Clear();
for (int i = 0; i < detectedFaceCount; i++)
{
NormalizedFacePositions.Add(new Vector3((_faces.X * DetectionDownScale) / CameraResolution.x, 1f - ((_faces.Y * DetectionDownScale) / CameraResolution.y), 1f - (_faces.Radius*DetectionDownScale)/DepthDevider + DepthAdder));
}
}
}
internal static class OpenCVInterop
{
[DllImport("OpenCVThomasPart2Try1")]
internal static extern int Init(ref int outCameraWidth, ref int outCameraHeight);
[DllImport("OpenCVThomasPart2Try1")]
internal static extern int Close();
[DllImport("OpenCVThomasPart2Try1")]
internal static extern int SetScale(int downscale);
[DllImport("OpenCVThomasPart2Try1")]
internal unsafe static extern void Detect(CvCircle* outFaces, int maxOutFacesCount, ref int outDetectedFacesCount);
}

[StructLayout(LayoutKind.Sequential, Size = 12)]
public struct CvCircle
{
public int X, Y, Radius;
}

 

i'm using opencv with unity.there is an object following the movements of face detection circle on x,y,z axises.and i wanna prevent y,z axises and just move only for x axis here is the code;

 

Hi, @Thomas-Mountainborn, i followed your tutorials, everything is going well and completed all parts. But in the end, I get these errors. I am developing for Hololens and use x86 bits for all library, placed .rsp files, change <Allowunsafeblocks> and everything. But still getting errors. Can you help?

 

You can now set allow unsafe code directly in Unity's Player settings > Other. Can you see if this fixes it?

 

Hi, I don't see that option. I'm using Unity 2017.4.16f1 (64-bit)

 

Ah, it looks like it might be new for Unity 2018. Can you update the project, or create a test project to check it out?

 

i updated unity, unsafe code problem is solved, yay! thanks
however, more errors now. I got DllNotFoundException and NullReferenceException.... i have all dlls inside Plugins folder though. Can you help?

 

Hmm, looks like you built the 32 bit version of OpenCV and your .dll. I'm not sure if that's the issue, but it's worth trying the x64 version instead, and see if Unity does recognize that one.

 

Tried x64, still getting same errors, DllNotFoundException

 

Just to be sure, did you recompile all of OpenCV to be x64? Are you getting any other errors in Unity? I just saw that you actually got an error about using x86 plugins in your above screenshot.

 

Hey @Thomas-Mountainborn

I am struggling to get decent frames using OpenCV in a project at the moment. I'm processing some frames and using the DNN module to detect things with my yolo model.

Will this approach significantly increase performance? And if I build OpenCV with Cuda enabled will I be able to take advantage of my GPU?

 

Processing frames on the CPU is indeed a very slow affair. Using CUDA can give a serious speed boost.

I've been looking into sharing the camera feed between Unity and OpenCV using a render texture so that pixels don't have to be copied over. Sadly my OpenCV crashes when doing anything with DirectX on the laptop on which I have to run these tests - I'll write an article once I get it working.

 

Looking forward to the article. If there is an email newsletter i'll sign up.

I think I will resort to using OpenCVSharp with Cuda enabled Dlls and see how I go for now though.

 

@Thomas-Mountainborn, first, thanks for the tutorial, it has been great. I'm trying your Sample, and have imported it into Unity 2019. I'm having the same problem that @Slaifir has, and maybe some of the other problems that @ed-spatialsauce has with OperCVFaceDetection.Start() and PositionAtFaceScreenSpace.Update()

EntryPointNotFoundException: Init
OpenCVFaceDetection.Start () (at Assets/Scripts/OpenCVFaceDetection.cs:22)

and subsequent errors every update:

NullReferenceException: Object reference not set to an instance of an object
PositionAtFaceScreenSpace.Update () (at Assets/Scripts/PositionAtFaceScreenSpace.cs:14)

Any idea what could be causing this?

 

Hey @Thomas-Mountainborn did you have an ETA on your next tutorial?

 

thanks for the tutorial @Thomas-Mountainborn ,
great tutorial and great results (y),
still waiting for part 4