[RELEASED] OpenCV for Unity

Does the MaskRCNN example work with inceptionv3 or others?

 

Yes they provide dnn support to load trained machine learning models. Training a model involves a lot of knowledge separate from OpenCVForUnity - maybe take a machine learning course for free on kaggle

 

OpenCVForUnity supports models trained by Python and other methods. Models that work with OpenCV4.1.0 will also work with OpenCVForUnity2.3.6.
https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API

 

I tried building for IOS using xcode and I got a lot of "undefined symbol" errors, all of them have to do with OpenCV functions it seems (e.x. cv::Mat::eye(cv::Size_<int>,int)). Am I missing something? I have the latest version of opencvforunity.

 

Unfortunately, I don't have an example of using inceptionv3.

 

Thank you very much for reporting.
Could you tell me about your test environment?
Unity version :
Xcode version :

 

Xode is version 11 and Unity is 2018.3.0f
Thank you.

 

From Unity2018.3.0f1 to Unity2018.3.3f1, it is necessary to turn off "Enable Bitcode" on Xcode.

 

Thank you for the fast reply. I disabled Bitcode but I'm still getting the 100 error. Here are pictures showing the errors I have. Could it be a problem with my version of Xcode? I have xcode 11 but I haven't updated to latest version since I'm low on space.

 

Is ImportSettings properly set?



Also, It is recommended to use Xcode10.2 when using Unity2018.3.0f1.

 

Dear Enox Software, building for iOS i'm reeveing this error:
Library not loaded: @rpath/opencv2.framework/opencv2
You can find details attached.
Thanks in advance,
Simone

 

Dear Enox Software, before purchasing the asset could you please help me to understand
will it be possible to add an "OpenPose deep learning library" to the project (https://github.com/CMU-Perceptual-Computing-Lab/openpose)
and then get something like this on ios device https://github.com/sarweshshah/gait_analysis/blob/master/results/pose trail.gif
Thank you in advance

 

Thank you very much for reporting.
Could you tell me about your test environment?
Unity version :
Xcode version :

Is ImportSettings properly set?

 

OpenPoseExample is included in OpenCVForUnity.
https://github.com/EnoxSoftware/Ope...inModules/dnn/CaffeExample/OpenPoseExample.cs
for now, Since OpenCV's Dnn module uses a CPU, OpenPoseExample takes more than 900 ms to estimate human pose. Perhaps real-time processing is difficult.
But, this repository using OpenCVForUnity seems to work in real-time.
https://twitter.com/yukihiko_a/status/1131174274708910080
https://github.com/yukihiko/ThreeDPoseUnityForiOS

 

Thanks a lot for your reply!

 

I think I have this set up correctly, both files in iOS are set to iOS in the platform. I'll look into using an older Xcode.

 

Hi.
Is there any update to whether one can achieve the same as with Vuforia Image Targets? Meaning, can I predefine a sample of photos (at least 2, hopefully up to 100) and then have the camera recognise at least one photo at a time?

 

Do build settings matter when trying to with OpenCVForUnity? I have my minimum iOS version at version 9.

 

Unfortunately, I don't have an example of detecting multiple MarkerTargets.
The PatternDetector class supports only one pattern marker image. If you want to use multiple different markers, you need to use multiple PatternDetector classes.

 

Hey together,

we purchased the OpenCV package and would like to run the SFM module that is included in OpenCV 3.0+ . Can this be included in an upcoming patch of this asset?

Any suggestions how we can use SFM with Unity right now?

Best regards

 

OpenCVForUnity requires iOS version 8 or higher.

 

Since this asset is a clone of OpenCV Java 4.1.0, you are able to use the same API as OpenCV Java 4.1.0.
For now, the sfm module is not planned to be implemented.

 

I updated my project from Unity 2018 to Unity 2019.2. My file is saved on GitHub but when I try to clone it and access it from my Mac I get this error in the editor when it tries to run OpenCV.

 

Is ImportSettings of "OpenCVForUnity / Assets / OpenCVForUnity / Plugins / macOS / opencvforunity.bundle" set correctly?

 

I don't have that folder. I have a version of OpenCVForUnity from 2018, do I need to get the latest version in order for it to work?

 

Thanks for the awesome plugin.
But recently with Unity 2019.2.9f1, running any of example using camera lead to crash.

I got error:

Code (CSharp):

1. E/Camera3-OutputStream: getBufferLocked: Stream 0: Can't dequeue next output buffer: Broken pipe (-32)
2. E/Camera3-Device: RequestThread: Can't get output buffer, skipping request: Broken pipe (-32)

Previous version of Unity I worked with (2018.4.6f1) has no such problem. I tried on galaxy S6 and S8+. It would be nice if you fix this problem.

 

Could you tell me about your test environment?
Unity version :
OpenCVForUnity version :
macOS version :

 

Thank you very much for reporting.
Does this problem only occur when using OpenCVForUnity?
Also,
Does the following simple WebCamTexture code work without problems?
https://docs.unity3d.com/ScriptReference/WebCamTexture.Play.html

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. public class ExampleClass : MonoBehaviour
5. {
6.     void Start()
7.     {
8.         WebCamTexture webcamTexture = new WebCamTexture();
9.         Renderer renderer = GetComponent<Renderer>();
10.         renderer.material.mainTexture = webcamTexture;
11.         webcamTexture.Play();
12.     }
13. }

 

Ok it was problem of WebCamTexture. Thanks for answering!

 

Good afternoon! I have a problem with Android:
1) cannot get the number of frames in the video. Always returns 0
2) I can not set the desired frame. Always returns false
Although I for example can take from all cadres on waiting lists.
Under Windows 10 everything works fine
I have:
Unity: 2018.2. 21f1
Watch honor of the 20 (of the Top 9)
Samsung Galaxy S6 Edge (Android 7.0)
Thanks in advance for the answer

 

Hi,

I'm building a pupil detector using Keras model using Open CV For Unity that I purchased. I am having trouble converting the OpenCVForUnity.CoreModule.Mat object to a Numpy.NDarray that is required for the model predict method. I searched many sites and docs but cannot figure it out. Can someone please help me? Thank you.

Code (CSharp):

1.            
2. var model = Keras.Models.Model.ModelFromJson(jsonAIModel);
3. model.LoadWeight(aiModelWeights);
4. // error on next line: cannot convert from 'OpenCVForUnity.CoreModule.Mat' to 'Numpy.NDarray'
5. var result = model.Predict(image);
6.  

 

Thank you very much for reporting.
Is the example you tested a VideoCaptureExample ("768x576_mjpeg.mjpeg") ?
Also, It seems that whether the "Videoio.CAP_PROP_POS_FRAMES" property can be acquired depends on the video format.

 

I took the code from the example as a basis. Modified under its task. Now, tested on ios (Iphone 6). Everything works correctly. On Android:

Code (CSharp):

1. _video Capture.set (Videoio.CAP_PROP_POS_FRAMES, index);

return false
The same code on ios returns true
Video formats I use: avi, mp4, mov

 

Sorry, on Ios, too, not correctly fulfills. Although the ios code returns true, the end result is still incorrect.
Editor:
Mac
frameCount: 734

Code (CSharp):

1. videoCapture.set (Videoio.CAP_PROP_POS_FRAMES, UpdateFrame.First().Number) - return true

Fulfills completely correctly.

Win10:
frameCount: 734

Code (CSharp):

1. videoCapture.set (Videoio.CAP_PROP_POS_FRAMES, UpdateFrame.First().Number) - return true

Fulfills completely correctly.

Mobile:
Android:
frameCount: 0

Code (CSharp):

1. videoCapture.set (Videoio.CAP_PROP_POS_FRAMES, UpdateFrame.First().Number) - return false

Fulfills incorrectly

Ios:
frameCount: 1

Code (CSharp):

1. videoCapture.set (Videoio.CAP_PROP_POS_FRAMES, UpdateFrame.First().Number) - return true

Fulfills incorrectly

My code is frame extraction

Code (CSharp):

1. if (UpdateFrame.First().Number >= 0)
2.      _videoCapture.set(Videoio.CAP_PROP_POS_FRAMES,
3.                                      UpdateFrame.First().Number);
4.      _videoCapture.grab();
5.     _videoCapture.retrieve(_imgMat, 0);
6.     Imgproc.resize(_imgMat, _imgMat, new Size(512, 512));
7. //Creating a texture from a new frame
8. #if UNITY_EDITOR || UNITY_IOS
9.             Imgproc.cvtColor(_imgMat, _imgMat, Imgproc.COLOR_BGR2RGB);
10. #endif
11.    if (UpdateFrame.First().texture == null)
12.    {
13.         var texture = new Texture2D(_imgMat.cols(),
14.         _imgMat.rows(), TextureFormat.RGB24, false);
15.        Utils.matToTexture2D(_imgMat, texture);
16.        UpdateFrame.First().texture = texture;
17.     }
18.     else
19.     {
20.          Utils.fastMatToTexture2D(_imgMat,
21.            (Texture2D)UpdateFrame.First().texture);
22.     }
23.     UpdateFrame.Remove(UpdateFrame.First());
24.  

Maybe I'm doing something wrong? Thanks in advance for the answer^^

 

When using the mjpeg codec, we confirmed that the following example works on all platforms.
https://www.dropbox.com/s/vxeutihxekb4nhq/New_VideoCaptureExample.unitypackage?dl=0
It seems that it depends on the platform whether video files other than mjpeg support "Videoio.CAP_PROP_POS_FRAMES".

 

To convert from the Mat class to the Numpy.NDarray class, you need to set Mat data to the Numpy.NDarray class in some way.
Utils.copyFromMat () can copy byte array data from Mat.
https://enoxsoftware.github.io/Open..._utils.html#ab14e60409dd3fa505da151d92e4870c0
Does Numpy.NDarray class have a method to initialize by specifying byte array data?

 

Dear EnoxSoftware,

I am working in licence car plate recognition. I have an issue with loading classification.xml file from android and put into KNN recognition. Here is C++ version, I can't take it to work in C# in unity with opencvforunity. Is there function to read xml file and put into Mat? Thank you for your answer.

Best regards,
Luke.

cv::Mat matClassificationInts;

cv::FileStorage fsClassifications("classifications.xml", cv::FileStorage::READ); // open the classifications file

if (fsClassifications.isOpened() == false) { // if the file was not opened successfully
std::cout << "error, unable to open training classifications file, exiting program\n\n"; // show error message
return(false); // and exit program
}

fsClassifications["classifications"] >> matClassificationInts; // read classifications section into Mat classifications variable
fsClassifications.release(); // close the classifications file
// read in training classifications

 

Hello

I've been struggling with this for a couple of days now, so I was wondering if anyone can give me some guidance.
I'm following an example from the book "OpenCV by example" for text recognition which is in c++, and so far have successfully converted every part to C# and unity except this function:

Code (CSharp):

1. Mat drawER(const vector<Mat> &channels, const vector<vector<ERStat> > &regions, const vector<Vec2i>& group, const Rect& rect)
2.     {
3.        Mat out = Mat::zeros(channels[0].rows+2, channels[0].cols+2, CV_8UC1);
4.    
5.        int flags = 4                   //4 neighbors
6.            + (255 << 8)               //paint mask in white (255)
7.            + FLOODFILL_FIXED_RANGE       //fixed range
8.            + FLOODFILL_MASK_ONLY;       //Paint just the mask
9.    
10.         for (int g=0; g < group.size(); g++)
11.         {
12.            int idx = group[g][0];    
13.             ERStat er = regions[idx][group[g][1]];
14.    
15.            //Ignore root region
16.             if (er.parent == NULL)
17.                continue;
18.    
19.            //Transform the linear pixel value to row and col
20.            int px = er.pixel % channels[idx].cols;
21.            int py = er.pixel / channels[idx].cols;
22.        
23.            //Create the point and adds it to the list.
24.            Point p(px, py);    
25.    
26.            //Draw the extremal region
27.             floodFill(
28.                channels[idx], out,               //Image and mask
29.                p, Scalar(255),                   //Seed and color
30.                nullptr,                       //No rect
31.                Scalar(er.level),Scalar(0),       //LoDiff and upDiff
32.                flags                           //Flags
33.            );    
34.         }
35.    
36.        //Crop just the text area and find it's points
37.        out = out(rect);
38.    
39.        vector<Point> points;
40.        findNonZero(out, points);
41.        //Use deskew and crop to crop it perfectly
42.        return deskewAndCrop(out, minAreaRect(points));
43.     }

What it does basically is to use the floodfill algorithm (which is available in openCV for unity) to clean and then crop the rotated text areas (photos attached).


This is the explanation of that part from the book:

Fortunately, the ERFilter provides us with an object called ERStat, which contains pixels inside each extremal region. With these pixels, we can use the OpenCV floodFill function to reconstruct each letter. This function is capable of painting similar colored pixels based in a seed point, just like the bucket tool of most drawing applications.


The ERStat object is not available in "openCV for unity" so I have a hard time figuring out how to get that information from the region objects which are MatOfPoints.

I'd appreciate any help.

Thanks.

 

Since OpenCVForUnity is a clone of OpenCV Java, you are able to use the same API as OpenCV Java 4.1.0(https://docs.opencv.org/master/javadoc/). FileStorage class has not been implemented.

 

I found solution, sometimes is better to go sleep and wake up with fresh mind. Here is solution to read xml file in opencv for unity. File path I get from Application.persistentDataPath where I put xml files.

Code (CSharp):

1. Mat classificationMat = new Mat (62, 62, CvType.CV_64FC1);
2.             //            string filename = "OCRHMM_transitions_table.xml";
3.             //            FileStorage fs(filename, FileStorage::READ);
4.             //            fs["transition_probabilities"] >> transition_p;
5.             //            fs.release();
6.  
7.             //Load ClassificationData.
8.             classificationMat.put (0, 0, GetClassificationsData(filepath));
9.  
10. ///....
11. //function to load data from xml
12. double[] GetClassificationsData(string filePath)
13.     {
14.         XmlDocument xmlDoc = new XmlDocument();
15.         xmlDoc.Load(filePath);
16.  
17.  
18.         XmlNode dataNode = xmlDoc.GetElementsByTagName("data").Item(0);
19.         //            Debug.Log ("dataNode.InnerText " + dataNode.InnerText);
20.         string[] dataString = dataNode.InnerText.Split(new string[] {
21.                 " ",
22.                 "\r\n", "\n"
23.             }, StringSplitOptions.RemoveEmptyEntries);
24.         //            Debug.Log ("dataString.Length " + dataString.Length);
25.  
26.         double[] data = new double[dataString.Length];
27.         for (int i = 0; i < data.Length; i++)
28.         {
29.             try
30.             {
31.                 data[i] = Convert.ToDouble(dataString[i]);
32.             }
33.             catch (FormatException)
34.             {
35.                 Debug.Log("Unable to convert '{" + dataString[i] + "}' to a Double.");
36.             }
37.             catch (OverflowException)
38.             {
39.                 Debug.Log("'{" + dataString[i] + "}' is outside the range of a Double.");
40.             }
41.         }
42.  
43.         return data;
44.     }
45.  
46.  

 

Hi again

So I'll give more context of what I'm trying to do. I'll appreciate if anyone can tell me what I'm doing wrong.
I'm using OpenCV for Unity and am trying to scan book covers and get their title/author as strings.
This is my code so far (sorry it's a little long. I thought I'll be helpful post the whole thing):

Code (CSharp):

1. void Start()
2.     {
3.         Mat transition_p = new Mat(62, 62, CvType.CV_64FC1);
4.         transition_p.put(0, 0, GetTransitionProbabilitiesData(OCRHMM_transitions_table_filepath));
5.         Mat emission_p = Mat.eye(62, 62, CvType.CV_64FC1);
6.         string voc = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
7.         _decoder = OCRHMMDecoder.create(OCRHMM_knn_model_data_filepath, voc, transition_p, emission_p);
8.     }
9.  
10.  
11.  
12.  
13.  
14.  
15.  
16.     List<Mat> SeparateChannnels(Mat src)
17.     {
18.         List<Mat> channels = new List<Mat>();
19.         //Grayscale images
20.         if (src.type() == CvType.CV_8U || src.type() == CvType.CV_8UC1)
21.         {
22.             channels.Add(src);
23.             channels.Add(new Scalar(255) - src);
24.         }
25.  
26.         //Colored images
27.         else if (src.type() == CvType.CV_8UC3)
28.         {
29.             Text.computeNMChannels(src, channels);//, Text.ERFILTER_NM_IHSGrad);
30.             int size = channels.Count;
31.             for (int c = 0; c < size; c++)
32.                 channels.Add(new Scalar(255) - channels[c]);
33.         }
34.         return channels;
35.     }
36.  
37.  
38.     public void DetectText()
39.     {
40.         Mat frame = GetMatFromWebcam();
41.         Mat original = new Mat(frame.size(), frame.type());
42.         frame.copyTo(original);
43.  
44.         _output.DisplayMat(frame);
45.  
46.         List<Mat> channels = SeparateChannnels(frame);
47.  
48.         foreach (Mat channel in channels)
49.             _output.DisplayMat(channel);
50.  
51.         ERFilter er_filter1 = Text.createERFilterNM1(trained_classifierNM1_filepath, 16, 0.00015f, 0.13f, 0.2f, true, 0.1f);
52.         ERFilter er_filter2 = Text.createERFilterNM2(trained_classifierNM2_filepath, 0.5f);
53.  
54.         string text = DetectTextRegions(original, channels, er_filter1, er_filter2);
55.         UnityEngine.Debug.Log(text);
56.     }
57.  
58.  
59.  
60.     public string DetectTextRegions(Mat original, List<Mat> channels, ERFilter filter1, ERFilter filter2)
61.     {
62.         List<MatOfPoint> regions = new List<MatOfPoint>();
63.         string outputText = "";
64.         foreach (Mat channel in channels)
65.         {
66.             Text.detectRegions(channel, filter1, filter2, regions);
67.             string groupingFilename = "text/trained_classifier_erGrouping.xml";
68.             string filePath = Utils.getFilePath(groupingFilename);
69.             MatOfRect groupRects = new MatOfRect();
70.             Text.erGrouping(original, channel, regions, groupRects, Text.ERGROUPING_ORIENTATION_HORIZ, groupingFilename);
71.  
72.            foreach(Rect rect in groupRects.toList())
73.             {
74.                 try
75.                 {
76.                     Mat m = channel.submat(rect);
77.                     m = binarize(m);
78.                     _output.DisplayMat(m);
79.                     outputText += _decoder.run(m, 0) + "\n";
80.                 }
81.                 catch (Exception e)
82.                 {
83.                 }
84.             }
85.         }
86.         return outputText;
87.     }
88.  
89.        
90.     Mat binarize(Mat mat)
91.     {
92.         //Uses otsu to threshold the input image
93.         Mat binaryImage = mat;
94.         Imgproc.cvtColor(mat, mat, Imgproc.COLOR_BGR2GRAY);
95.         Imgproc.threshold(mat, binaryImage, 0, 255, Imgproc.THRESH_OTSU);
96.         int white = Core.countNonZero(binaryImage);
97.         int black = (int)binaryImage.size().area() - white;
98.         return binaryImage;
99.     }
100.  
101.  
102.  
103.  

Basically I'm getting the image of the book cover from the webcam, then separate channels, and for each channel, I separate the text into its own mat, and then pass that mat to the decoder function, which returns a text.
I call the binarize function on the Mat before passing it to the decoder, which uses threshold to simplify the image into two colors only.
I've attached the images for a sample harry potter book and of the channels and the separated text mats.
The problem is, the text I'm getting is not close to the text at all. For the attached book, the text I'm getting is this:

Wo
MPU
FU
IEWFI

I'm not sure what I am doing wrong. Do I need to process the mats further before passing on to the decoder function? or do I need to not use channels? I've tried a couple of solutions, but have not been successfull at all.

Here are the images.
These are the original image and the channels:




And these are the separated text mats:



As you see, the texts are separated nicely. I have separated POTTER and Cursed Child, which I pass to the decoder, but the text I get back is not even close. What am I missing?

 

What recognition results can your code get when using the test images below?
Differences in text fonts may significantly affect recognition results.

Results of TextRecognitionExample

 

Hii

Thanks a lot for getting back to me.
On that image the results are near perfect. It gets almost all of it correctly.
That's the thing though. Every book has a different font so it becomes challenging.

I think if I get to clean the text area from the whole cover, I'm almost there. Do you know how I can imitate the ERStat code I posted using what we have in openCV for unity?

I'm talking about this part:

Code (CSharp):

1. for (int g=0; g < group.size(); g++)
2.         {
3.            int idx = group[g][0];  
4.             ERStat er = regions[idx][group[g][1]];
5.  
6.            //Ignore root region
7.             if (er.parent == NULL)
8.                continue;
9.  
10.            //Transform the linear pixel value to row and col
11.            int px = er.pixel % channels[idx].cols;
12.            int py = er.pixel / channels[idx].cols;
13.      
14.            //Create the point and adds it to the list.
15.            Point p(px, py);  
16.  
17.            //Draw the extremal region
18.             floodFill(
19.                channels[idx], out,               //Image and mask
20.                p, Scalar(255),                   //Seed and color
21.                nullptr,                       //No rect
22.                Scalar(er.level),Scalar(0),       //LoDiff and upDiff
23.                flags                           //Flags
24.            );  
25.         }

Is there an object like ERStat in OpenCV for unity or any way to get that info?
Thanks again.

 

The "void erGrouping (InputArray image, InputArray channel, vector <vector <Point>> contours, CV_OUT std :: vector <Rect> & groups_rects, int method, const String & filename, float minProbability)" method is implemented in OpenCVForUnity. However, the "void erGrouping(InputArray image, InputArrayOfArrays channels, vector<vector<ERStat> > &regions, vector<vector<Vec2i> > &groups, vector<Rect> &groups_rects, int method, const string& filename, float minProbability)" method is not implemented.
In OpenCV C ++, "vector <vector <Point>>" is converted to "vector <vector <ERStat >>" with the following code.
https://github.com/opencv/opencv_co...b4b0e5971/modules/text/src/erfilter.cpp#L3958
https://github.com/opencv/opencv_co...971/modules/text/src/erfilter.cpp#L3965-L4036

 

Thank you very much. I'll keep that in mind.
For now, I'm actually back to using features to compare images. I think I have a better chance with that instead of OCR, although I'm not sure about the speed of searching features in thousands or millions of covers yet. I'll report back as soon as I test it.

 

Hi there

I'm customizing the marker based example, scene GyroSensorMarkerBasedARExample. Is there a way to stretch the webcam's quad to fit the width of the landscape screen while keeping its aspect ratio? Looking into GyroSensorMarkerBasedARExample.cs line 119:

Code (CSharp):

1.  
2.             if (widthScale < heightScale) {
3.                 Camera.main.orthographicSize = (width * (float)Screen.height / (float)Screen.width) / 2;
4.                 imageSizeScale = (float)Screen.height / (float)Screen.width;
5.             } else {
6.                 Camera.main.orthographicSize = height / 2;
7.             }
8.  

I tried forcing the first block and that makes it fit but the position of ARObjects is wrong, there is something missing.

 

You will probably need to customize the ARCamera.fieldOfView value.
https://forum.unity.com/threads/released-opencv-for-unity.277080/page-23#post-3086194

 

(ENOX IGNORE I have fixed the issue) If anyone comes across this and wants to use it go for it. Only thing is I don't have any pretty things going on ie filters, so you'll have to add those in..

Code (CSharp):

1.     private static void FaceSwap(Texture2D destination, Texture2D origin)
2.     {
3.         var faceLandmarkDetector = new FaceLandmarkDetector(DlibFaceLandmarkDetector.UnityUtils.Utils.getFilePath("sp_human_face_68.dat"));
4.         List<UnityEngine.Rect> dRects = new List<UnityEngine.Rect>();
5.         List<UnityEngine.Rect> oRects = new List<UnityEngine.Rect>();
6.         List<List<Vector2>> landmarkPoints = new List<List<Vector2>>();
7.         Mat dMat = new Mat(destination.height, destination.width, CvType.CV_8UC4);
8.         Mat oMat = new Mat(origin.height, origin.width, CvType.CV_8UC4);
9.         OpenCVForUnity.UnityUtils.Utils.texture2DToMat(destination, dMat);
10.         OpenCVForUnity.UnityUtils.Utils.texture2DToMat(origin, oMat);
11.  
12.         //Destination Landmarks
13.         OpenCVForUnityUtils.SetImage(faceLandmarkDetector, dMat);
14.         dRects = faceLandmarkDetector.Detect();
15.         landmarkPoints.Add(faceLandmarkDetector.DetectLandmark(dRects[0]));
16.         //Origin LandMarks
17.         OpenCVForUnityUtils.SetImage(faceLandmarkDetector, oMat); //<---THIS IS THE FIX
18.         oRects = faceLandmarkDetector.Detect();
19.         landmarkPoints.Add(faceLandmarkDetector.DetectLandmark(oRects[0]));
20.  
21.         DlibFaceChanger faceSwapper = new DlibFaceChanger();
22.         faceSwapper.SetTargetImage(dMat);
23.         faceSwapper.AddFaceChangeData(oMat, landmarkPoints[1], landmarkPoints[0], 1);
24.         faceSwapper.ChangeFace();
25.         OpenCVForUnity.UnityUtils.Utils.matToTexture2D(dMat, destination);
26.         faceSwapper.Dispose();
27.     }

 

Hey EnoxSoftware,

I thought I had fixed the issue, but for some reason I cannot get the above code to work on mobile, it seems like its working but the face swap does not happen. any tips please thank you levon.

Update.. the face swap wont work if the image is flipped a different direction from the camera, Is there a way to detect the face no matter what way the image is rotated?

 

You need to rotate the input Mat.
static void OpenCVForUnity.CoreModule.Core.rotate ( Mat src,,Mat dst, int rotateCode )

https://enoxsoftware.github.io/Open...1_core.html#a8d11b0f392585a665722be8e1e7e428c