[RELEASED] Dlib FaceLandmark Detector

Hi

Is there any software or plugin for some popular graphic editor (like Photoshop), to easily draw and export custom landmark coordinates from image?

I would like to use some fancy animal face that need custom 68 points coordinates. I can measure it and write it manually, but it is a hell lot of a work. It would be nice to have some software to "draw" landmarks on image, and then export a ready to use landmarks array

Thanks for this great asset by the way

 

This tool may be useful.
https://www.mobilefish.com/services/record_mouse_coordinates/record_mouse_coordinates.php

 

Hi,

I bought opencv and Dlib can you help me to apply hair style not face mask , I want only to put 2d image hair on the user head-hair ?

Thanks,

 

I have found such a use example.
https://twitter.com/miyumiyuna5/status/841924714058641409
It may be feasible by displaying the hair model instead of displaying the red head model.

 

Hi EnoxSoftware,
I bought Dlib FaceLandmark Detector and OpenCV for Unity assets and are working great but I have a simple question: I want to create my dataset file by using train_shape_predictor_ex.cpp but I don't know what settings to use for shape_predictor_trainer. I tried leave the existing ones from the cpp file but I'm not sure that those ones are the best (_oversampling_amount = 300, _nu = 0.05 and _tree_depth = 2). Can you please share what settings (what parameters values) are you using when you create sp_human_face_68.dat and sp_human_face_68_for_mobile.dat ?

 

https://sourceforge.net/p/dclib/discussion/442517/thread/71b4b7c8/
cascade_depth = 15
tree_depth = 4 //sp_human_face_68.dat
or
tree_depth = 2 //sp_human_face_68_for_mobile.dat
The other parameters are the default values.

I changed "train_shape_predictor_ex.cpp" as follows.

Code (CSharp):

1.         // Now make the object responsible for training the model.
2.         shape_predictor_trainer trainer;
3.         // This algorithm has a bunch of parameters you can mess with.  The
4.         // documentation for the shape_predictor_trainer explains all of them.
5.         // You should also read Kazemi's paper which explains all the parameters
6.         // in great detail.  However, here I'm just setting three of them
7.         // differently than their default values.  I'm doing this because we
8.         // have a very small dataset.  In particular, setting the oversampling
9.         // to a high amount (300) effectively boosts the training set size, so
10.         // that helps this example.
11.         //trainer.set_oversampling_amount(300);
12.         // I'm also reducing the capacity of the model by explicitly increasing
13.         // the regularization (making nu smaller) and by using trees with
14.         // smaller depths.
15.         //trainer.set_nu(0.05);
16.         trainer.set_tree_depth(4);
17.         //trainer.set_tree_depth(2);
18.  
19.         // some parts of training process can be parallelized.
20.         // Trainer will use this count of threads when possible
21.         trainer.set_num_threads(4);
22.  
23.         trainer.set_cascade_depth(15);

 

Another question: if I train any kind of object I want along with some landmark points for shape predictor then is fine to use the code from your CatDetectionExample.cs ?

Code (CSharp):

1. FaceLandmarkDetector faceLandmarkDetector = new FaceLandmarkDetector (frontal_cat_face_svm_filepath, sp_cat_face_68_dat_filepath);

Or this is working only with face kind object?

 

FaceLandmarkDetector works with non-face models as well.

 

Hi,
There is any chance to can use dlib CNN based object detection?

 

There are currently no plans to add the CNN-based face detector.
An example of face detection using the DNN module is included in OpenCV for Unity.
https://github.com/EnoxSoftware/Ope...CaffeExample/ResnetSSDFaceDetectionExample.cs

 

the link is broken

 

https://github.com/EnoxSoftware/Ope...CaffeExample/ResnetSSDFaceDetectionExample.cs

 

Hi.
Does anyone have this weird issue?
While using ARHead example,the webcam texture is very dark in some android devices (eg OnePlus 3) and works great on other Devices.
is there any solution for this problem?

 

I think that this problem is probably caused by a bug in UnityEditor.
Please change the initialization parameter of WebCamTexture.
requestedFPS=1 rear camera(fps is 30) is fine. front camera(fps is 15) is fine.
https://forum.unity.com/threads/released-opencv-for-unity.277080/page-33#post-3445178

 

hi @EnoxSoftware ,

I have question about AR head detection. I have been following the example for video and webcam; its working fine.
What i am try to do is to get the AR head example to work with a Supplied texture2D and its not working as the AR camera matrix might be not supporting for Texture2D.

Can you guide me, How to make the AR head example to work for an Texture2D image [not from camera, i want to load a normal image as texture2D and have the AR example work]?

Thanks.

 

Could you try this example?
https://www.dropbox.com/s/4z36obqxpy0hpkr/ARHeadTexture2DExample.unitypackage?dl=0

 

Thanks a lot @EnoxSoftware

It worked.

 

Hello! I ran webcam AR scene on my Samsung Galaxy s7 and got only 10 fps. Is there anyway to get higher fps or is the phone not powerful enough?

Thank you!

 

I think that FrameOptimizationExample will be helpful.

The face detection step is faster using OpenCV than using Dlib. In FaceMaskExample, face detection is done with OpenCV for speeding up.
Also,
The smaller the OpenCV Mat size is, the shorter the processing time becomes.
There is a possibility that the processing speed is faster when you set RequestWidth and RequestHeight smaller value.

 

Hi,
I'm testing your DNN TensorFlowExample and I have a hard time in getting the detected rects and the edge points. Can you help me with these or can you lead me to the correct way?
Also, do you have any idea how to get the output shape from TensorFlow?

 

It is not possible to detect the rects of objects using "tensorflow_inception_graph.pb" model.
Please refer to this post.
https://forum.unity.com/threads/released-opencv-for-unity.277080/page-33#post-3455418

 

Got it, thank you. That worked fine.
There is any way to detect the shape with any of these 3, Caffe, TensorFlow of Yolo? I mean to get the shape points not just the rect.

 

Thank you! I ran FrameOptimizationExample and FaceMask example and 1st one is faster. Is there a way to run WebCamAR scene with OpenCV as well? I am trying to make a 3d filter that has to do with blinking and various 3d objects and so far was able to play around with the red head object and eye lasers in WebCamAR. So is there a way to either transfer those objects to FrameOpt example so that i could see them in hierarchy or enable opencv for WebCamAR?

I would really prefer to enable OpenCV for WebCamAR or speed it up since i had success working with it.

Thank you!!

 

I own OpenCV and dlib, it seems OpenCV detects eye direction. Any way to do this with dlib?

 

It is not possible to estimate the direction and angle of the face only with the function of DlibFaceLandmarkDetector.

 

Hi! Will you be implementing this CNN MMOD face detection? Seems like it's very good for tracking lots of different poses. http://blog.dlib.net/2016/10/easily-create-high-quality-object.html

 

There are currently no plans to add the CNN-based face detector.
An example of face detection using the DNN module is included in OpenCV for Unity.
https://github.com/EnoxSoftware/Ope...CaffeExample/ResnetSSDFaceDetectionExample.cs

 

Hello! I've been working on a project that uses Dlib and so far everything has been good. However, I am trying to download a picture from the internet and then process it and detect the different facial landmarks. I pass the texture into the same logic that has been working thus far, but now it won't Dlib detects a face, but then when I try to detect the facial landmarks it returns an empty array.

I am saving the texture to the disk in StreamingAssets before trying to process it. If I try to use that same image later on (in a different scene, same script to process the texture though), it works fine! Any clues on what could be causing this odd behavior and what I could do to fix it?

 

Is the following error displayed?

Code (CSharp):

1. UnityException: Texture '' is not readable, the texture memory can not be accessed from scripts. You can make the texture readable in the Texture Import Settings.

Texture2D used by DlibFaceLandmarkDetector must be readable.
https://forum.unity.com/threads/tex...est-gettexture-is-not-readable-solved.447163/

 

Hi, how can I add my own masks in the FaceMaskExample and not the ones used in the Resources folder??
And can I add a mask that isn't using the whole face but the eyes, like glasses?

Edit: Can I use a different picture for Landmarking the Facepoints and use an other mask based on the picture for Landmarking?

 

Unfortunately it does not. I get no errors or warnings at all. I call the DetectLandmark function in FaceLandmarkDetector.cs and when it gets to the internal DetectLandmark call, it just returns a count of 0. It returns the rect of the face perfectly fine, but its just the landmarks where it seems to fail.

 

Thank you very much for reporting.
Could you send me a simple code to reproduce this problem?
https://enoxsoftware.com/opencvforunity/contact/technical-inquiry/

 

You can add new mask data by the following procedure.

 

Hello. In order to speed up the app I've been playing with height and width in Quad's webcam texture to mat helper. There are two options which I don't understand their purpose: requested FPS and Timeoutframe count.
Whatever number I set for them, doesn't seem to change anything for my FPS. Can you please describe what do they do and how do they affect performance/fps?

Also, is there an option to skip the frame somewhere? I've read DLIB speed up manual but didn't understand how to do it.

Thank you!

 

requestedFPS is the initialization parameter of Unity's WebCamTexture class.
https://docs.unity3d.com/ScriptReference/WebCamTexture-requestedFPS.html

Timeoutframe count is the number of frames until generating a TIMEOUT error when acquiring images of WebCamTexture.

Please see "DlibFaceLandmarkDetectorWithOpenCVExample / ImageOptimizationHelper.cs".
ImageOptimizationHelper.cs is used to skip frames with FrameOptimizationExample.
https://github.com/EnoxSoftware/Dli...ationExample/FrameOptimizationExample.cs#L282

 

When running ARwebcam example I've noticed that even if the head is held perfectly still, the mesh is twitching. Is there a way to make it more still so that animations don't appear jumping?

​

 

Could you try changing the value of the low pass filter?
The larger the value, the more filter the vibration.

 

I tried playing around with those numbers but its still vibrating. The more I increase those numbers, the head become more still but then it affects the performance to the point where I move my head but filter is still (and still rocking back and forth but less frequent). Is there any other method to help it to stabilize?

Thank you very much!

 

Hi, I already own OpenCV for Unity and I'm thinking to have a go with Dlib FaceLandmark Detector, but first I would just understand if it fits my needs.

I should develop a couple of projects that basically require face tracking AND recognition, applied to public installations. So, because of this last point, I can't rely on some prior learning process, because I've to track and recognize people passing by "on the fly", then store face data and compare it with a db of persons already passed by, to establish if he's one of them or a new one.

Is it possibile to reach that kind of result with this specific plugin? Or maybe with another one of your plugins?

 

Hi sorry for the late response but my screen looks a bit different. I dont have the "FaceMaskData" Object. Whats wrong with my version?

 

DlibFaceLandmarkDetector has no facial recognition function.
Have you already tried RealTime FaceRecognition Example?

 

Please import FaceMask Example v1.0.6 or later.

 

Thank you for the answer.

I still haven't tried it but i'll do for sure.
Do you mean I could use it - maybe making some adjustments - to fulfill my project's requirements?

 

I tried playing around with those numbers but its still vibrating. The more I increase those numbers, the head become more still but then it affects the performance to the point where I move my head but filter is still (and still rocking back and forth but less frequent). Is there any other method to help it to stabilize?

Thank you very much!

 

RealTime Face Recognition Example is a tutorial example project of detecting a face and recognizing a face using only OpenCVforUnity.
I do not know if this example meets your project requirements, but I think it will be useful for creating projects.

 

If you change "objectPoints68" and "imagePoints" you set for the Calib3d.solvePnP () method, the accuracy of the position estimate may be improved.
https://github.com/EnoxSoftware/Dli...eadExample/ARHeadWebCamTextureExample.cs#L523
Calib3d.solvePnP ()
https://docs.opencv.org/3.4.0/d9/d0c/group__calib3d.html#ga549c2075fac14829ff4a58bc931c033d

 

Okay thank you for the reply!! I updated the FaceMaskExample and when i want to track a face i get an error:

ArgumentException: Invalid landmark_points.

FaceMaskExample.FaceMaskColorCorrector.GetFacialContourPoints (System.Collections.Generic.List`1 landmark_points, OpenCVForUnity.Point[] dst_points) (at Assets/Addons/FaceMaskExample/Scripts/FaceMaskColorCorrector/FaceMaskColorCorrector.cs:108)

FaceMaskExample.FaceMaskColorCorrector.UpdateLUTTex (Int32 id, OpenCVForUnity.Mat src, OpenCVForUnity.Mat dst, System.Collections.Generic.List`1 src_landmarkPoints, System.Collections.Generic.List`1 dst_landmarkPoints) (at Assets/Addons/FaceMaskExample/Scripts/FaceMaskColorCorrector/FaceMaskColorCorrector.cs:55)

FaceMaskExample.VideoCaptureFaceMaskExample.CorrectFaceMaskColor (Int32 id, OpenCVForUnity.Mat src, OpenCVForUnity.Mat dst, System.Collections.Generic.List`1 src_landmarkPoints, System.Collections.Generic.List`1 dst_landmarkPoints) (at Assets/Addons/FaceMaskExample/VideoCaptureFaceMaskExample/VideoCaptureFaceMaskExample.cs:595)

FaceMaskExample.VideoCaptureFaceMaskExample.Update () (at Assets/Addons/FaceMaskExample/VideoCaptureFaceMaskExample/VideoCaptureFaceMaskExample.cs:458)

Sorry for causing that much trouble.

 

Please move the “DlibFaceLandmarkDetector/StreamingAssets/” folder to the “Assets/” folder.

 

Thank you! I see what you are suggesting. Is there some sort of a map with x,y,z's that I can look at to base my own points on?? I see all the points that were created but if I change the numbers I don't really know what am I changing space wise. For example based on 68 map, right mouth breath is number 54 and its coordinates are Point3 (26, 15, 83). Where do I get the map with those coordinates?

Also, would you recommend changing only existing points or actually adding my own?

 

Please open "human_male.fbx" in the 3D model editor, check 3D point coordinates and set 3D point coordinates (objectPoints) corresponding to 2D point coordinates (imagePoints).
I do not know if it is better to increase the point to more than 7 points.




Also, this page is very helpful.
http://www.morethantechnical.com/20...timation-with-opencv-opengl-revisited-w-code/