[RELEASED] Dlib FaceLandmark Detector

Have the "DlibFaceLandmarkDetector / StreamingAssets /" folder been moved to the "Assets/" folder?

 

If I was to add a 3D object onto the users eyes do I need OpenCV as well? @EnoxSoftware

 

WebCamTextureARSample and VideoCaptureARSample use the OpenCV's Calib3d.solvePnP() method to calculate the camera matrix.

 

Dlib FaceLandmark Detector v1.0.5 is now available.

Version changes
1.0.5
[Common]Fixed WebCamTextureToMatHelper class.
[Common]Added Utils.getVersion().
[Common]Fixed Utils.getFilePathAsync().

 

does it support remote IP camera ? or just local camera ?

 

DlibFaceLandmarkDetector supports to Unity's WebCamTexture. If IP Camera is compatible with WebCamTexture, DlibFaceLandmarkDetector SetImage() method will work.
Also, The SetImage() method also supports a pixel byte array as an argument.

http://enoxsoftware.github.io/DlibF...mark_detector_1_1_face_landmark_detector.html

Code (CSharp):

1. void     SetImage (Texture2D texture2D)
2. void     SetImage (WebCamTexture webCamTexture)
3. void     SetImage (WebCamTexture webCamTexture, Color32[] bufferColors)
4. void     SetImage (IntPtr intPtr, int width, int height, int bytesPerPixel)
5. void     SetImage (IntPtr intPtr, int width, int height, int bytesPerPixel, bool flip)
6. void     SetImage< T > (IList< T > array, int width, int height, int bytesPerPixel)
7. void     SetImage< T > (IList< T > array, int width, int height, int bytesPerPixel, bool flip)

 

When setting MenuItem[Tools/Dlib FaceLandmark Detector/Set Plugin Import Settings], it seems that the following error occurs on Unity5.5 or later.

No valid name for platform: 11
UnityEditor.PluginImporter:SetCompatibleWithPlatform(BuildTarget, Boolean)
DlibFaceLandmarkDetector.DlibFaceLandmarkDetectorMenuItem:SetPlugins(String[], Dictionary`2, Dictionary`2) (at Assets/DlibFaceLandmarkDetector/Editor/DlibFaceLandmarkDetectorMenuItem.cs:193)
DlibFaceLandmarkDetector.DlibFaceLandmarkDetectorMenuItem:SetPluginImportSettings() (at Assets/DlibFaceLandmarkDetector/Editor/DlibFaceLandmarkDetectorMenuItem.cs:124)


To fix, Replace line 192 - line 194 with the following code.

from

Code (CSharp):

1.                                                 foreach (BuildTarget target in Enum.GetValues(typeof(BuildTarget))) {
2.                                                     if ((int)target != -1)pluginImporter.SetCompatibleWithPlatform (target, false);
3.                                                 }

to

Code (CSharp):

1.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.Android, false);
2.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.iOS, false);
3.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneWindows, false);
4.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneWindows64, false);
5.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneOSXIntel, false);
6.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneOSXIntel64, false);
7.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneOSXUniversal, false);
8.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneLinux, false);
9.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneLinux64, false);
10.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneLinuxUniversal, false);
11.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.WSAPlayer, false);
12.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.WebGL, false);

This bug will be fixed in the next version.

 

I tried to attach MovieTexture to an ARobject, but it doesn't play the sound. Does it support sound?

 

Are you using the MovieTexture class? https://docs.unity3d.com/Manual/class-MovieTexture.html
It seems necessary to control the sound by script.

Code (CSharp):

1.             var audioSource = GetComponent<AudioSource>();
2.             audioSource.clip = movieTexture.audioClip;
3.             audioSource.loop = true;
4.             audioSource.Play ();

 

Is it possible for WebCamTextureARSample to use CatDetectionSample? if so, how? Basically, I want to use AR with the Cat Sample.

 

”frontal_cat_face.svm" and "shape_predictor_68_cat_face_landmarks.dat" are the files that I trained. Because this file is not good detection accuracy, It is not work well on WebCamTexture.

 

What's the recommended resolution for front facing camera and rear camera?

 

When an Android phone is in portrait mode, the WebCamTextureSample demo does not show the red box or green face overlay. The OpenCV webcam demos do show the overlay in any orientation. Is there something I can do to get the overlay to show in portrait mode without OpenCVforUnity? Maybe an edit to WebCamTextureSample.cs?

These results were from testing DlibFaceLandmarkDetector v1.0.6 and the Android app (https://play.google.com/store/apps/details?id=com.enoxsoftware.dlibfacelandmarkdetectorsample) on a Nexus5 and a MotoG3.

 

WebCamTextureSample can detect landscape direction only.

But, WebCamTextureToMatSample can detect landscape and portrait direction using OpenCV's method.

 

I re-read through earlier posts on this thread, you can do portrait mode with only this asset. This saves, among other things, on build size.

Thank you so much @shawhu for posting your code: https://forum.unity3d.com/threads/released-dlib-facelandmark-detector.408958/#post-2748104

I had to make a small fix. adding [i ] into result[i ] = input[x + y * width];
As @shawhu suggested, I swapped width and height as follows to match the rotated value in colors:

Code (CSharp):

1. #if UNITY_ANDROID && !UNITY_EDITOR
2. colors = RotateCW(colors, webCamTexture.width, webCamTexture.height);
3. ...
4. faceLandmarkDetector.SetImage<Color32> (colors, webCamTexture.height, webCamTexture.width, 4, flip);
5. ...
6. faceLandmarkDetector.DrawDetectLandmarkResult<Color32> (colors, webCamTexture.height, webCamTexture.width, 4, flip, 0, 255, 0, 255);
7. ...
8. faceLandmarkDetector.DrawDetectResult<Color32> (colors, webCamTexture.height, webCamTexture.width, 4, flip, 255, 0, 0, 255, 2);
9. ...
10. colors = RotateCCW(colors, webCamTexture.height, webCamTexture.width);

Each facePoint in the points[] set also needs to be converted:

Code (CSharp):

1. #if UNITY_ANDROID && !UNITY_EDITOR
2.         // correct rotated value sent to faceLandmarkDetector
3.         facePoint.x += (320 - 240) / 2f;
4.         facePoint.y += (240 - 320) / 2f;
5. #endif

 

Dlib FaceLandmark Detector v1.0.7 is now available.

Version changes
1.0.7
[Common]Fixed WebCamTextureToMatHelper.cs.(flipVertical and flipHorizontal flag)

1.0.6
[Common]Fixed OpenCVForUnityMenuItem.cs.(No valid name for platform: 11 Error)
[Common]Added OptimizationWebCamTextureToMatHelper.cs.

 

The face detection step - when you actually find faces and store then in a list of Rect - seems to be much, much slower than the landmark detection step. Would it be possible in order to speed it up to use a much lower image resolution for face detection (I understand you don't need big images if faces are big enough), keeping the higher resolution image for landmark detection? It seems both steps currently use the same image.

 

It is possible to speed up processing by downscaling image. OptimizationSample is optimized by downscale and skipframe using "OpenCV for Unity".
Also, Since face detection is faster using OpenCV than using Dlib,
I think that using "OpenCV for Unity" instead of "DlibFaceLandmarkDetector" will speed up face detection.

 

just curious, do you think it can track an open hand?

 

If using HOGObjectDetector trained for open hand, I think that it is possible to do so.However, the HOGObjectDetector may not be robust against rotation.

 

How is the dlib compiled?
Which compiler optimizations did you turn on, especially for the OSX version. SSE2/4/AVX?
Because running dlib examples inside a VM on OSX is actually faster than running your dlib examples on the same machine in Unity.

 

SSE2 is enabled now.
I tried the SSE 4 and AVX, but the FPS improved slightly.
In the future release, I am going to include a plug-in that enabled SSE4 or AVX in the asset.

 

Dlib FaceLandmark Detector v1.0.9 is now available.

Version changes
1.0.9
[WebGL]Added WebGL Plugin for Unity5.6.

1.0.8
[Common]Changed the name of asset project.("Sample" to "Example")
[Common]Fixed VideoCaptureARExample and WebCamTextureARExample.

 

but i think HOGObjectDetector just return a rectangle, i am talking about like the face tracking, which return the position of key points.

 

public void SetImage<Color32> (List<Color32> array, int width, int height, int bytesPerPixel, bool flip) seems to be causing memory leaks on an iPhone 4s when called in OnPreviewUpdate from NatCam . Maybe it is not a memory leak but the memory keeps going up until it crashes and CPU goes to 100%. Any idea what can be causing this?

 

There is currently no such example using DlibLandmarkDetector. But,There seems to be such an implementation example using OpenCV for Unity.

 

Do memory leaks occur even when only DlibFaceLandmarkDetector is used?

 

Please help Please tell me how to change face exchange including "forehead" with "Texture2DToMatSample"?

 

shape_predictor_68_face_landmarks.dat is the shapepredictor file which dlib provides.
If you want to add the landmark points, you need to train the more landmarks shapepredictor file including "forehead".
http://dlib.net/train_shape_predictor_ex.cpp.html
http://stackoverflow.com/questions/36711905/dlib-train-shape-predictor-ex-cpp?rq=1
http://stackoverflow.com/questions/...ape-predictor-for-194-landmarks-helen-dataset

 

Thank you for answering
Can you share the image file you used with the xml file you created with imglab?

 

I used this file.
http://dlib.net/files/data/ibug_300W_large_face_landmark_dataset.tar.gz
The comment column of this article is helpful.
http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html
http://dlib.net/train_shape_predictor_ex.cpp.html

 

Dlib FaceLandmark Detector v1.1.0 is now available.

Version changes
1.1.0
[Win][Mac][Linux][UWP]Added the native plugin file enabled SSE4 or AVX compiler option.

 

Hi, do you have

Do you have a snippet/example for a cropped matrix?

If not - I assume this is declaring mat to be cropped size in WebCamTextureTOMathelper.cs and in Utils.webCamTextureToMat() - custom populate colors based on crop size

 

Sorry for the reply.
Thank you for your answer I'd like to try it out soon

 

How do I add a new face_mask in the FaceMaskExample > Resources? I've added a new image with the same settings as the other masks. When changing my image's name to "face_mask5" it then inherits the shape of the examples "face_mask5". How do I change/debug the face points for my new image?

 

Found the "ExampleDataSet.cs" Any good suggestions for point placing tools?

 

To add a new face, you need to add the face information you want to add to "Assets \ FaceMaskExample \ Scripts \ Data \ ExampleDataSet.cs".If you add a face that can not be automatically detected with a face detector like a panda or an animated face, you need to manually add landmark point data.You also need to add the image of the face you want to add to the "Resource" folder.
Please refer to this image for index of face point.

 

For the AR mask example, what is the scene to real world scale to be able to size real objects properly?

 

I'm experiencing a weird behaviour on Android. We recently released a free app on iOS that uses this solution and works like a charm.

On Android I can Build&Run so it's uploaded to my Android device and it runs. My device is old and therefore it's a bit slow, but it works.

However, when I publish this app in Google Play (beta) and download it on my device... it doesn't detect my face. According to logcat, the model file is not found! (I have not moved it, it's still in StreamingAssets and it does work when I build & run)

Is anyone experiencing this too?

 

Did you split the APK?
If you did, you will have to tell your app where exactly is your data.

 

That's it! I've never split an APK before, so I was not expecting this! No idea Unity splitted apps automatically when they are larger than 100Mb. But indeed, there's an OBB file alongside the apk file. I'll figure out how to load it, thanks! (here's the doc, just in case, because given the size of most model files others may experience the same problem https://docs.unity3d.com/Manual/android-OBBsupport.html)

 

Any advice for changing the human_male gameobject scale based on different head sizes in the AR Sample?

 

https://play.google.com/store/apps/details?id=com.enoxsoftware.dlibfacelandmarkdetectorexample
This demo app is downloading the OBB file at the first launch. I referred to the following pages.
http://naochang.me/?p=488
https://www.assetstore.unity3d.com/jp/#!/content/3189

 

@EnoxSoftware I tried the FaceMask application from Google Play, I wanted to ask if optimization explained here was applied to that application.
Thanks

 

Yes.

• Use OpenCV face detection instead of dlib's face detection - Processing speed of OpenCV face detection is faster than Dlib's face detection.

 

would it be possible to send the data via OSC to another software (like maxmsp)?

 

You can acquire face landmark data in List< Vector2 > and double[] format.
http://enoxsoftware.github.io/DlibF...tector.html#abb02ad0b17cc25d61960e9569c335d38

I think it is possible to send these data via OSC.

 

is the coordinate system bottom left hand corner for origin - or like Unity's top left hand corner

 

Which dlib detector to keep when importing both this package and the facemask package?

 

The OpenCV's coordinate system has the X axis horizontal and the Y axis vertical with the origin [0, 0] at the upper-left corner.