[RELEASED] Dlib FaceLandmark Detector

hey @EnoxSoftware ,

so we bought the "DlibFaceLandmarkDetector" and the "OpenCV for Unity" packages, and managed to run all the examples on desktop macos.

Now, when I just install "DlibFaceLandmarkDetector", and compile for webgl, it takes about 30s-1min for the 4 example scenes included.
Then after adding the "OpenCV for Unity" package to my test project and doing the 3 steps detailed in the Readme under "Quick setup procedure to run the Advanced examples", I build again for webgl and the build procedure takes 3887s, so more than an hour.
This is on a 4Ghz Quad Core i7 with a fast SSD.
How are people dealing with these build times? As it stands it becomes basically impossible to iterate on a project that includes the OpenCV package.

 

Adding to the above, when I follow the Readme of the 'FaceMask Example' by the book and try to compile the basic exmaple for WebGL it also took about 10mins to compile.
• clean 2020.2.2f1 3D project
• Import 'FaceMask Example'
• Import 'OpenCV for Unity'
• Import 'DLib FaceLandmark Detector'
• move the sp_human_face_68.dat and click the two import settings in the Tools menu

 

Hi . i work for a school project "Reflection of a user's facial expression by avatar " and i have creat my own avatar and i i implemented openCV and openFace and i have create face position and orientation but i have some echo i didn't found how create emotional recognition with C# .
Thank you

 

I disable compression during development.
https://answers.unity.com/questions/1722260/how-do-i-speed-up-webgl-builds-for-testing.html

 

Unfortunately, I don't have any examples of emotion recognition using the DlibFaceLandmarkDetector.

The following page, OpenCVDnnEmotionFerPlusExample, is a good example of using the Dnn module of OpenCV.
https://qiita.com/utibenkei/items/adc7f526d629bdfc41fc

 

Yeah, thats what I did too. The unity team said in a different thread that webgl builds are just generally slow, with the C# to IL2CPP and then to wasm steps. If you look closely at the system resources used, some of these parts aren't even multi-threaded and just use one CPU at 100% (instead of using multiple Cores like most build systems these days).

@EnoxSoftware
PS: I had another question, but I found you already had the `_downscaleRatio` option *facepalm*
So, after some tests, on my 4.0Ghz iMac, I got these findings (all on WebGL, Safari and Chrome seem to be equal. In the Editor I get >100FPS usually but I'm only interest in the -in-browser performance ):

(Res, Downsampling - DS, Frameskip - FS)
640x480 - no DS - no FS - 5 FPS
640x480 - 2x DS - no FS - 14 FPS
640x480 - 2x DS - 2x FS - 33 FPS
800x600 - 2x DS - no FS - 10 FPS
800x600 - 2x DS - 2x FS - 13 FPS
800x600 - 3.2x DS - 2x FS - 19 FPS
800x600 - 3.2x DS - 4x FS - 22 FPS
800x600 - 3.2x DS - 8x FS - 24 FPS
1024x768 - 2x DS - 2x FS - 9 FPS
1024x768 - 3.2x DS - 2x FS - 12 FPS
1024x768 - 3.2x DS - 4x FS - 15 FPS

I'm surprised that a higher resolution has such a big performance impact, especially since 800/3.2 is only 250, so I expected that test to be much faster then the 320x240 it uses for recognition for the 640 /2 image.
Also, extra frameskips don't make then FPS difference I would expect them to.

 

Hello guys, I am trying to instantiate spectacle on the eyes but I don't know how to manage the position and resize the spectacle after track the eyes.Do anyone help me in this .I've purchased the above plugin also but still I am facing these issues.

 

Hey guys! I'm trying to deploy the FaceMaskAdditionalExample scene on Android but getting stuck on loading the shape predictor through "DlibFaceLandmarkDetector_LoadShapePredictor(nativeObj, shapePredictorFilePath);".

I am unpacking it from streaming asset and saving to persistant data path on android. This works fine on PC and it seems that it saves the file on Android as well to /storage/emulated/0/Android/companyname/files/sp_human_face_68.dat.

However when I pass the file path to DlibFaceLandmarkDetector_LoadShapePredictor it doesn't seem to load anything. Has anyone tried this before successfully?

Thanks in advance!

 

/storage/emulated/0/Android/companyname/files/sp_human_face_68.dat.
Is this the path you got from "Utils.getFilePath ("sp_human_face_68.dat");"?

 

Hello sorry for late reply. Yes (OpenCVForUnity)Utils.getFilePath ("sp_human_face_68.dat"); return following path /storage/emulated/0/Android/data/com.DefaultCompany/files/opencvforunity/sp_human_face_68.dat and it failes to create new FaceLandmarkDetector(sp_human_face_68_dat_filepath);

However the exact same code work for PC.

Thanks in advance

 

Ok so on further investigation I noticed that it cast an exception on Android when calling DlibFaceLandmarkDetector_Init(); "Unable to load DLL 'dlibfacelandmarkdetector': The specified module could not be found."

Not really sure what to do here.

UPDATE

Ok so found this post https://answers.unity.com/questions/1752636/android-64-bit-so-dll-file-not-found-error.html and did what ninjafishnz wrote, changed \Assets\DlibFaceLandmarkDetector\Plugins\Android\libs\arm64-v8a/libdlibfacelandmarkdetector to Android ARM64 and got it to work!

Thanks for taking time and reply!

 

Does this problem also occur with the default Texture2DExample?
https://github.com/EnoxSoftware/Dli...Examples/Texture2DExample/Texture2DExample.cs

Could you tell me the environment you tested?
FaceMaskExample version :
DlibFaceLandmarkDetector :
Android Device :

 

Hello.
please tell me.
I want to track the ugui image at the top left of face detection
In the code below, it matches at the center, but it shifts toward the edge.
What's wrong?

Code (CSharp):

1.  
2.  
3. public RectTransform objimage;
4.  
5. void Update()
6. {
7. .
8. .
9.  
10.   foreach (var rect in webcamtextureexample.detectResult)
11.         {
12.            
13.             var scalex = Screen.width /1280f;           // 2436/1280  2436..iPhone11
14.             var scaley = Screen.height / 720;            // 1125/720
15.  
16.             //draw landmark points
17.             //webcamtextureexample.faceLandmarkDetector.DrawDetectLandmarkResult<Color32>(webcamtextureexample.colors, webcamtextureexample.texture.width, webcamtextureexample.texture.height, 4, true, 0, 255, 0, 255);
18.  
19.             var x = rect.x * scalex;
20.             var y = Screen.height - (rect.y * scaley);
21.  
22.             objimage.position = new Vector3(x, y, 0);
23.            
24.             //draw face rect
25.             webcamtextureexample.faceLandmarkDetector.DrawDetectResult<Color32>(webcamtextureexample.colors, webcamtextureexample.texture.width, webcamtextureexample.texture.height, 4, true, 255, 0, 0, 255, 2);
26.             webcamtextureexample.texture.SetPixels32(webcamtextureexample.colors);
27.             webcamtextureexample.texture.Apply(false);
28.  
29.             break;
30.  
31.         }
32.  
33. }
34.  

 

Dear @EnoxSoftware , after updating to Unity to 2021.2.0b1 without changing anything our WebGL project fails to build with the error:

Code (CSharp):

1. Building Library/Bee/artifacts/WebGL/build/debug_WebGL_wasm/build.js failed with output:
2. emcc:WARNING: --llvm-lto ignored when using llvm backend
3. emcc2: warning: EXTRA_EXPORTED_RUNTIME_METHODS is deprecated, please use EXPORTED_RUNTIME_METHODS instead [-Wdeprecated]
4. emcc2: error: /var/folders/qb/xtrq8wsn0lz_s_b08h7n_s2w0000gn/T/tmp8kd72bngopencvforunity.a: File has a suffix of a static library ('.a',), but instead is an LLVM bitcode file! When linking LLVM bitcode files use .bc or .o.
5. UnityEditor.BuildPipeline:BuildPlayer (UnityEditor.BuildPlayerOptions)

The key issue here is the error compiling the `tmp8kd72bngopencvforunity.a`. I did read that Unity updated emscripten, so maybe this has broken something in the way the OpenCVForUnity project builds?
I did see the exact same error for `DlibFaceLandmarkDetector` so it seems to be affecting both assets equally.

We also tried updating to OpenCV 2.4.4 and DLibFace 1.3.2, still same error.

PS: Tried again with a clean new project, only added OpenCV and DLibFace, same error.

 

Thank you very much for reporting.
I will check it.

 

Thank you for looking into it @EnoxSoftware. It seems with 2021.2.0b2 the issue remains, was hoping it may be a fluke in 0b1

 

@EnoxSoftware can you give a status update about the 2021.2 emscripten issue?
We're waiting to make a new build of our WebGL app, and we need to use 2021.2 since 2021.1 and before does not get Safari Camera permissions correctly.

 

Upon investigation, the following error occurs and the compilation fails. I posted this issue on the Unity forum.
https://forum.unity.com/threads/fai...tform-contain-native-plugin-files-bc.1137496/
This issue occurs even with very simple native plugins.

 

Hello,

Is it possible to find a specific person's face with readily available ? What I want to achieve is determine a persons face at image A and find that person at image B. Is it possible with this asset ?

 

Unfortunately, the DlibFaceLandmarkDetector does not have such a feature.

 

What frame rate does this run on for mobile iOS / Android

Does it run at 60fps yet for mobile?

 

Hey Enox,

I am using your assets of dlib and opencv for my some college project. Its working well. Facial landmarks works but it works bit less accurate. So any suggestions for higher accuracy and stability.

 

HI @EnoxSoftware !

Using Dlib/OpenCV, is there a way to just get the detection rectangle returned - in one of the scenarios I'm using DLIB I don't need to detect any landmarks.

Thanks.

 

Of course it is possible, you can use the Detect() method to get a list of rectangles.
https://github.com/EnoxSoftware/Dli...les/Texture2DExample/Texture2DExample.cs#L100

 

Hi, thanks for that, but what i want to do is just load and run the SVM object detector only, not load the shape predictor file at all.... the faceLandmarkDetector class will only run if only the dat file is supplied or the dat file and the SVM... i was wondering if there was an object only detector that just runs the SVM?

 

If you set shapePredictorFilePath to null, you will only be able to load the objectDetector.
FaceLandmarkDetector ( objectDetectorFilePath,　null );

 

Excellent thank you!

 

I set it to width 280 and height 200 but it still big and didn't make any change

 

It is possible that your WebCamera does not support the desired frame size.

 

Hi, i want to ask a question, how many faces can the library realistically tracks simultaneously?

 

This asset provides the ability to detect facial landmarks, but does not provide face tracking.
There is no limit to the number of faces that can be detected at one time.

 

Thank you, another question, is there any way to improve detection with person wearing a mask?

 

hey guys, great tool thanks so much. I have one question regarding the ARHead example with openCV. I've noticed that the ar head flickrs quite alot in this example, as well as the DLIB facial points, whereas in OpenCV's library examples, although theres no ar head features, the DLIB facial points are quite stable. Any ideas how i can improve the quality? I am using a 720p webcam, maybe upping quality to 1080p can help?

 

I just switched to 2021.3 and I think this package (and OpenCV for unity) needs an update. The only way I could get my WebGL builds to work was to remove the 2019 folder's file (I think Unity reverted to using the 2019 plugin):

 

After importing, could you run "Set Plugin Import Settings"?

 

Does this work with Unity 2021.3.3f1 WebGL? dlibfacelandmarkdetector machine type must be wasm32 or wasm64

 

Can We track Eyes Moving Left right using Dlib

 

Unfortunately, I do not have an example of Eyes tracking. The following code is probably helpful.
https://towardsdatascience.com/real-time-eye-tracking-using-opencv-and-dlib-b504ca724ac6

 

Hello! We are interesting in a solution, which will let us set a mask from an image and put in on the needed place. Current solution moves mask over the detected face now. But we need to change a part of the image (a person's face on photo). Where do we need to dig? How can we achieve this aim? We just need to get a user's face and let him change the face of an image.
Thank you a lot!

 

it is possible to change the limit distance of the camera with respect to the person, because for now it seems that the maximum is 70 cm

 

@EnoxSoftware does DLIB now support GPUs?I caught a thread from 6 years ago saying it did not. Is that still the case?

 

The shape_predictor and frontal_face_detector used in DlibFaceLandmarkDetector do not support GPUs.
https://github.com/davisking/dlib/issues/2305

 

Hi. I've used the webCamTextureToMatHelper example in my project which is running at nearly 400fps in the FpsMonitor.

But as soon as introduce the function below, the fps drops to 16:
List<UnityEngine.Rect> detectResult = faceLandmarkDetector.Detect();

Is there anyway to make this function run faster?

EDIT: Now realised I'm mixing your OpenCV and Dlib packages together, when I shouldn't - is this the reason for the speed issue?

 

I used DLib years ago on my mac and am about to start playing with some custom characters that have facial bones. Are there any additional values to this plugin that is not already in the original DLib?

 

Could you try the FrameOptimizationExample?
The faceLandmarkDetector.Detect() method takes longer to process than OpenCV's face detection. Therefore, the FrameOptimizationExample is optimized by using OpenCV's face detection instead.
https://github.com/EnoxSoftware/Dli...timizationExample/FrameOptimizationExample.cs

 

The model included with DlibFaceLandmarkDetector has a different dataset than the original model, but the order and position of the facial feature points are the same.

http://blog.dlib.net/2018/01/correctly-mirroring-datasets.html

 

Hi, i'm testing this example and trying to change or add objects to the head (we want to put hats, glasses and other things), but the new elements are not showing on play.

For now i'm just putting basic 3d objects (cubes, etc) and change the head reference in Quad script to it. But no mather where i put it , is not showed on screen. They need to use a specific shader to work or something like that?

 

Ok nevermind.

It was the layer. No UI layer, no party.

 

One more thing. Is there any tool or something to put the points on a mask? Or move every one of them manually is the only way?

 

I think i finally make the merge of the ARHead with facemask example and the MultiHead example (plus the eye transparency on the mask) works somehow.

But still the first ARHead don't want to be desactivated for some reason. And some times the gameobject hosting the head shows inverted and far away from the mask.