Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only.

    Please, do not make any changes to your username or email addresses at id.unity.com during this transition time.

    It's still possible to reply to existing private message conversations during the migration, but any new replies you post will be missing after the main migration is complete. We'll do our best to migrate these messages in a follow-up step.

    On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live.


    Read our full announcement for more information and let us know if you have any questions.

[RELEASED] Dlib FaceLandmark Detector

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Jun 4, 2016.

  1. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    111
    hey @EnoxSoftware ,

    so we bought the "DlibFaceLandmarkDetector" and the "OpenCV for Unity" packages, and managed to run all the examples on desktop macos.

    Now, when I just install "DlibFaceLandmarkDetector", and compile for webgl, it takes about 30s-1min for the 4 example scenes included.
    Then after adding the "OpenCV for Unity" package to my test project and doing the 3 steps detailed in the Readme under "Quick setup procedure to run the Advanced examples", I build again for webgl and the build procedure takes 3887s, so more than an hour.
    This is on a 4Ghz Quad Core i7 with a fast SSD.
    How are people dealing with these build times? As it stands it becomes basically impossible to iterate on a project that includes the OpenCV package.
     
  2. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    111
    Adding to the above, when I follow the Readme of the 'FaceMask Example' by the book and try to compile the basic exmaple for WebGL it also took about 10mins to compile.
    • clean 2020.2.2f1 3D project
    • Import 'FaceMask Example'
    • Import 'OpenCV for Unity'
    • Import 'DLib FaceLandmark Detector'
    • move the sp_human_face_68.dat and click the two import settings in the Tools menu
     
  3. sfaxiyamina

    sfaxiyamina

    Joined:
    Feb 5, 2021
    Posts:
    1
    Hi . i work for a school project "Reflection of a user's facial expression by avatar " and i have creat my own avatar and i i implemented openCV and openFace and i have create face position and orientation but i have some echo i didn't found how create emotional recognition with C# .
    Thank you
     
  4. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    I disable compression during development.
    https://answers.unity.com/questions/1722260/how-do-i-speed-up-webgl-builds-for-testing.html
     
  5. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    Unfortunately, I don't have any examples of emotion recognition using the DlibFaceLandmarkDetector.

    The following page, OpenCVDnnEmotionFerPlusExample, is a good example of using the Dnn module of OpenCV.
    https://qiita.com/utibenkei/items/adc7f526d629bdfc41fc
     
  6. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    111
    Yeah, thats what I did too. The unity team said in a different thread that webgl builds are just generally slow, with the C# to IL2CPP and then to wasm steps. If you look closely at the system resources used, some of these parts aren't even multi-threaded and just use one CPU at 100% (instead of using multiple Cores like most build systems these days).

    @EnoxSoftware
    PS: I had another question, but I found you already had the `_downscaleRatio` option *facepalm*
    So, after some tests, on my 4.0Ghz iMac, I got these findings (all on WebGL, Safari and Chrome seem to be equal. In the Editor I get >100FPS usually but I'm only interest in the -in-browser performance ):

    (Res, Downsampling - DS, Frameskip - FS)
    640x480 - no DS - no FS - 5 FPS
    640x480 - 2x DS - no FS - 14 FPS
    640x480 - 2x DS - 2x FS - 33 FPS
    800x600 - 2x DS - no FS - 10 FPS
    800x600 - 2x DS - 2x FS - 13 FPS
    800x600 - 3.2x DS - 2x FS - 19 FPS
    800x600 - 3.2x DS - 4x FS - 22 FPS
    800x600 - 3.2x DS - 8x FS - 24 FPS
    1024x768 - 2x DS - 2x FS - 9 FPS
    1024x768 - 3.2x DS - 2x FS - 12 FPS
    1024x768 - 3.2x DS - 4x FS - 15 FPS

    I'm surprised that a higher resolution has such a big performance impact, especially since 800/3.2 is only 250, so I expected that test to be much faster then the 320x240 it uses for recognition for the 640 /2 image.
    Also, extra frameskips don't make then FPS difference I would expect them to.
     
    Last edited: Mar 1, 2021
    EnoxSoftware likes this.
  7. nextgen-ss

    nextgen-ss

    Joined:
    Mar 3, 2021
    Posts:
    1
    Hello guys, I am trying to instantiate spectacle on the eyes but I don't know how to manage the position and resize the spectacle after track the eyes.Do anyone help me in this .I've purchased the above plugin also but still I am facing these issues.
     
  8. ScriptedHero

    ScriptedHero

    Joined:
    Apr 23, 2014
    Posts:
    3
    Hey guys! I'm trying to deploy the FaceMaskAdditionalExample scene on Android but getting stuck on loading the shape predictor through "DlibFaceLandmarkDetector_LoadShapePredictor(nativeObj, shapePredictorFilePath);".

    I am unpacking it from streaming asset and saving to persistant data path on android. This works fine on PC and it seems that it saves the file on Android as well to /storage/emulated/0/Android/companyname/files/sp_human_face_68.dat.

    However when I pass the file path to DlibFaceLandmarkDetector_LoadShapePredictor it doesn't seem to load anything. Has anyone tried this before successfully? :)

    Thanks in advance!
     
  9. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    /storage/emulated/0/Android/companyname/files/sp_human_face_68.dat.
    Is this the path you got from "Utils.getFilePath ("sp_human_face_68.dat");"?
     
  10. ScriptedHero

    ScriptedHero

    Joined:
    Apr 23, 2014
    Posts:
    3
    Hello sorry for late reply. Yes (OpenCVForUnity)Utils.getFilePath ("sp_human_face_68.dat"); return following path /storage/emulated/0/Android/data/com.DefaultCompany/files/opencvforunity/sp_human_face_68.dat and it failes to create new FaceLandmarkDetector(sp_human_face_68_dat_filepath);

    However the exact same code work for PC.

    Thanks in advance :)
     
  11. ScriptedHero

    ScriptedHero

    Joined:
    Apr 23, 2014
    Posts:
    3
    Ok so on further investigation I noticed that it cast an exception on Android when calling DlibFaceLandmarkDetector_Init(); "Unable to load DLL 'dlibfacelandmarkdetector': The specified module could not be found."

    Not really sure what to do here.

    UPDATE

    Ok so found this post https://answers.unity.com/questions/1752636/android-64-bit-so-dll-file-not-found-error.html and did what ninjafishnz wrote, changed \Assets\DlibFaceLandmarkDetector\Plugins\Android\libs\arm64-v8a/libdlibfacelandmarkdetector to Android ARM64 and got it to work! :)

    Thanks for taking time and reply!
     
    Last edited: Apr 19, 2021
  12. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    Does this problem also occur with the default Texture2DExample?
    https://github.com/EnoxSoftware/Dli...Examples/Texture2DExample/Texture2DExample.cs

    Could you tell me the environment you tested?
    FaceMaskExample version :
    DlibFaceLandmarkDetector :
    Android Device :
     
  13. QoQoo1321

    QoQoo1321

    Joined:
    Apr 5, 2014
    Posts:
    6
    Hello.
    please tell me.
    I want to track the ugui image at the top left of face detection
    In the code below, it matches at the center, but it shifts toward the edge.
    What's wrong?


    Code (CSharp):
    1.  
    2.  
    3. public RectTransform objimage;
    4.  
    5. void Update()
    6. {
    7. .
    8. .
    9.  
    10.   foreach (var rect in webcamtextureexample.detectResult)
    11.         {
    12.            
    13.             var scalex = Screen.width /1280f;           // 2436/1280  2436..iPhone11
    14.             var scaley = Screen.height / 720;            // 1125/720
    15.  
    16.             //draw landmark points
    17.             //webcamtextureexample.faceLandmarkDetector.DrawDetectLandmarkResult<Color32>(webcamtextureexample.colors, webcamtextureexample.texture.width, webcamtextureexample.texture.height, 4, true, 0, 255, 0, 255);
    18.  
    19.             var x = rect.x * scalex;
    20.             var y = Screen.height - (rect.y * scaley);
    21.  
    22.             objimage.position = new Vector3(x, y, 0);
    23.            
    24.             //draw face rect
    25.             webcamtextureexample.faceLandmarkDetector.DrawDetectResult<Color32>(webcamtextureexample.colors, webcamtextureexample.texture.width, webcamtextureexample.texture.height, 4, true, 255, 0, 0, 255, 2);
    26.             webcamtextureexample.texture.SetPixels32(webcamtextureexample.colors);
    27.             webcamtextureexample.texture.Apply(false);
    28.  
    29.             break;
    30.  
    31.         }
    32.  
    33. }
    34.  
     
  14. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    111
    Dear @EnoxSoftware , after updating to Unity to 2021.2.0b1 without changing anything our WebGL project fails to build with the error:

    Code (CSharp):
    1. Building Library/Bee/artifacts/WebGL/build/debug_WebGL_wasm/build.js failed with output:
    2. emcc:WARNING: --llvm-lto ignored when using llvm backend
    3. emcc2: warning: EXTRA_EXPORTED_RUNTIME_METHODS is deprecated, please use EXPORTED_RUNTIME_METHODS instead [-Wdeprecated]
    4. emcc2: error: /var/folders/qb/xtrq8wsn0lz_s_b08h7n_s2w0000gn/T/tmp8kd72bngopencvforunity.a: File has a suffix of a static library ('.a',), but instead is an LLVM bitcode file! When linking LLVM bitcode files use .bc or .o.
    5. UnityEditor.BuildPipeline:BuildPlayer (UnityEditor.BuildPlayerOptions)
    The key issue here is the error compiling the `tmp8kd72bngopencvforunity.a`. I did read that Unity updated emscripten, so maybe this has broken something in the way the OpenCVForUnity project builds?
    I did see the exact same error for `DlibFaceLandmarkDetector` so it seems to be affecting both assets equally.

    We also tried updating to OpenCV 2.4.4 and DLibFace 1.3.2, still same error.

    PS: Tried again with a clean new project, only added OpenCV and DLibFace, same error.
     
    Last edited: Jun 29, 2021
  15. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    Thank you very much for reporting.
    I will check it.
     
    Bersaelor likes this.
  16. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    111
    Thank you for looking into it @EnoxSoftware. It seems with 2021.2.0b2 the issue remains, was hoping it may be a fluke in 0b1
     
  17. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    111
    @EnoxSoftware can you give a status update about the 2021.2 emscripten issue?
    We're waiting to make a new build of our WebGL app, and we need to use 2021.2 since 2021.1 and before does not get Safari Camera permissions correctly.
     
  18. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    Upon investigation, the following error occurs and the compilation fails. I posted this issue on the Unity forum.
    https://forum.unity.com/threads/fai...tform-contain-native-plugin-files-bc.1137496/
    This issue occurs even with very simple native plugins.
     
    Bersaelor likes this.
  19. lizkin

    lizkin

    Joined:
    Jun 22, 2013
    Posts:
    18
    Hello,

    Is it possible to find a specific person's face with readily available ? What I want to achieve is determine a persons face at image A and find that person at image B. Is it possible with this asset ?
     
  20. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    Unfortunately, the DlibFaceLandmarkDetector does not have such a feature.
     
  21. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,095
    What frame rate does this run on for mobile iOS / Android

    Does it run at 60fps yet for mobile?
     
  22. svapn

    svapn

    Joined:
    May 27, 2018
    Posts:
    2
    Hey Enox,

    I am using your assets of dlib and opencv for my some college project. Its working well. Facial landmarks works but it works bit less accurate. So any suggestions for higher accuracy and stability.
     
  23. MattLovell

    MattLovell

    Joined:
    Oct 17, 2016
    Posts:
    4
    HI @EnoxSoftware !

    Using Dlib/OpenCV, is there a way to just get the detection rectangle returned - in one of the scenarios I'm using DLIB I don't need to detect any landmarks.

    Thanks.
     
  24. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    MattLovell likes this.
  25. MattLovell

    MattLovell

    Joined:
    Oct 17, 2016
    Posts:
    4
    Hi, thanks for that, but what i want to do is just load and run the SVM object detector only, not load the shape predictor file at all.... the faceLandmarkDetector class will only run if only the dat file is supplied or the dat file and the SVM... i was wondering if there was an object only detector that just runs the SVM?
     
    Last edited: Dec 27, 2021
  26. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    If you set shapePredictorFilePath to null, you will only be able to load the objectDetector.
    FaceLandmarkDetector ( objectDetectorFilePath, null );
     
    MattLovell likes this.
  27. MattLovell

    MattLovell

    Joined:
    Oct 17, 2016
    Posts:
    4
    Excellent thank you!
     
  28. unity_0nuttawat0

    unity_0nuttawat0

    Joined:
    Nov 14, 2021
    Posts:
    1
    I set it to width 280 and height 200 but it still big and didn't make any change
     
  29. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    It is possible that your WebCamera does not support the desired frame size.
     
  30. Wuna

    Wuna

    Joined:
    Nov 4, 2015
    Posts:
    7
    Hi, i want to ask a question, how many faces can the library realistically tracks simultaneously?
     
  31. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    This asset provides the ability to detect facial landmarks, but does not provide face tracking.
    There is no limit to the number of faces that can be detected at one time.
     
  32. Wuna

    Wuna

    Joined:
    Nov 4, 2015
    Posts:
    7
    Thank you, another question, is there any way to improve detection with person wearing a mask?
     
  33. mchangxe

    mchangxe

    Joined:
    Jun 16, 2019
    Posts:
    69
    hey guys, great tool thanks so much. I have one question regarding the ARHead example with openCV. I've noticed that the ar head flickrs quite alot in this example, as well as the DLIB facial points, whereas in OpenCV's library examples, although theres no ar head features, the DLIB facial points are quite stable. Any ideas how i can improve the quality? I am using a 720p webcam, maybe upping quality to 1080p can help?
     
  34. Bersaelor

    Bersaelor

    Joined:
    Oct 8, 2016
    Posts:
    111
    I just switched to 2021.3 and I think this package (and OpenCV for unity) needs an update. The only way I could get my WebGL builds to work was to remove the 2019 folder's file (I think Unity reverted to using the 2019 plugin):
     

    Attached Files:

  35. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    After importing, could you run "Set Plugin Import Settings"?
    MenuItem.png
    MenuItem.PNG
     
    Bersaelor likes this.
  36. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,095
    Does this work with Unity 2021.3.3f1 WebGL? dlibfacelandmarkdetector machine type must be wasm32 or wasm64
     
    pulinc39 likes this.
  37. asimjee021

    asimjee021

    Joined:
    Jun 10, 2022
    Posts:
    1
    Can We track Eyes Moving Left right using Dlib
     
  38. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
  39. prognosist

    prognosist

    Joined:
    Jun 4, 2017
    Posts:
    2
    Hello! We are interesting in a solution, which will let us set a mask from an image and put in on the needed place. Current solution moves mask over the detected face now. But we need to change a part of the image (a person's face on photo). Where do we need to dig? How can we achieve this aim? We just need to get a user's face and let him change the face of an image.
    Thank you a lot!
     
  40. CrisSkyeDev

    CrisSkyeDev

    Joined:
    Aug 16, 2022
    Posts:
    1
    it is possible to change the limit distance of the camera with respect to the person, because for now it seems that the maximum is 70 cm
     
  41. LT23Live

    LT23Live

    Joined:
    Jul 8, 2014
    Posts:
    99
    @EnoxSoftware does DLIB now support GPUs?I caught a thread from 6 years ago saying it did not. Is that still the case?
     
  42. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    The shape_predictor and frontal_face_detector used in DlibFaceLandmarkDetector do not support GPUs.
    https://github.com/davisking/dlib/issues/2305
     
    LT23Live likes this.
  43. darrenbellenger

    darrenbellenger

    Joined:
    Feb 16, 2014
    Posts:
    6
    Hi. I've used the webCamTextureToMatHelper example in my project which is running at nearly 400fps in the FpsMonitor.

    But as soon as introduce the function below, the fps drops to 16:
    List<UnityEngine.Rect> detectResult = faceLandmarkDetector.Detect();

    Is there anyway to make this function run faster?

    EDIT: Now realised I'm mixing your OpenCV and Dlib packages together, when I shouldn't - is this the reason for the speed issue?
     
    Last edited: Mar 7, 2023
  44. mcroswell

    mcroswell

    Joined:
    Jan 6, 2010
    Posts:
    80
    I used DLib years ago on my mac and am about to start playing with some custom characters that have facial bones. Are there any additional values to this plugin that is not already in the original DLib?
     
  45. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    Could you try the FrameOptimizationExample?
    The faceLandmarkDetector.Detect() method takes longer to process than OpenCV's face detection. Therefore, the FrameOptimizationExample is optimized by using OpenCV's face detection instead.
    https://github.com/EnoxSoftware/Dli...timizationExample/FrameOptimizationExample.cs
     
  46. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,581
    The model included with DlibFaceLandmarkDetector has a different dataset than the original model, but the order and position of the facial feature points are the same.

    http://blog.dlib.net/2018/01/correctly-mirroring-datasets.html

    dlib1.2.6_features.png
     
  47. Harko_yo

    Harko_yo

    Joined:
    Feb 25, 2017
    Posts:
    6
    Hi, i'm testing this example and trying to change or add objects to the head (we want to put hats, glasses and other things), but the new elements are not showing on play.

    For now i'm just putting basic 3d objects (cubes, etc) and change the head reference in Quad script to it. But no mather where i put it , is not showed on screen. They need to use a specific shader to work or something like that?
     
  48. Harko_yo

    Harko_yo

    Joined:
    Feb 25, 2017
    Posts:
    6
    Ok nevermind.

    It was the layer. No UI layer, no party.
     
  49. Harko_yo

    Harko_yo

    Joined:
    Feb 25, 2017
    Posts:
    6
    One more thing. Is there any tool or something to put the points on a mask? Or move every one of them manually is the only way?
     
  50. Harko_yo

    Harko_yo

    Joined:
    Feb 25, 2017
    Posts:
    6
    I think i finally make the merge of the ARHead with facemask example and the MultiHead example (plus the eye transparency on the mask) works somehow.

    But still the first ARHead don't want to be desactivated for some reason. And some times the gameobject hosting the head shows inverted and far away from the mask.