Search Unity

[RELEASED] Dlib FaceLandmark Detector

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Jun 4, 2016.

  1. integence

    integence

    Joined:
    Feb 17, 2012
    Posts:
    19
    Hi

    Is there any software or plugin for some popular graphic editor (like Photoshop), to easily draw and export custom landmark coordinates from image?

    I would like to use some fancy animal face that need custom 68 points coordinates. I can measure it and write it manually, but it is a hell lot of a work. It would be nice to have some software to "draw" landmarks on image, and then export a ready to use landmarks array :)

    Thanks for this great asset by the way :)
     
  2. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    This tool may be useful.
    https://www.mobilefish.com/services/record_mouse_coordinates/record_mouse_coordinates.php
     
  3. belalbadr

    belalbadr

    Joined:
    Jan 9, 2013
    Posts:
    3
    Hi,

    I bought opencv and Dlib can you help me to apply hair style not face mask , I want only to put 2d image hair on the user head-hair ?

    Thanks,
     
  4. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    I have found such a use example.
    https://twitter.com/miyumiyuna5/status/841924714058641409
    It may be feasible by displaying the hair model instead of displaying the red head model.
     
  5. neomarian

    neomarian

    Joined:
    May 11, 2012
    Posts:
    42
    Hi EnoxSoftware,
    I bought Dlib FaceLandmark Detector and OpenCV for Unity assets and are working great but I have a simple question: I want to create my dataset file by using train_shape_predictor_ex.cpp but I don't know what settings to use for shape_predictor_trainer. I tried leave the existing ones from the cpp file but I'm not sure that those ones are the best (_oversampling_amount = 300, _nu = 0.05 and _tree_depth = 2). Can you please share what settings (what parameters values) are you using when you create sp_human_face_68.dat and sp_human_face_68_for_mobile.dat ?
     
  6. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    https://sourceforge.net/p/dclib/discussion/442517/thread/71b4b7c8/
    cascade_depth = 15
    tree_depth = 4 //sp_human_face_68.dat
    or
    tree_depth = 2 //sp_human_face_68_for_mobile.dat
    The other parameters are the default values.

    I changed "train_shape_predictor_ex.cpp" as follows.
    Code (CSharp):
    1.         // Now make the object responsible for training the model.
    2.         shape_predictor_trainer trainer;
    3.         // This algorithm has a bunch of parameters you can mess with.  The
    4.         // documentation for the shape_predictor_trainer explains all of them.
    5.         // You should also read Kazemi's paper which explains all the parameters
    6.         // in great detail.  However, here I'm just setting three of them
    7.         // differently than their default values.  I'm doing this because we
    8.         // have a very small dataset.  In particular, setting the oversampling
    9.         // to a high amount (300) effectively boosts the training set size, so
    10.         // that helps this example.
    11.         //trainer.set_oversampling_amount(300);
    12.         // I'm also reducing the capacity of the model by explicitly increasing
    13.         // the regularization (making nu smaller) and by using trees with
    14.         // smaller depths.
    15.         //trainer.set_nu(0.05);
    16.         trainer.set_tree_depth(4);
    17.         //trainer.set_tree_depth(2);
    18.  
    19.         // some parts of training process can be parallelized.
    20.         // Trainer will use this count of threads when possible
    21.         trainer.set_num_threads(4);
    22.  
    23.         trainer.set_cascade_depth(15);
     
    neomarian likes this.
  7. neomarian

    neomarian

    Joined:
    May 11, 2012
    Posts:
    42
    Another question: if I train any kind of object I want along with some landmark points for shape predictor then is fine to use the code from your CatDetectionExample.cs ?
    Code (CSharp):
    1. FaceLandmarkDetector faceLandmarkDetector = new FaceLandmarkDetector (frontal_cat_face_svm_filepath, sp_cat_face_68_dat_filepath);
    Or this is working only with face kind object?
     
  8. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    FaceLandmarkDetector works with non-face models as well.
     
    neomarian likes this.
  9. neomarian

    neomarian

    Joined:
    May 11, 2012
    Posts:
    42
    Hi,
    There is any chance to can use dlib CNN based object detection?
     
  10. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
  11. neomarian

    neomarian

    Joined:
    May 11, 2012
    Posts:
    42
  12. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
  13. Bigg_P

    Bigg_P

    Joined:
    Jul 16, 2014
    Posts:
    9
    Hi.
    Does anyone have this weird issue?
    While using ARHead example,the webcam texture is very dark in some android devices (eg OnePlus 3) and works great on other Devices.
    is there any solution for this problem?
     
  14. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    I think that this problem is probably caused by a bug in UnityEditor.
    Please change the initialization parameter of WebCamTexture.
    requestedFPS=1 rear camera(fps is 30) is fine. front camera(fps is 15) is fine.
    https://forum.unity.com/threads/released-opencv-for-unity.277080/page-33#post-3445178
     
    Bigg_P likes this.
  15. deadlyGolum

    deadlyGolum

    Joined:
    Dec 31, 2013
    Posts:
    8
    hi @EnoxSoftware ,

    I have question about AR head detection. I have been following the example for video and webcam; its working fine.
    What i am try to do is to get the AR head example to work with a Supplied texture2D and its not working as the AR camera matrix might be not supporting for Texture2D.

    Can you guide me, How to make the AR head example to work for an Texture2D image [not from camera, i want to load a normal image as texture2D and have the AR example work]?

    Thanks.
     
  16. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    Could you try this example?
    https://www.dropbox.com/s/4z36obqxpy0hpkr/ARHeadTexture2DExample.unitypackage?dl=0
    ARHeadTexture2DExample.PNG
     
  17. deadlyGolum

    deadlyGolum

    Joined:
    Dec 31, 2013
    Posts:
    8
  18. denisatglaza

    denisatglaza

    Joined:
    May 14, 2018
    Posts:
    18
    Hello! I ran webcam AR scene on my Samsung Galaxy s7 and got only 10 fps. Is there anyway to get higher fps or is the phone not powerful enough?

    Thank you!
     
  19. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    I think that FrameOptimizationExample will be helpful.

    The face detection step is faster using OpenCV than using Dlib. In FaceMaskExample, face detection is done with OpenCV for speeding up.
    Also,
    The smaller the OpenCV Mat size is, the shorter the processing time becomes.
    There is a possibility that the processing speed is faster when you set RequestWidth and RequestHeight smaller value.
     
  20. neomarian

    neomarian

    Joined:
    May 11, 2012
    Posts:
    42
    Hi,
    I'm testing your DNN TensorFlowExample and I have a hard time in getting the detected rects and the edge points. Can you help me with these or can you lead me to the correct way?
    Also, do you have any idea how to get the output shape from TensorFlow?
     
    Last edited: May 15, 2018
  21. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    It is not possible to detect the rects of objects using "tensorflow_inception_graph.pb" model.
    Please refer to this post.
    https://forum.unity.com/threads/released-opencv-for-unity.277080/page-33#post-3455418
     
  22. neomarian

    neomarian

    Joined:
    May 11, 2012
    Posts:
    42
    Got it, thank you. That worked fine.
    There is any way to detect the shape with any of these 3, Caffe, TensorFlow of Yolo? I mean to get the shape points not just the rect.
     
    Last edited: May 17, 2018
  23. denisatglaza

    denisatglaza

    Joined:
    May 14, 2018
    Posts:
    18
    Thank you! I ran FrameOptimizationExample and FaceMask example and 1st one is faster. Is there a way to run WebCamAR scene with OpenCV as well? I am trying to make a 3d filter that has to do with blinking and various 3d objects and so far was able to play around with the red head object and eye lasers in WebCamAR. So is there a way to either transfer those objects to FrameOpt example so that i could see them in hierarchy or enable opencv for WebCamAR?

    I would really prefer to enable OpenCV for WebCamAR or speed it up since i had success working with it. :)

    Thank you!!
     
    Last edited: May 22, 2018
  24. HonorableDaniel

    HonorableDaniel

    Joined:
    Feb 28, 2007
    Posts:
    2,877
    I own OpenCV and dlib, it seems OpenCV detects eye direction. Any way to do this with dlib?
     
  25. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    It is not possible to estimate the direction and angle of the face only with the function of DlibFaceLandmarkDetector.
     
  26. GiveMe6

    GiveMe6

    Joined:
    Mar 8, 2018
    Posts:
    1
    ina likes this.
  27. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
  28. NathanNegreiroMyPad3D

    NathanNegreiroMyPad3D

    Joined:
    May 10, 2018
    Posts:
    4
    Hello! I've been working on a project that uses Dlib and so far everything has been good. However, I am trying to download a picture from the internet and then process it and detect the different facial landmarks. I pass the texture into the same logic that has been working thus far, but now it won't Dlib detects a face, but then when I try to detect the facial landmarks it returns an empty array.

    I am saving the texture to the disk in StreamingAssets before trying to process it. If I try to use that same image later on (in a different scene, same script to process the texture though), it works fine! Any clues on what could be causing this odd behavior and what I could do to fix it?
     
  29. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    Is the following error displayed?
    Code (CSharp):
    1. UnityException: Texture '' is not readable, the texture memory can not be accessed from scripts. You can make the texture readable in the Texture Import Settings.
    Texture2D used by DlibFaceLandmarkDetector must be readable.
    https://forum.unity.com/threads/tex...est-gettexture-is-not-readable-solved.447163/
     
  30. bastianmeyerbm3

    bastianmeyerbm3

    Joined:
    May 22, 2018
    Posts:
    15
    Hi, how can I add my own masks in the FaceMaskExample and not the ones used in the Resources folder??
    And can I add a mask that isn't using the whole face but the eyes, like glasses?

    Edit: Can I use a different picture for Landmarking the Facepoints and use an other mask based on the picture for Landmarking?
     
    Last edited: Jun 4, 2018
  31. NathanNegreiroMyPad3D

    NathanNegreiroMyPad3D

    Joined:
    May 10, 2018
    Posts:
    4
    Unfortunately it does not. I get no errors or warnings at all. I call the DetectLandmark function in FaceLandmarkDetector.cs and when it gets to the internal DetectLandmark call, it just returns a count of 0. It returns the rect of the face perfectly fine, but its just the landmarks where it seems to fail.
     
  32. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    Thank you very much for reporting.
    Could you send me a simple code to reproduce this problem?
    https://enoxsoftware.com/opencvforunity/contact/technical-inquiry/
     
  33. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    You can add new mask data by the following procedure.
    add_new_mask_data.png
     
  34. denisatglaza

    denisatglaza

    Joined:
    May 14, 2018
    Posts:
    18
    Hello. In order to speed up the app I've been playing with height and width in Quad's webcam texture to mat helper. There are two options which I don't understand their purpose: requested FPS and Timeoutframe count.
    Whatever number I set for them, doesn't seem to change anything for my FPS. Can you please describe what do they do and how do they affect performance/fps?

    Also, is there an option to skip the frame somewhere? I've read DLIB speed up manual but didn't understand how to do it.

    Thank you!
     
  35. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    requestedFPS is the initialization parameter of Unity's WebCamTexture class.
    https://docs.unity3d.com/ScriptReference/WebCamTexture-requestedFPS.html

    Timeoutframe count is the number of frames until generating a TIMEOUT error when acquiring images of WebCamTexture.

    Please see "DlibFaceLandmarkDetectorWithOpenCVExample / ImageOptimizationHelper.cs".
    ImageOptimizationHelper.cs is used to skip frames with FrameOptimizationExample.
    https://github.com/EnoxSoftware/Dli...ationExample/FrameOptimizationExample.cs#L282

    ImageOptimizationHelper.PNG
     
  36. denisatglaza

    denisatglaza

    Joined:
    May 14, 2018
    Posts:
    18
    When running ARwebcam example I've noticed that even if the head is held perfectly still, the mesh is twitching. Is there a way to make it more still so that animations don't appear jumping?


    Gif.gif
     
    wookxxx likes this.
  37. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    Could you try changing the value of the low pass filter?
    The larger the value, the more filter the vibration.
    dlib_ar_lowpass_filter.png
     
  38. denisatglaza

    denisatglaza

    Joined:
    May 14, 2018
    Posts:
    18
    I tried playing around with those numbers but its still vibrating. The more I increase those numbers, the head become more still but then it affects the performance to the point where I move my head but filter is still (and still rocking back and forth but less frequent). Is there any other method to help it to stabilize?

    Thank you very much!
     
  39. RiccardoAxed

    RiccardoAxed

    Joined:
    Aug 29, 2017
    Posts:
    58
    Hi, I already own OpenCV for Unity and I'm thinking to have a go with Dlib FaceLandmark Detector, but first I would just understand if it fits my needs.

    I should develop a couple of projects that basically require face tracking AND recognition, applied to public installations. So, because of this last point, I can't rely on some prior learning process, because I've to track and recognize people passing by "on the fly", then store face data and compare it with a db of persons already passed by, to establish if he's one of them or a new one.

    Is it possibile to reach that kind of result with this specific plugin? Or maybe with another one of your plugins?
     
  40. bastianmeyerbm3

    bastianmeyerbm3

    Joined:
    May 22, 2018
    Posts:
    15
    Hi sorry for the late response but my screen looks a bit different. I dont have the "FaceMaskData" Object. Whats wrong with my version? Screenshotssss.png
     
  41. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    DlibFaceLandmarkDetector has no facial recognition function.
    Have you already tried RealTime FaceRecognition Example?
     
  42. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    Please import FaceMask Example v1.0.6 or later.
     
  43. RiccardoAxed

    RiccardoAxed

    Joined:
    Aug 29, 2017
    Posts:
    58
    Thank you for the answer.

    I still haven't tried it but i'll do for sure.
    Do you mean I could use it - maybe making some adjustments - to fulfill my project's requirements?
     
  44. denisatglaza

    denisatglaza

    Joined:
    May 14, 2018
    Posts:
    18
    I tried playing around with those numbers but its still vibrating. The more I increase those numbers, the head become more still but then it affects the performance to the point where I move my head but filter is still (and still rocking back and forth but less frequent). Is there any other method to help it to stabilize?

    Thank you very much!
     
  45. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    RealTime Face Recognition Example is a tutorial example project of detecting a face and recognizing a face using only OpenCVforUnity.
    I do not know if this example meets your project requirements, but I think it will be useful for creating projects.
     
  46. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    If you change "objectPoints68" and "imagePoints" you set for the Calib3d.solvePnP () method, the accuracy of the position estimate may be improved.
    https://github.com/EnoxSoftware/Dli...eadExample/ARHeadWebCamTextureExample.cs#L523
    Calib3d.solvePnP ()
    https://docs.opencv.org/3.4.0/d9/d0c/group__calib3d.html#ga549c2075fac14829ff4a58bc931c033d
     
  47. bastianmeyerbm3

    bastianmeyerbm3

    Joined:
    May 22, 2018
    Posts:
    15
    Okay thank you for the reply!! I updated the FaceMaskExample and when i want to track a face i get an error:

    ArgumentException: Invalid landmark_points.
    FaceMaskExample.FaceMaskColorCorrector.GetFacialContourPoints (System.Collections.Generic.List`1 landmark_points, OpenCVForUnity.Point[] dst_points) (at Assets/Addons/FaceMaskExample/Scripts/FaceMaskColorCorrector/FaceMaskColorCorrector.cs:108)
    FaceMaskExample.FaceMaskColorCorrector.UpdateLUTTex (Int32 id, OpenCVForUnity.Mat src, OpenCVForUnity.Mat dst, System.Collections.Generic.List`1 src_landmarkPoints, System.Collections.Generic.List`1 dst_landmarkPoints) (at Assets/Addons/FaceMaskExample/Scripts/FaceMaskColorCorrector/FaceMaskColorCorrector.cs:55)
    FaceMaskExample.VideoCaptureFaceMaskExample.CorrectFaceMaskColor (Int32 id, OpenCVForUnity.Mat src, OpenCVForUnity.Mat dst, System.Collections.Generic.List`1 src_landmarkPoints, System.Collections.Generic.List`1 dst_landmarkPoints) (at Assets/Addons/FaceMaskExample/VideoCaptureFaceMaskExample/VideoCaptureFaceMaskExample.cs:595)
    FaceMaskExample.VideoCaptureFaceMaskExample.Update () (at Assets/Addons/FaceMaskExample/VideoCaptureFaceMaskExample/VideoCaptureFaceMaskExample.cs:458)


    Sorry for causing that much trouble. HELP.png
     
  48. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    Please move the “DlibFaceLandmarkDetector/StreamingAssets/” folder to the “Assets/” folder.
    Setup.PNG
     
  49. denisatglaza

    denisatglaza

    Joined:
    May 14, 2018
    Posts:
    18
    Thank you! I see what you are suggesting. Is there some sort of a map with x,y,z's that I can look at to base my own points on?? I see all the points that were created but if I change the numbers I don't really know what am I changing space wise. For example based on 68 map, right mouth breath is number 54 and its coordinates are Point3 (26, 15, 83). Where do I get the map with those coordinates?

    Also, would you recommend changing only existing points or actually adding my own?
     
  50. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,164
    Please open "human_male.fbx" in the 3D model editor, check 3D point coordinates and set 3D point coordinates (objectPoints) corresponding to 2D point coordinates (imagePoints).
    I do not know if it is better to increase the point to more than 7 points.
    dlibfacepoint.png
    ar_fbx.PNG
    ar_fbx_3d.png

    Also, this page is very helpful.
    http://www.morethantechnical.com/20...timation-with-opencv-opengl-revisited-w-code/
     
    ina likes this.
unityunity