Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  2. Ever participated in one our Game Jams? Want pointers on your project? Our Evangelists will be available on Friday to give feedback. Come share your games with us!
    Dismiss Notice

[RELEASED] Dlib FaceLandmark Detector

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Jun 4, 2016.

  1. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    Have the "DlibFaceLandmarkDetector / StreamingAssets /" folder been moved to the "Assets/" folder?
    Setup.PNG
     
  2. mimminito

    mimminito

    Joined:
    Feb 10, 2010
    Posts:
    780
    If I was to add a 3D object onto the users eyes do I need OpenCV as well? @EnoxSoftware
     
    Last edited: Dec 14, 2016
  3. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    WebCamTextureARSample and VideoCaptureARSample use the OpenCV's Calib3d.solvePnP() method to calculate the camera matrix.
     
  4. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    Dlib FaceLandmark Detector v1.0.5 is now available.

    Version changes
    1.0.5
    [Common]Fixed WebCamTextureToMatHelper class.
    [Common]Added Utils.getVersion().
    [Common]Fixed Utils.getFilePathAsync().
     
  5. QUDAGAMES

    QUDAGAMES

    Joined:
    Dec 25, 2016
    Posts:
    2
    does it support remote IP camera ? or just local camera ?
     
  6. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    DlibFaceLandmarkDetector supports to Unity's WebCamTexture. If IP Camera is compatible with WebCamTexture, DlibFaceLandmarkDetector SetImage() method will work.
    Also, The SetImage() method also supports a pixel byte array as an argument.

    http://enoxsoftware.github.io/DlibF...mark_detector_1_1_face_landmark_detector.html
    Code (CSharp):
    1. void     SetImage (Texture2D texture2D)
    2. void     SetImage (WebCamTexture webCamTexture)
    3. void     SetImage (WebCamTexture webCamTexture, Color32[] bufferColors)
    4. void     SetImage (IntPtr intPtr, int width, int height, int bytesPerPixel)
    5. void     SetImage (IntPtr intPtr, int width, int height, int bytesPerPixel, bool flip)
    6. void     SetImage< T > (IList< T > array, int width, int height, int bytesPerPixel)
    7. void     SetImage< T > (IList< T > array, int width, int height, int bytesPerPixel, bool flip)
     
  7. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    When setting MenuItem[Tools/Dlib FaceLandmark Detector/Set Plugin Import Settings], it seems that the following error occurs on Unity5.5 or later.

    No valid name for platform: 11
    UnityEditor.PluginImporter:SetCompatibleWithPlatform(BuildTarget, Boolean)
    DlibFaceLandmarkDetector.DlibFaceLandmarkDetectorMenuItem:SetPlugins(String[], Dictionary`2, Dictionary`2) (at Assets/DlibFaceLandmarkDetector/Editor/DlibFaceLandmarkDetectorMenuItem.cs:193)
    DlibFaceLandmarkDetector.DlibFaceLandmarkDetectorMenuItem:SetPluginImportSettings() (at Assets/DlibFaceLandmarkDetector/Editor/DlibFaceLandmarkDetectorMenuItem.cs:124)


    To fix, Replace line 192 - line 194 with the following code.

    from
    Code (CSharp):
    1.                                                 foreach (BuildTarget target in Enum.GetValues(typeof(BuildTarget))) {
    2.                                                     if ((int)target != -1)pluginImporter.SetCompatibleWithPlatform (target, false);
    3.                                                 }
    to
    Code (CSharp):
    1.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.Android, false);
    2.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.iOS, false);
    3.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneWindows, false);
    4.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneWindows64, false);
    5.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneOSXIntel, false);
    6.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneOSXIntel64, false);
    7.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneOSXUniversal, false);
    8.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneLinux, false);
    9.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneLinux64, false);
    10.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.StandaloneLinuxUniversal, false);
    11.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.WSAPlayer, false);
    12.                         pluginImporter.SetCompatibleWithPlatform (BuildTarget.WebGL, false);
    This bug will be fixed in the next version.
     
  8. minimalmally

    minimalmally

    Joined:
    Oct 21, 2016
    Posts:
    12
    I tried to attach MovieTexture to an ARobject, but it doesn't play the sound. Does it support sound?
     
    Last edited: Dec 30, 2016
  9. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    Are you using the MovieTexture class? https://docs.unity3d.com/Manual/class-MovieTexture.html
    It seems necessary to control the sound by script.
    Code (CSharp):
    1.             var audioSource = GetComponent<AudioSource>();
    2.             audioSource.clip = movieTexture.audioClip;
    3.             audioSource.loop = true;
    4.             audioSource.Play ();
     
    minimalmally likes this.
  10. minimalmally

    minimalmally

    Joined:
    Oct 21, 2016
    Posts:
    12
    Is it possible for WebCamTextureARSample to use CatDetectionSample? if so, how? Basically, I want to use AR with the Cat Sample.
     
  11. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    ”frontal_cat_face.svm" and "shape_predictor_68_cat_face_landmarks.dat" are the files that I trained. Because this file is not good detection accuracy, It is not work well on WebCamTexture.
     
  12. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    873
    What's the recommended resolution for front facing camera and rear camera?
     
  13. ShawnFeatherly

    ShawnFeatherly

    Joined:
    Feb 22, 2013
    Posts:
    40
    When an Android phone is in portrait mode, the WebCamTextureSample demo does not show the red box or green face overlay. The OpenCV webcam demos do show the overlay in any orientation. Is there something I can do to get the overlay to show in portrait mode without OpenCVforUnity? Maybe an edit to WebCamTextureSample.cs?

    These results were from testing DlibFaceLandmarkDetector v1.0.6 and the Android app (https://play.google.com/store/apps/details?id=com.enoxsoftware.dlibfacelandmarkdetectorsample) on a Nexus5 and a MotoG3.
     
    ina likes this.
  14. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    WebCamTextureSample can detect landscape direction only.
    写真 2016-08-10 17 20 50.png
    But, WebCamTextureToMatSample can detect landscape and portrait direction using OpenCV's method.
     
  15. ShawnFeatherly

    ShawnFeatherly

    Joined:
    Feb 22, 2013
    Posts:
    40
    I re-read through earlier posts on this thread, you can do portrait mode with only this asset. This saves, among other things, on build size.

    Thank you so much @shawhu for posting your code: https://forum.unity3d.com/threads/released-dlib-facelandmark-detector.408958/#post-2748104

    I had to make a small fix. adding [i ] into result[i ] = input[x + y * width];
    As @shawhu suggested, I swapped width and height as follows to match the rotated value in colors:
    Code (CSharp):
    1. #if UNITY_ANDROID && !UNITY_EDITOR
    2. colors = RotateCW(colors, webCamTexture.width, webCamTexture.height);
    3. ...
    4. faceLandmarkDetector.SetImage<Color32> (colors, webCamTexture.height, webCamTexture.width, 4, flip);
    5. ...
    6. faceLandmarkDetector.DrawDetectLandmarkResult<Color32> (colors, webCamTexture.height, webCamTexture.width, 4, flip, 0, 255, 0, 255);
    7. ...
    8. faceLandmarkDetector.DrawDetectResult<Color32> (colors, webCamTexture.height, webCamTexture.width, 4, flip, 255, 0, 0, 255, 2);
    9. ...
    10. colors = RotateCCW(colors, webCamTexture.height, webCamTexture.width);
    Each facePoint in the points[] set also needs to be converted:
    Code (CSharp):
    1. #if UNITY_ANDROID && !UNITY_EDITOR
    2.         // correct rotated value sent to faceLandmarkDetector
    3.         facePoint.x += (320 - 240) / 2f;
    4.         facePoint.y += (240 - 320) / 2f;
    5. #endif
     
    Last edited: Jan 12, 2017
    colenso and EnoxSoftware like this.
  16. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    Dlib FaceLandmark Detector v1.0.7 is now available.

    Version changes
    1.0.7
    [Common]Fixed WebCamTextureToMatHelper.cs.(flipVertical and flipHorizontal flag)

    1.0.6
    [Common]Fixed OpenCVForUnityMenuItem.cs.(No valid name for platform: 11 Error)
    [Common]Added OptimizationWebCamTextureToMatHelper.cs.
     
  17. luisanton

    luisanton

    Joined:
    Aug 25, 2009
    Posts:
    325
    The face detection step - when you actually find faces and store then in a list of Rect - seems to be much, much slower than the landmark detection step. Would it be possible in order to speed it up to use a much lower image resolution for face detection (I understand you don't need big images if faces are big enough), keeping the higher resolution image for landmark detection? It seems both steps currently use the same image.
     
  18. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    It is possible to speed up processing by downscaling image. OptimizationSample is optimized by downscale and skipframe using "OpenCV for Unity".
    Also, Since face detection is faster using OpenCV than using Dlib,
    I think that using "OpenCV for Unity" instead of "DlibFaceLandmarkDetector" will speed up face detection.
     
  19. SunnyChow

    SunnyChow

    Joined:
    Jun 6, 2013
    Posts:
    342
    just curious, do you think it can track an open hand?
     
    theolagendijk likes this.
  20. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    If using HOGObjectDetector trained for open hand, I think that it is possible to do so.However, the HOGObjectDetector may not be robust against rotation.
     
  21. mcmelmac

    mcmelmac

    Joined:
    Apr 23, 2014
    Posts:
    2
    How is the dlib compiled?
    Which compiler optimizations did you turn on, especially for the OSX version. SSE2/4/AVX?
    Because running dlib examples inside a VM on OSX is actually faster than running your dlib examples on the same machine in Unity.
     
  22. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    SSE2 is enabled now.
    I tried the SSE 4 and AVX, but the FPS improved slightly.
    In the future release, I am going to include a plug-in that enabled SSE4 or AVX in the asset.
     
  23. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    Dlib FaceLandmark Detector v1.0.9 is now available.

    Version changes
    1.0.9
    [WebGL]Added WebGL Plugin for Unity5.6.

    1.0.8
    [Common]Changed the name of asset project.("Sample" to "Example")
    [Common]Fixed VideoCaptureARExample and WebCamTextureARExample.
     
  24. SunnyChow

    SunnyChow

    Joined:
    Jun 6, 2013
    Posts:
    342
    but i think HOGObjectDetector just return a rectangle, i am talking about like the face tracking, which return the position of key points.
     
  25. MorphVGX

    MorphVGX

    Joined:
    Jun 21, 2010
    Posts:
    57
    public void SetImage<Color32> (List<Color32> array, int width, int height, int bytesPerPixel, bool flip) seems to be causing memory leaks on an iPhone 4s when called in OnPreviewUpdate from NatCam . Maybe it is not a memory leak but the memory keeps going up until it crashes and CPU goes to 100%. Any idea what can be causing this?
     
  26. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    There is currently no such example using DlibLandmarkDetector. But,There seems to be such an implementation example using OpenCV for Unity.
     
  27. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    Do memory leaks occur even when only DlibFaceLandmarkDetector is used?
     
  28. yamamotoBT

    yamamotoBT

    Joined:
    Oct 31, 2014
    Posts:
    7
    Please help Please tell me how to change face exchange including "forehead" with "Texture2DToMatSample"?
     
  29. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    shape_predictor_68_face_landmarks.dat is the shapepredictor file which dlib provides.
    If you want to add the landmark points, you need to train the more landmarks shapepredictor file including "forehead".
    http://dlib.net/train_shape_predictor_ex.cpp.html
    http://stackoverflow.com/questions/36711905/dlib-train-shape-predictor-ex-cpp?rq=1
    http://stackoverflow.com/questions/...ape-predictor-for-194-landmarks-helen-dataset
     
  30. yamamotoBT

    yamamotoBT

    Joined:
    Oct 31, 2014
    Posts:
    7
    Thank you for answering
    Can you share the image file you used with the xml file you created with imglab?
     
  31. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
  32. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    Dlib FaceLandmark Detector v1.1.0 is now available.

    Version changes
    1.1.0
    [Win][Mac][Linux][UWP]Added the native plugin file enabled SSE4 or AVX compiler option.
     
  33. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    873
    Hi, do you have
    Do you have a snippet/example for a cropped matrix?

    If not - I assume this is declaring mat to be cropped size in WebCamTextureTOMathelper.cs and in Utils.webCamTextureToMat() - custom populate colors based on crop size
     
  34. yamamotoBT

    yamamotoBT

    Joined:
    Oct 31, 2014
    Posts:
    7
  35. rforgeon

    rforgeon

    Joined:
    Jul 3, 2015
    Posts:
    6
    How do I add a new face_mask in the FaceMaskExample > Resources? I've added a new image with the same settings as the other masks. When changing my image's name to "face_mask5" it then inherits the shape of the examples "face_mask5". How do I change/debug the face points for my new image?
     
  36. rforgeon

    rforgeon

    Joined:
    Jul 3, 2015
    Posts:
    6
    Found the "ExampleDataSet.cs" Any good suggestions for point placing tools?
     
  37. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    To add a new face, you need to add the face information you want to add to "Assets \ FaceMaskExample \ Scripts \ Data \ ExampleDataSet.cs".If you add a face that can not be automatically detected with a face detector like a panda or an animated face, you need to manually add landmark point data.You also need to add the image of the face you want to add to the "Resource" folder.
    Please refer to this image for index of face point.
    1-AbEg31EgkbXSQehuNJBlWg.png
     
  38. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    873
    For the AR mask example, what is the scene to real world scale to be able to size real objects properly?
     
  39. luisanton

    luisanton

    Joined:
    Aug 25, 2009
    Posts:
    325
    I'm experiencing a weird behaviour on Android. We recently released a free app on iOS that uses this solution and works like a charm.

    On Android I can Build&Run so it's uploaded to my Android device and it runs. My device is old and therefore it's a bit slow, but it works.

    However, when I publish this app in Google Play (beta) and download it on my device... it doesn't detect my face. According to logcat, the model file is not found! (I have not moved it, it's still in StreamingAssets and it does work when I build & run)

    Is anyone experiencing this too?
     
  40. Gustavo-Quiroz

    Gustavo-Quiroz

    Joined:
    Jul 26, 2013
    Posts:
    36
    Did you split the APK?
    If you did, you will have to tell your app where exactly is your data.
     
  41. luisanton

    luisanton

    Joined:
    Aug 25, 2009
    Posts:
    325
    That's it! I've never split an APK before, so I was not expecting this! No idea Unity splitted apps automatically when they are larger than 100Mb. But indeed, there's an OBB file alongside the apk file. I'll figure out how to load it, thanks! (here's the doc, just in case, because given the size of most model files others may experience the same problem https://docs.unity3d.com/Manual/android-OBBsupport.html)
     
    Gustavo-Quiroz likes this.
  42. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    873
    Any advice for changing the human_male gameobject scale based on different head sizes in the AR Sample?
     
  43. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    https://play.google.com/store/apps/details?id=com.enoxsoftware.dlibfacelandmarkdetectorexample
    This demo app is downloading the OBB file at the first launch. I referred to the following pages.
    http://naochang.me/?p=488
    https://www.assetstore.unity3d.com/jp/#!/content/3189
     
  44. smtabatabaie

    smtabatabaie

    Joined:
    May 6, 2013
    Posts:
    44
    @EnoxSoftware I tried the FaceMask application from Google Play, I wanted to ask if optimization explained here was applied to that application.
    Thanks
     
  45. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    Yes.
    • Use OpenCV face detection instead of dlib's face detection - Processing speed of OpenCV face detection is faster than Dlib's face detection.
     
  46. martinamenegon

    martinamenegon

    Joined:
    Aug 28, 2014
    Posts:
    29
    would it be possible to send the data via OSC to another software (like maxmsp)?
     
  47. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
  48. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    873
    is the coordinate system bottom left hand corner for origin - or like Unity's top left hand corner
     
  49. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    873
    Which dlib detector to keep when importing both this package and the facemask package?

     
  50. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,249
    The OpenCV's coordinate system has the X axis horizontal and the Y axis vertical with the origin [0, 0] at the upper-left corner.
     
unityunity