[RELEASED] Dlib FaceLandmark Detector

Is there an example for getting Natcam and OpenCV for Unity working with the Dlib AR sample? .

That would be great. I wonder if that could improve the webcam resolution and speed....

 

When i replace video dance.avi by my video ... The system not work !
How i can fix this ?

 

https://github.com/opencv/opencv_extra/blob/master/testdata/cv/video/768x576.avi
I converted this video file to mjpeg codec video file using ffmpeg.

ffmpeg -i 768x576.avi -r 10 -s 768x576 -vcodec mjpeg -an 768x576_mjpeg.avi

 

Please replace the WebCamTextureToMatHelper of ARHeadWebCamTextureExample with NatCamPreviewToMatHelper.

 

Hi, there is any way to use several fhog object detectors in same instance (

dlib.fhog_object_detector.run_multiple

), like in this example https://github.com/davisking/dlib/blob/master/python_examples/train_object_detector.py#L134 ?

 

Unfortunately, the methods that simultaneously run multiple detectors are not implemented in this asset.

 

It will be implemented in a new update? It will be very useful to me. Or how much do you want to implement this new feature?

 

For now, I do not plan to implement that method in DlibFaceLandmarkDetector.

 

I apologize for the inconvenience.
OpenCV for Unity v2.3.3 is in pending review status.
The latest version will be published to Asset Store within a few days.

You can get an older version of DlibFaceLandmarkDetectorWithOpenCVExample from the github repository. Older version code corresponds to OpenCVForUnity v2.3.2.
https://github.com/EnoxSoftware/Dli...ets/DlibFaceLandmarkDetectorWithOpenCVExample

 

Until OpenCVforUntiy v2.3.3 is published in AssetStore it is possible to download older version of Example from DL link.
OpenCV for Unity v2.3.3 is in pending review status.
The latest version will be published to Asset Store within a few days.

DlibFaceLandmarkDetectorWithOpenCVExample
https://www.dropbox.com/s/0g8xwtxvw...rkDetectorWithOpenCVExample.unitypackage?dl=0

CVVTuberExample1.0.1
https://www.dropbox.com/s/tccnr216bwq1xiq/CVVTuberExample1.0.1.unitypackage?dl=0

DlibFaceLandmarkDetectorPlayMakerActions1.0.2
https://www.dropbox.com/s/jkwsk6c2p...tectorPlayMakerActions1.0.2.unitypackage?dl=0

OpenCVForUnityPlayMakerActions1.0.5
https://www.dropbox.com/s/x4xo907pnda7chh/OpenCVForUnityPlayMakerActions1.0.5.unitypackage?dl=0

FaceMaskExample1.0.6
https://www.dropbox.com/s/aly2gtf5vmji6x6/FaceMaskExample1.0.6.unitypackage?dl=0

FaceSwapperExample1.0.7
https://www.dropbox.com/s/6gwa0v8eunqjnv5/FaceSwapperExample1.0.7.unitypackage?dl=0

FaceTrackerExample1.1.8
https://www.dropbox.com/s/u2cx1nlkddz8lnn/FaceTrackerExample1.1.8.unitypackage?dl=0

MarkerBasedARExample1.2.1
https://www.dropbox.com/s/u2rfto39ohu18ja/MarkerBasedARExample1.2.1.unitypackage?dl=0

MarkerLessARExample1.0.4
https://www.dropbox.com/s/rpovz53y3hhhlgj/MarkerLessARExample1.0.4.unitypackage?dl=0

RealTimeFaceRecognitionExample1.0.4
https://www.dropbox.com/s/rjot3qaih580xyp/RealTimeFaceRecognitionExample1.0.4.unitypackage?dl=0

 

Can anyone verify that the frame rate optimizer example for the Dlib FaceLandmark Detector works on IOS? I would like to know before I buy it. I am skeptical because the openCV examples don't work on my android phone (crashes). Thank you!

 

I apologize for the inconvenience.
OpenCV for Unity v2.3.3 is in pending review status.
I asked Unity Support about the review. I got a reply that the Asset Review Team is now in the holiday season so it will take some more time to finish the review.

Until OpenCVForUnity 2.3.3 is published in AssetStore, you can download examples of older versions from the following DL link.


DlibFaceLandmarkDetector_v1.2.5_to_v1.2.4_Patch.unitypackage
https://www.dropbox.com/s/s4ggi4ps4...ctor_v1.2.5_to_v1.2.4_Patch.unitypackage?dl=0
By importing this unitypackage you can downgrade DlibFaceLandmarkDetector v1.2.5 to DlibFaceLandmarkDetector v1.2.4. Please import DlibFaceLandmarkDetector v1.2.5 from AssetStore and then import this patch. FaceSwapperExample 1.0.7, FaceMaskExample 1.0.6 and CVVTuberExample 1.0.1 require DlibFaceLandmarkDetector 1.2.4.


CVVTuberExample1.0.1
https://www.dropbox.com/s/tccnr216bwq1xiq/CVVTuberExample1.0.1.unitypackage?dl=0

DlibFaceLandmarkDetectorPlayMakerActions1.0.2
https://www.dropbox.com/s/jkwsk6c2p...tectorPlayMakerActions1.0.2.unitypackage?dl=0

OpenCVForUnityPlayMakerActions1.0.5
https://www.dropbox.com/s/x4xo907pnda7chh/OpenCVForUnityPlayMakerActions1.0.5.unitypackage?dl=0

FaceMaskExample1.0.6
https://www.dropbox.com/s/aly2gtf5vmji6x6/FaceMaskExample1.0.6.unitypackage?dl=0

FaceSwapperExample1.0.7
https://www.dropbox.com/s/6gwa0v8eunqjnv5/FaceSwapperExample1.0.7.unitypackage?dl=0

FaceTrackerExample1.1.8
https://www.dropbox.com/s/u2cx1nlkddz8lnn/FaceTrackerExample1.1.8.unitypackage?dl=0

MarkerBasedARExample1.2.1
https://www.dropbox.com/s/u2rfto39ohu18ja/MarkerBasedARExample1.2.1.unitypackage?dl=0

MarkerLessARExample1.0.4
https://www.dropbox.com/s/rpovz53y3hhhlgj/MarkerLessARExample1.0.4.unitypackage?dl=0

RealTimeFaceRecognitionExample1.0.4
https://www.dropbox.com/s/rjot3qaih580xyp/RealTimeFaceRecognitionExample1.0.4.unitypackage?dl=0

 

I'm having an issue with DlibFaceLandMark Detector running the ARHead example in the editor, but not when compiled (with both scenes selected in build settings). It loads the scene, the camera starts, but then shuts off, and I never do see a video feed. The ARHead example compiles fine when it is the only scene in the build.

 

Could you tell me the environment you tried?
DlibFaceLandmarkDetector version :
OpenCVForUnity version :
Unity version :

 

I've noticed that the WebcamTextureExample only gets about 5 fps on my machine (a newer Macbook Pro), whether I run it in the editor or not. Is there any way to speed this up (other than to run it on a more powerful machine). 5fps is almost unusable.

I've noticed if I stop rendering the quad (that renders the webcam image), it jumps up to 80fps.

 

The resolution of the image that can be acquired by Unity's WebCamTexture seems to differ depending on the WebCam you use. Perhaps the image resolution that can be acquired with webcams of macbook pro is the maximum resolution only.

Also,
Using ImageOptimizationHelper.cs it is possible to scale down images easily. ImageOptimizationHelper.cs is used in FrameOptimizationExample.
https://github.com/EnoxSoftware/Dli...timizationExample/FrameOptimizationExample.cs

 

Hello! I bought OpenCV for Unity and DlibFaceLandmarkDetector and everything is working fine.

However, I'd like your help regarding a problem I'm facing: using "WebCamTextureFaceMaskAdditionalExample" scene, how can I increase the size of the FaceMask to cover more of a face? I mean, is it possible to make it cover the ears and also more of the chin and forehead (with the forehead I was able to by slightly changing "AddForeheadPoints()" methods and increasing its points values)?
Let me know if i wasn't clear enough, thanks in advance!

 

If you use your own extended face landmark points, you need to create corresponding mesh data.
For details on how to create new mesh data using scripts, please refer to the "ExtendForehead()" method.

 

Hi, @EnoxSoftware,

I have bought your OpenCVForUnity and DlibFaceLandmarkDetector packages. Everything works fine, but in my project I need some more detalisation in lipsync. I have tried to train my own shape predictors on publically available datasets (like Helen), but your sp_human_face_68.dat looks much better in most cases.

Can you please tell which dataset you used to train your sp_human_face_68 shape predictor?
Which parameters did you use?

 

Since the training dataset consists of flickr CC0 licensed images, the Shape Predictor model files are available for commercial use. For details of the training data set, please refer to "DlibFaceLandmarkDetector/DlibFaceLandmarkDetectorTrainingDataset.txt".

 

@EnoxSoftware
I have just bought OpenCV and Dlib Facelandmark detector.

'ARHeadWebCamTextureExample' is not included in the package, despite that scene being used in the asset store thumbnail. Also, the test scene has a button to take me there that is simply broken. Where is the scene?

I only need simple face tracking, and to be able to overlay a 2D image onto that face in real time.

 

In the example combining OpenCVForUnity and DlibFaceLandmarkDetector, you need to import additional packages. Please see ReadMe.pdf for details.

• Import and Setup the “OpenCV for Unity”.
  
• Import the "Assets/DlibFaceLandmarkDetector/ DlibFaceLandmarkDetectorWithOpenCVExample.unitypackage".
• Add all of the “***.unity” in the “DlibFaceLandmarkDetectorWithOpenCVExample” folder to [Build Settings] – [Scene In Build].

 

Purchased the assets(OpenCV for Unity,Dlib face landmark detector) to create facemask and final build size is 170+ Mb and performence in devices like nokia 6.1+ is 10-15 fps.
is there a way to reduce the size and improve the performence(Removed unwanted examples also).

 

Hi! Thanks for creating this! I am having a lot of fun with the Mask example. My question is: can I get rid of the small MaskFace picture that shows next to my face? Trying to make a simple Makeup game and I would love to get rid of that image while trying different makeup products. I realized the FaceMaks image thumbnail is rendered in the Canvas with the phone camera image.

Thank you!
Screensho with my coments in red so I can make myself understood better.

 

The example does not provide the option to hide the mask preview.
Could you comment out lines 530 to 561 of the code (WebCamTextureFaceMaskExample.cs)?
https://github.com/EnoxSoftware/Fac...ple/WebCamTextureFaceMaskExample.cs#L530-L561

 

hi
is there any way to draw landmarks for foot detection?

 

In order to detect the foot, it is necessary to load a model for foot detection.

 

@EnoxSoftware Can this detect and track a patch on the forehead? (and how?)

 

i guess its not worthy for mobile apps becuause the size of APK is minimum 170 MB which is not worthy for me at all.if it still works for you than use it. And as much i have read this plugin there is nothing to detect just forehead.

 

Sorry for my late reply.
It is possible to reduce the app size by using model file for mobile apps ( sp_human_face_68_for_mobile.dat or sp_human_face_17_for_mobile.dat).
Unfortunately, the model file included with this asset does not contain the forehead points.

 

How to Cover All Faces with Masks?

 

Could you try "Assets/FaceMaskExample/WebCamTextureFaceMaskAdditionalExample" ?
WebCamTextureFaceMaskAdditionalExample is an example of adding forehead points.

 

Hey, amazing asset!
Cannot get it to run on webgl though.
I'm getting a missing function: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6resizeEjc exception on the build.
Also, I had to modify the example code for the facemask detection since this line:
DlibFaceLandmarkDetector.UnityUtils.Utils.getFilePath ("sp_human_face_68.dat");
was returning an empty string.

I have both OpenCV and DLib installed, the latest releases on the store,
I have done no configuration, I'm expecting it to work out of the box.

Here is another report of the same problem:
https://forum.unity.com/threads/webgl-development-build-always-fails-2019-1-8f1.700754/

 

Could you tell me the environment you tried?
Unity version :

 

It was Unity 2019.2.0f1

 

If properly "ImportSettings" of "opencvforunity.bc" is not set, an emcc error will occur.
By selecting MenuItem[Set Plugin Import Settings], ”opencvforunity.bc” corresponding to version of Unity editor is automatically set.

 

I have tried that without success:

missing function: _ZNSt3__212basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEE6resizeEjc
printErr @ UnityLoader.js:4

 

I somehow managed to solve the issue with the build. Now the camera doesn't activate. The permission is never requested.

http://ideas.farm/TestWebGLDLib/

 

Is there any planning to support gpu acceleration for dlib ?

 

Hi, I'm having an issue where the webgl build takes too long to start the camera. Any ideia what could be causing it? It seems like it's downloading something before the initialization.

 

It seems that GPU is not used in dlib object detector and shape predictor.
https://github.com/davisking/dlib/issues/1250

 

The Utils.getFilePathAsync() method takes a long time to download sp_human_face_68.dat ( 98MB ) .
https://github.com/EnoxSoftware/Dli...amTextureExample/WebCamTextureExample.cs#L149

 

is there any way to mitigate the download time? Other than switching to sp_human_face_68_for_mobile.

 

Hello! Do you have an example of ArHead WebCam with NatCam Texture?

 

Please replace the WebCamTextureToMatHelper of ARHeadWebCamTextureExample with NatCamPreviewToMatHelper.

 

I mean do you have an apk or something? I want to see fps with it..

 

Can I create a transparent facemash behind which I will hide 3D objects(that is, if I open my mouth, the object will be visible inside it, when closed-not) Tried to make it with ARHeadWebCamTextureExample, apply a transparent shader to it, but how can I do same thing with face mask?

 

You can get example using the previous version NatCam here.
https://github.com/EnoxSoftware/NatCamWithOpenCVForUnityExample/releases

An example of the latest version will be created from now.

 

Hi! is there any way to align the face when is tilted?

I have found some examples in phyton but I can´t find anything for Unity OpenCV

Thanks!

https://www.pyimagesearch.com/2017/05/22/face-alignment-with-opencv-and-python/

 

I don't have an example of Face Alignment with OpenCVForUnity and DlibFaceLandmarkDetector. However, since getRotationMatrix2D method and warpAffine () method is implemented, I think it is possible to implement Face Alignment using OpenCVForUnity.

getRotationMatrix2D
https://enoxsoftware.github.io/Open...mgproc.html#ab9b8b89051891d37da62abf9bd0dfb5a

warpAffine()
https://enoxsoftware.github.io/Open...mgproc.html#a74017885add09e19513333b797acc29b