[PYTHON] Machine learning library dlib
I noticed one of the machine learning C ++ Python libraries dlib. Python has a powerful library called Scikit-learn, Think of it as one of your options.
** Machine learning library dlib algorithm selection guide **
The machine learning library dlib's algorithm selection guide It was in the figure.
This is a scikit-learn guide ![Choosing the right estimator] (http://scikit-learn.org/stable/_static/ml_map.png)
** dlib article **
--SlideShare Face organ detection by 20160417dlib
-
YouTube [dlib vs OpenCV face detection] (https://www.youtube.com/watch?v=LsK0hzcEyHI)
-
YouTube Facial Landmark Detection
C++ -Face detection program that links OpenCV and dlib Face detection C ++ --Qiita Try Dlib's high-performance facial organ detection on Mac quickly Face organ detection C ++ --Qiita Try face recognition of dlib Face organ detection C ++ --Qiita Face recognition using dlib with Xcode Face organ detection C ++ -dlib guide "Python, C ++" -C ++ machine learning library "Dlib" Face organ detection C ++
The official blog describes the new Deep Learning API in version 19.0
-I tried using dlib on Raspberry Pi 3
Python -[Ubuntu] [Python] Face detection comparison between dlib and OpenCV Face detection Python
-Face recognition using dlib on the image acquired from the camera. Face detection Python
-Face detection with Python + dlib Face detection Python -Face detection and findings summary with OpenCV, dlib and OpenFace Face detection Python
-selective search using dlib Python [Ubuntu] [Python] Object tracking using dlib
-Display the result of face landmark recognition of dlib with matplotlib in real time
C# -I want to be a person inside (two-dimensional) without FaceRig![Unity x OpenCV x Dlib x Live2D] C #
-Starting face organ detection app using DlibFaceLandmarkDetector in Unity -Dlib FaceLandmark Detector #Asset Adcare that recognizes cat faces with HoloLens
- [https://enoxsoftware.com/dlibfacelandmarkdetector/] --It seems that there are human facial organ point detection and cat facial organ point detection.
SlideShare Face organ detection by 20160417dlib
Facial organ detection is used to know facial expressions, blinks, mouth movements, etc. by obtaining the positions of the eyes, nose, mouth, etc. It is known as an indispensable function when performing face recognition of a person. We are in an era where open source libraries are so rich.
On Linux, you can install it by following the steps at the following site. https://github.com/davisking/dlib
However, I think that some people want to use it on Windows, so I will describe the following procedure.
** From installation to facial organ detection (for Python) **
environment: Windows7
procedure:
- Install the dlib python binding. Command prompt > pip install dlib I was able to install it on Windows as well.
Python prompt > > > import dlib If you don't get any errors, the installation is successful.
-
Download standard sample programs etc. from github with [Download Zip] https://github.com/davisking/dlib Extract the downloaded zip file
-
- Download additional data When running face_landmark_detection.py http://sourceforge.net/projects/dclib/files/dlib/v18.10/shape_predictor_68_face_landmarks.dat.bz2 Download (there is 61.1MB even if the compressed data file is used) Extract it with 7-zip. Move the extracted files to the python_example directory.
Four. python_example script execution Command prompt > cd python_example Command prompt > python face_detector.py Image file name with face
Command Prompt > python face_landmark_detection.py shape_predictor_68_face_landmarks.dat .. \ examples \ faces
shape_predictor_68_face_landmarks.dat Since the file size of is very large, you will have to wait for a while immediately after executing the command.
- If you want to see the performance with other images, ..\examples\faces The easiest test method is to put a jpg file in.
face_landmark_detection.py https://github.com/mit-nlp/MITIE/blob/master/dlib/python_examples/face_landmark_detection.py If you look at, the processing for each face found in each image (display of the result of high-performance facial organ detection) You can see that the program is not as easy to understand.
Let's touch on other python_examples examples.
https://github.com/davisking/dlib/tree/master/python_examples
First of all, if you include the following command, in most cases it will show you the correct usage.
$ python face_alignment.py
Call this program like this:
./face_alignment.py shape_predictor_5_face_landmarks.dat ../examples/faces/bald_guys.jpg
You can download a trained facial shape predictor from:
http://dlib.net/files/shape_predictor_5_face_landmarks.dat.bz2
$
It says to get the data file, so get it with wget. Then unzip.
$ wget http://dlib.net/files/shape_predictor_5_face_landmarks.dat.bz2
--2017-11-08 00:19:30-- http://dlib.net/files/shape_predictor_5_face_landmarks.dat.bz2
Resolving dlib.net (dlib.net)... 107.180.26.78
Connecting to dlib.net (dlib.net)|107.180.26.78|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5706710 (5.4M)
Saving to: ‘shape_predictor_5_face_landmarks.dat.bz2’
shape_predictor_5_f 100%[===================>] 5.44M 164KB/s in 38s
2017-11-08 00:20:09 (145 KB/s) - ‘shape_predictor_5_face_landmarks.dat.bz2’ saved [5706710/5706710]
$
After that, run the python script with the arguments of the example.
./face_alignment.py shape_predictor_5_face_landmarks.dat ../examples/faces/bald_guys.jpg
After that, the input image may be replaced.
By doing this, you can feel free to try python_example. https://github.com/davisking/dlib/blob/master/python_examples/face_alignment.py
Below is a display of how to use such a script on the command line.
$ ./cnn_face_detector.py
Call this program like this:
./cnn_face_detector.py mmod_human_face_detector.dat ../examples/faces/*.jpg
You can get the mmod_human_face_detector.dat file from:
http://dlib.net/files/mmod_human_face_detector.dat.bz2
$ ./face_alignment.py
Call this program like this:
./face_alignment.py shape_predictor_5_face_landmarks.dat ../examples/faces/bald_guys.jpg
You can download a trained facial shape predictor from:
http://dlib.net/files/shape_predictor_5_face_landmarks.dat.bz2
$ ./face_clustering.py
Call this program like this:
./face_clustering.py shape_predictor_5_face_landmarks.dat dlib_face_recognition_resnet_model_v1.dat ../examples/faces output_folder
You can download a trained facial shape predictor and recognition model from:
http://dlib.net/files/shape_predictor_5_face_landmarks.dat.bz2
http://dlib.net/files/dlib_face_recognition_resnet_model_v1.dat.bz2
$ ./face_landmark_detection.py
Give the path to the trained shape predictor model as the first argument and then the directory containing the facial images.
For example, if you are in the python_examples folder then execute this program by running:
./face_landmark_detection.py shape_predictor_68_face_landmarks.dat ../examples/faces
You can download a trained facial shape predictor from:
http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2
face_alignment.py is Detect face Put out facial organ points Normalize the position of the image and display it in the window. When the return key is issued, the next face image is displayed.
$ ./face_alignment.py shape_predictor_5_face_landmarks.dat ../examples/faces/bald_guys.jpg
face_clustering.py Detect faces in the images in the input image folder, calculate the features of each, and Clustering based on the features Save the images in the largest cluster to the output folder.
$ ./face_clustering.py shape_predictor_5_face_landmarks.dat dlib_face_recognition_resnet_model_v1.dat ../examples/faces output_folder
Processing file: ../examples/faces/bald_guys.jpg
Number of faces detected: 24
Processing file: ../examples/faces/2009_004587.jpg
Number of faces detected: 2
Processing file: ../examples/faces/2008_002506.jpg
Number of faces detected: 3
Processing file: ../examples/faces/2008_002079.jpg
Number of faces detected: 6
Processing file: ../examples/faces/2008_004176.jpg
Number of faces detected: 7
Processing file: ../examples/faces/dogs.jpg
Number of faces detected: 0
Processing file: ../examples/faces/2008_007676.jpg
Number of faces detected: 7
Processing file: ../examples/faces/2008_002470.jpg
Number of faces detected: 6
Processing file: ../examples/faces/2008_001009.jpg
Number of faces detected: 2
Processing file: ../examples/faces/2007_007763.jpg
Number of faces detected: 7
Processing file: ../examples/faces/2008_001322.jpg
Number of faces detected: 3
Number of clusters: 45
Biggest cluster id number: 0
Number of faces in biggest cluster: 6
Indices of images in the biggest cluster: [0, 1, 5, 11, 21, 22]
Saving faces in largest cluster to output folder...
$
** dlib python API **
dlib documentation http://dlib.net/python/index.html http://dlib.net/ The dlib documentation seems to be getting better and better.
There are two ways to use the face detector.
Example of how to use the face detector 1
detector = dlib.get_frontal_face_detector()
dets = detector(img, 1)
Example of how to use the face detector 2
detector = dlib.get_frontal_face_detector()
#RGB conversion(Convert from opencv format to skimage format)
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
# frontal_face_detector class is rectangular,Score,Returns the result of the sub-detector
dets, scores, idx = detector.run(img_rgb, 0)
for det in dets:
cv2.rectangle(img, (det.left(), det.top()), (det.right(), det.bottom()), (0, 0, 255))
This dets Both cv :: Rect () are (x, y, w, h) for cv2 Note that it is different from.
dlib.rectangle(left, top, right, bottom))
It seems to convert to this data type like this.
About the minimum detection size of face detection in dlib
According to the description below, detector.run (img_rgb, numUpSampling) The minimum detection size at numUpSampling = 0 seems to be 80x80 pixel. http://dlib.net/face_detection_ex.cpp.html
If you want to detect with 40x40 pixel, set numUpSampling to 1. If you want to detect with 20x20 pixel, set numUpSampling to 2. It seems that you need to. Every time you increase numUpSampling, the area of the image is quadrupled and the processing time is quadrupled.
About face_recognition.py in dlib itself
https://github.com/davisking/dlib/blob/master/python_examples/face_recognition.py Then, it seems that the feature amount for face matching is calculated, I can't find the calculation of face matching itself.
For that reason, a library for face matching has been developed separately at the following site. Please note that the script file name may be the same. https://github.com/ageitgey/face_recognition
** How to know the API from the documentation browser **
If you don't have a Japanese commentary blog about dlib's Python implementation, don't be afraid. Start the start menu [Python (x, y)] [Documentation] [Python documentation server]. [open browser] When you execute On the browser (Python installation destination) \ site-packages If you find dlib in it, you can see all the features provided by the Python interface.
If you are using Linux instead of Python (x, y) $ /usr/lib/python2.7/pydoc.py -p port number &
Start the server as You can open it in your browser.
** How to make an object detector **
dlib provides tools for creating object detectors (Note). http://dlib.net/train_object_detector.py.html See the description in. ./train_object_detector.py ../examples/faces
There is a file named training.xml in the ../examples/faces directory It is assumed that it is placed in the following file format. When this xml file is displayed in a browser, the face frame is displayed at which position in the image. https://github.com/davisking/dlib/blob/master/examples/faces/training.xml
Images are also located in the ../examples/faces directory.
dlib.train_simple_object_detector(training_xml_path, "detector.svm", options)
In this way, the learning result file will be created as a detector.svm file.
detector = dlib.simple_object_detector("detector.svm")
dets = detector(img)
You can get a list of detected rectangles like this.
The following article explains the series of learning tasks.
-Learning an object detector in Python using dlib's Simple_Object_detector
Since dlib's object detection also samples internally, it is only necessary to collect images and rectangles that are positive data, and the accuracy is good, so it is quite good. I think he is a person who can detect objects with few parameters. Memory and CPU are stupid but charming.
Furthermore, you will want to actually make an object detector with your own image data. The procedure is described in the next article. -HOG features and object detection by SVM -Machine learning with dlib to detect objects
The trained model file is in the following location.
[dlib-models] (https://github.com/davisking/dlib-models#dlib-models)
mmod_front_and_rear_end_vehicle_detector.dat.bz2 mmod_rear_end_vehicle_detector.dat.bz2 mmod_dog_hipsterizer.dat.bz2
** Article on object detection and learning by HOG and SVM **
- https://github.com/vaaaaanquish/dlib_detection_python_script --You can train using the following script in this.
- https://github.com/vaaaaanquish/dlib_detection_python_script/blob/master/dlib_train_detector.py
Learning facial organ points
How Dlib's own facial organ learning is done is described in the comments of the script itself. http://dlib.net/face_landmark_detection.py.html
# The face detector we use is made using the classic Histogram of Oriented
# Gradients (HOG) feature combined with a linear classifier, an image pyramid,
# and sliding window detection scheme. The pose estimator was created by
# using dlib's implementation of the paper:
# One Millisecond Face Alignment with an Ensemble of Regression Trees by
# Vahid Kazemi and Josephine Sullivan, CVPR 2014
# and was trained on the iBUG 300-W face landmark dataset (see
# https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/):
# C. Sagonas, E. Antonakos, G, Tzimiropoulos, S. Zafeiriou, M. Pantic.
# 300 faces In-the-wild challenge: Database and results.
# Image and Vision Computing (IMAVIS), Special Issue on Facial Landmark Localisation "In-The-Wild". 2016.
# You can get the trained model file from:
# http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2.
# Note that the license for the iBUG 300-W dataset excludes commercial use.
# So you should contact Imperial College London to find out if it's OK for
# you to use this model file in a commercial product.
-
[One Millisecond Face Alignment with an Ensemble of Regression Trees] (https://pdfs.semanticscholar.org/d78b/6a5b0dcaa81b1faea5fb0000045a62513567.pdf)
-
Github How to train shape_predictor to obtain shape_predictor_68_face_landmarks.dat
-
I used the training data here: http://dlib.net/files/data/ibug_300W_large_face_landmark_dataset.tar.gz. It's just 300-W, but with mirrored images added in.
-
-It was confirmed that face part detection by DCNN is also effective for manga characters. - https://github.com/TadaoYamaoka/gochiusa
--There are face detection and organ point detection for cats. -Dlib FaceLandmark Detector If you combine Dlib (face organ detection) that instantly detects the shape of the face of a person or animal with OpenCV ...
** Challenges for dlib face detection **
--Profile is not detected ――Where is the profile detector? Is anyone making it?
** dlib also DNN **
-Try DNN face detection on Dlib 19.2 with webcam (in Windows)
-I tried using CNN with DLIB in Windows
** dlib-based features that can replace other libraries **
An excerpt from the script below. https://github.com/davisking/dlib/blob/master/python_examples/face_recognition.py
The following instance win and its methods can be replaced with cv2.imshow (), cv2.rectangle (), etc. There is a part of dlib that you don't have to use OpenCV.
win = dlib.image_window()
img = io.imread(f)
win.clear_overlay()
win.set_image(img)
dets = detector(img, 1)
for k, d in enumerate(dets):
shape = sp(img, d)
win.clear_overlay()
win.add_overlay(d)
win.add_overlay(shape)
dlib.hit_enter_to_continue()
** dlib itself has a tracking library **
- http://dlib.net/python/index.html#dlib.correlation_tracker
-[Ubuntu] [Python] Object tracking using dlib
- https://github.com/gdiepen/face-recognition/blob/master/track%20multiple%20faces/demo%20-%20track%20multiple%20faces.py
- Youtube person tracking with dlib correlation tracker
** dlib-based face matching is available **
Introductory article dlib-based face recognition program face_recognition
How to implement dlib Python bindings
As you can see from the source below, boost :: python is used. https://github.com/davisking/dlib/blob/master/tools/python/src/object_detection.cpp
#include <boost/python/args.hpp>
using namespace boost::python;
Why you can't ignore dlib for facial relationships
--OpenCV has a library that asks for facial landmarks, but it's not widespread yet. - https://docs.opencv.org/3.4/dc/d63/classcv_1_1face_1_1FacemarkLBF.html - Tutorials for face module Face landmark detection in an image - Facemark : Facial Landmark Detection using OpenCV --Various libraries use the result of deriving the landmark of dlib's face. ――If you look at the following page, you can see what kind of projects and papers are used. - https://sourceforge.net/p/dclib/wiki/Known_users/ --Also, it is being developed with a sponsor.
- https://github.com/davisking/dlib#dlib-sponsors
There is a dlib blog site. Here, the explanation is written about the implemented function.
http://blog.dlib.net/
** Note: **
I changed the memo of face matching of OpenCV itself written here to an independent item.
Supplementary article to "OpenCV 3.3.0 Face Recognition with OpenCV"
Investigating face orientation estimation Investigating the implementation of open source face matching
Japanese translation of Dlib README.md
Building and installing Dlib 19.15 on Ubuntu
Postscript 2020.02
At the time of writing this postscript, there were more practical options for face detection and face matching. At the time of writing this article for the first time, OpenCV's own face matching-related features were lacking in terms of being the top level of open source. This is because the library of OpenCV itself, which displays face landmarks and performs face matching, did not fully reflect the results of deep learning. However, as of 2020, the face library of OpenVino is substantial, and various Model ZOO models that have been converted by OpenVino's Model Optimizer can be used within the framework of OpenCV. For that reason, at some point, OpenVino is becoming a situation that cannot be ignored, from a situation close to dlib selection. Please look at the trends each time to see what will happen in the future. How much the oriental person's face matching ability increases depends on how much the oriental person's face is used for learning in each model. Improving the performance of open source implementations can be a countermeasure against the excessive social impact of commercial library developers.
Recommended Posts