[PYTHON] I examined Yuki Kashiwagi's facial features to understand TensorFlow [Part 2]

I examined Yuki Kashiwagi's facial features to understand TensorFlow and CNN (Convolutional Neural Network). This time, I will explain and summarize the whole process focusing on the judgment part of TensorFlow in the whole process. In the judgment, the trained model is used to judge whether the individual image is Yuki Kashiwagi. The premise of this article is Part 1 and Part 2.

Determination process

As I wrote in Part 1, image judgment continues with face detection using OpenCV + cropping and image judgment using TensorFlow as shown in the figure below. doing. The learned parameters saved in the process of "3. Learning" are used when judging the image.

Judgment execution

Ready to run

The folder structure looks like this, and put the learning model used for face detection used in OpenCV in the "models" folder. The acquisition source is Github. For the basics of OpenCV, please refer to the article "[Explanation for beginners] OpenCV face detection mechanism and practice (detectMultiScale)". Then, put the image file to be judged in the "images" folder. Multiple files can be placed, but only the image file that succeeded in face detection first is judged, and the remaining image files are not judged. Images other than the "images" folder can also be loaded by specifying parameters at runtime.

models
│  haarcascade_frontalface_default.xml
│  haarcascade_frontalface_alt.xml
│  haarcascade_frontalface_alt2.xml
│  haarcascade_frontalface_alt_tree.xml
│  haarcascade_profileface.xml
images
│ Image file to be judged

Learning execution

python judge_image.py

Execute with the above command, and if face detection is successful, the face image detected in the pop-up will be displayed. Anything is fine, so press the key to execute the judgment.

The following are provided as run-time parameters. Implemented using the library argparse, which is explained a little in the article "Specify parameters for face detection in openCV to quickly improve detection accuracy" doing.

Python program

Judge part (judge_image.py)

[Model part (model_deep.py)](http://qiita.com/FukuharaYohei/items/1192ca038b0052ef300f#%E3%83%A2%E3%83%87%E3%83%AB%E9%83%A8%E5 % 88% 86model_deeppy) is called to judge the image.

import cv2, argparse, os
import numpy as np
import tensorflow as tf
import model_deep  #Estimated part of TensorFlow

#Location of trained image discriminative model
CKPT_PATH = '/tmp/tensorflow/kashiwagi/logs/'

#Basic model parameters
FLAGS = None

#Identification label and the name corresponding to each label number
HUMAN_NAMES = { 0: u'Kashiwagi', 1: u'others',}

#Face detection trained model type
CASCADE = ['default','alt','alt2','tree','profile']

#Acquisition of features of classifier
def get_cascade():
    #Classifier directory(Obtained from)
    # https://github.com/opencv/opencv/blob/master/data/haarcascades/

    #Trained model file
    if   FLAGS.cascade == CASCADE[0]:#'default':
        cascade_path = './models/haarcascade_frontalface_default.xml'
    elif FLAGS.cascade == CASCADE[1]:#'alt':
        cascade_path = './models/haarcascade_frontalface_alt.xml'
    elif FLAGS.cascade == CASCADE[2]:#'alt2':
        cascade_path = './models/haarcascade_frontalface_alt2.xml'
    elif FLAGS.cascade == CASCADE[3]:#'tree':
        cascade_path = './models/haarcascade_frontalface_alt_tree.xml'
    elif FLAGS.cascade == CASCADE[4]:#'profile':
        cascade_path = './models/haarcascade_profileface.xml'

    #Acquire features of cascade classifier
    return(cv2.CascadeClassifier(cascade_path))

#Face judgment
def run_judge(image):
    
    #Outputs and returns the probability of each label for the input image
    logits = model_deep.inference(image, 1.0, FLAGS.image_size, FLAGS.pool_size)
    
    #Softmax calculation
    loss   = tf.nn.softmax(logits)

    #Output the contents so far to TensorBoard
    summary = tf.summary.merge_all()

    #Trained model call preparation
    saver = tf.train.Saver()
    
    #Creating a Session(TensorFlow calculations must be done in an absolute Session)
    with tf.Session() as sess:
    
        #Preparing to write to TensorBoard
        summary_writer = tf.summary.FileWriter(FLAGS.log_dir, sess.graph)

        #Variable initialization(Run before loading trained model)
        sess.run(tf.global_variables_initializer())
        
        #Acquiring a trained model
        ckpt = tf.train.get_checkpoint_state(CKPT_PATH)

        #Read if there is a trained model
        if ckpt:
            saver.restore(sess, ckpt.model_checkpoint_path)

        #Run the session and get the TensorBoard summary and error
        summary_str, loss_value = sess.run([summary, loss])

        #Add summary to TensorBoard and close
        summary_writer.add_summary(summary_str)
        summary_writer.close()

        #judgment result
        result = loss_value[0]
        
        humans = []
        #Create a hash of name and probability
        for index, rate in enumerate(result):
            humans.append({
                'name': HUMAN_NAMES[index],
                'rate': rate * 100
            })
        #Sort by probability in descending order and output the most probable answer
        rank = sorted(humans, key=lambda x: x['rate'], reverse=True)        
        print('Probalibity %d %% :This image is %s' % (rank[0]['rate'], rank[0]['name']))

#Face detection and judgment using OpenCV
def read_and_edit_images(file_name, faceCascade):

    #Image file reading
    img = cv2.imread(FLAGS.input_dir + file_name)

    #Since there are files that rarely fail when there are a large number of images, log output and skip(Cause unknown)
    if img is None:
        print(file_name + ':Cannot read image file')
        return 0

    #Face detection
    face = faceCascade.detectMultiScale(cv2.cvtColor(img, cv2.COLOR_BGR2GRAY), 
                                        scaleFactor=FLAGS.scale, minNeighbors=FLAGS.neighbors, minSize=(FLAGS.min, FLAGS.min))

    #If it cannot be detected, log output and exit
    if len(face) <= 0:
        print(file_name + ':No Face')
    else:
        
        #Cut out the face part and resize the image using only the first face detected
        cut_image = cv2.resize(img[face[0,1]:face[0,1] + face[0,3], face[0,0]:face[0,0] + face[0,2]], (FLAGS.image_size, FLAGS.image_size))
        
        #Show the image of the face cut in the pop-up(Press the key to close the popup and proceed to the next process)
        cv2.imshow('image', cut_image)
        cv2.waitKey(0)
        cv2.destroyAllWindows()

        image = []

        #Convert to TensorFlow format(Float 0-Range of 1)
        image.append(cut_image.astype(np.float32)/255.0)
        
        #Return as an array
        return np.asarray(image)

        
def cut_and_judge_images():
    
    #Acquisition of features of classifier
    faceCascade = get_cascade()

    #Store files in folders in variables(Also stores directories)
    files =  os.listdir(FLAGS.input_dir)

    #When a face is detected from the collected image data, cut it out and save it.
    for file_name in files:

        #For files(If not a directory)
        if os.path.isfile(FLAGS.input_dir + file_name):
            
            #Image reading and face cropping
            image = read_and_edit_images(file_name, faceCascade)
            
            if image is not None:
                #Image judgment by passing to TensorFlow
                run_judge(image)
                
                #If you judge even once, it ends
                return 0.
            
            
#Pass if run directly(Imported and does not pass at runtime)
if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument(
        '--cascade',
        type=str,
        default='alt',
        choices=CASCADE,
        help='cascade file.'
  )
    parser.add_argument(
        '--scale',
        type=float,
        default=1.3,
        help='scaleFactor value of detectMultiScale.'
  )
    parser.add_argument(
        '--neighbors',
        type=int,
        default=2,
        help='minNeighbors value of detectMultiScale.'
  )
    parser.add_argument(
        '--min',
        type=int,
        default=50,
        help='minSize value of detectMultiScale.'
  )
    parser.add_argument(
        '--input_dir',
        type=str,
        default='./images/',
        help='The path of input directory.'
  )
    parser.add_argument(
        '--image_size',
        type=int,
        default=81,
        help='Input image size'
    )
    parser.add_argument(
        '--pool_size',
        type=int,
        default=3,
        help='Pooling size'
    )
    parser.add_argument(
        '--log_dir',
        type=str,
        default='/tmp/tensorflow/kashiwagi_judges',
        help='Directory to put the log data.'
    )

    #Parameter acquisition and execution
    FLAGS, unparsed = parser.parse_known_args()
    
    #If the TensorBoard save directory exists, delete it and recreate it.
    if tf.gfile.Exists(FLAGS.log_dir):
        tf.gfile.DeleteRecursively(FLAGS.log_dir)
    tf.gfile.MakeDirs(FLAGS.log_dir)

    #Face detection and judgment
    cut_and_judge_images()

Overall improvement

Someday I want to overcome the following improvements and revenge: lifter_tone1: When will it be?

Lack of training data

We believe that the small amount of training data was a major factor in reducing accuracy. As you can see in the article "Calling Bing Image Search API v5 from Python to collect images", I'm using Bing, but Google and others Was it good? Other than me [Phenomenon that images cannot be acquired very much with Bing](http://qiita.com/ysdyt/items/49e99416079546b65dfc#%E7%B5%90%E6%9E%9C%E3%81%A8%E6% 89% 80% E6% 84% 9F) was happening. There was also an increase in Image padding.

CNN learning model

This time, I used the same learning model as Deep MNIST for Experts, but Image Recognition You should refer to / tutorials / image_recognition).

Finally

Even beginners of Python and Deep Learning, let alone TensorFlow, could do this quality! TensorFlow is great. It took a while, but it was fun. "I made a Dir en gray face classifier using TensorFlow" and ["Making an AI that identifies Zuckerberg's face by deep learning ]](Http://qiita.com/AkiyoshiOkano/items/72f3e4ba9caf514460ee) was very helpful.