[PYTHON] Create an AI that identifies Zuckerberg's face by deep learning ① (Learning data preparation)

⓪ Introduction

I've always wanted to try trendy AI and deep learning, so I decided to study from zero prerequisite knowledge of machine learning (deep learning) as a free study of GW in May 2017 from the spirit of Miha. .. Click here here for sample videos of what you actually made. The final code is here.

[Made]

Three faces of Silicon Valley mogul entrepreneur ** Mark Zuckerberg (Facebook) **, ** Bill Gates (Microsoft) **, ** Elon Musk (SpaceX, Tesla) ** AI to identify. (Currently, this has not been uploaded on the net yet. I would like to upload it when I have time * As of May 2017)

[Profile of the author (@AKIKIKIKIKIKIK)]

No prerequisite knowledge for data analysis or machine learning. Most programming is self-taught. I like to make web apps with Rails.

[References referenced in machine learning & deep learning]

I think it's bad to know anything about machine learning when doing deep learning, so first look at various information on the web to find out what you can and cannot do. (I usually read the information on the WEB that appears in "TensorFlow" and "Deep Learning". After that, I searched for "What is a neural network?")

I especially referred to it. Momokuro member face recognition by TensorFlow [Identify the anime Yuruyuri production company with TensorFlow] (http://kivantium.hateblo.jp/entry/2015/11/18/233834)

It is an article of two people. I was so helpful that I wouldn't have been able to reach the one that actually worked without this article. Thank you very much m (_ _) m

[Machine learning library to use]

There are various machine learning libraries, but this time I decided to focus on TensorFlow, Google's machine learning library. The reason is as follows.

・ Because it is a Google library, the community is thriving and it seems to be exciting in the future ・ It is said that it is also used in the topic of Go AI such as AlphaGo. ・ There are many convenient functions, so even beginners of machine learning can easily experience "machine learning". ・ The only machine learning library I knew before working on this time

I chose it for some reason. To be honest, I haven't compared it with other machine learning libraries too deeply. There may be something that is easier to use. In this article, TensorFlow setup procedure etc. are written in many other articles, so I will omit the description here.

[Determine what to do with TensorFlow]

I chose "Implementation of face recognition function in TensorFlow" which seemed to be just right difficulty because relatively various people were doing it. Past ancestors are Momokuro, Osomatsu-san / 22/161858), Yuruyuri, etc., and the target to recognize the face is your favorite character or person I decided to select "Mark Zuckerberg", the founder of Facebook, who is my favorite person. (From the viewpoint of making it easier to collect face images)

[Actually try]

The procedure I worked on is described below. In addition, this time, we proceeded with "experience deep learning" in mind rather than "understanding and mastering TensorFlow". It takes more time to master TensorFlow. .. ..

① Collection and preparation of face data for learning

(First of all, the flow until preparing this slightly creepy face data of Mr. Zuckerberg)

1. Collect image data of Mr. Zuckerberg

Using the API of Microsoft's search engine Bing, crawl the image data at the time of search with the word ** "zuckerberg" ** and save it on the PC. [The story of the transition from Bing Search API v2 to v5](http://techblog.adish.co.jp/entry/2016/12/03/Bing_Search_API_v2_%E3%81%8B%E3%82%89_v5_%E3%81 % B8% E7% A7% BB% E8% A1% 8C% E3% 81% 97% E3% 81% 9F% E8% A9% B1) (For how to use Bing's image search API, I referred to here. Thank you. This time it is basic python, but since I am good at ruby, I write it in ruby where python is not necessary. )

image_crawler.rb

require "open-uri"
require "FileUtils"
require 'net/http'
require 'json'

#Destination directory
@dirName = "./zuckerberg_image/"
#Create a storage directory(If the directory does not exist, create a new directory)
FileUtils.mkdir_p(@dirName) unless FileTest.exist?(@dirName)

#Function to save the image in the specified folder from the image URL
def save_image(url, num)
  filePath = "#{@dirName}/zuckerberg#{num.to_s}.jpg "
  open(filePath, 'wb') do |output|
    open(url) do |data|
      output.write(data.read)
    end
  end
end

#Search word
search_word = 'zuckerberg'
#Number of sheets to save(150 sheets seems to be the limit at a stretch due to API specifications)
count = 150

# Bing Search API(Use the official code as is)
# https://dev.cognitive.microsoft.com/docs/services/56b43f0ccf5ff8098cef3808/operations/571fab09dbe2d933e891028f
uri = URI('https://api.cognitive.microsoft.com/bing/v5.0/images/search')
uri.query = URI.encode_www_form({
    'q' => search_word,
    'count' => count
    # 'offset' => 150(Skip search results for the specified number)
})
request = Net::HTTP::Post.new(uri.request_uri)
request['Content-Type'] = 'multipart/form-data'
request['Ocp-Apim-Subscription-Key'] = 'Assign your API key' # Fix Me
request.body = "{body}"
response = Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
  http.request(request)
end

#Save the search image of search word as many as count
count.times do |i|
  begin
    image_url = JSON.parse(response.body)["value"][i]["thumbnailUrl"]
    save_image(image_url, i)
  rescue => e
    puts "image#{i} is error!"
    puts e
  end
end

I think that the image data is saved in the specified directory like zuckerberg1.jpg, zuckerberg2.jpg, zuckerberg3.jpg .... It looks like this. Next, only the face parts of these images are extracted and cut out.

2. Extract and cut out Mr. Zuckerberg's face

Use the image processing library OpenCV to cut out the face from the collected image data.

I used OpenCV for the first time, but the following blogs etc. were very helpful for setup and usage. Thank you m (_ _) m

How to set up the development environment of OpenCV 3 + Python 2/3 on Mac OS X Face recognition using OpenCV (Haar-like feature classifier) Basics of Python OpenCV: I will finally detect the face

I think there are many other things if you google. This time, I will omit the procedure and detailed usage of OpenCV setup. [^ 1]

(Cut out and save the face part detected by the OpenCV classifier. OpenCV convenient!)

face_detect.py

# -*- coding:utf-8 -*-

import cv2
import numpy as np

#Directory with the image data collected earlier
input_data_path = './zuckerberg_images/zuckerberg'
#Directory to save the cropped image(Please create a directory in advance)
save_path = './cutted_zuck_images/'
#OpenCV default classifier path.(https://github.com/opencv/opencv/blob/master/data/haarcascades/haarcascade_frontalface_default.Use xml file)
cascade_path = './opencv/data/haarcascades/haarcascade_frontalface_default.xml'
faceCascade = cv2.CascadeClassifier(cascade_path)

#Number of images collected(Optional change)
image_count = 300
#Number of successful face detections(Specify 0 by default)
face_detect_count = 0

#When a face is detected from the collected image data, cut it out and save it.
for i in range(image_count):
  img = cv2.imread(input_data_path + str(i) + '.jpg', cv2.IMREAD_COLOR)
  gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
  face = faceCascade.detectMultiScale(gray, 1.1, 3)
  if len(face) > 0:
    for rect in face:
      #Surround the face recognition part with a red line and save(I don't need this part now)
      # cv2.rectangle(img, tuple(rect[0:2]), tuple(rect[0:2]+rect[2:4]), (0, 0,255), thickness=1)
      # cv2.imwrite('detected.jpg', img)
      x = rect[0]
      y = rect[1]
      w = rect[2]
      h = rect[3]
      cv2.imwrite(save_path + 'cutted_zuck' + str(face_detect_count) + '.jpg', img[y:y+h, x:x+w])
      face_detect_count = face_detect_count + 1
  else:
    print 'image' + str(i) + ':NoFace'

I think that the images with the face part cut out are saved in the save destination specified by this, like cutted_zuck1.jpg, cutted_zuck2.jpg, cutted_zuck3.jpg. By the way, in the case of Mr. Zuckerberg's image, there were unexpectedly many pictures of his wife, and his face was also cut out and saved, so I removed it manually. (After that, I also excluded things that somehow recognized non-faces as faces.)

It's OK if it looks like this! A total of about 350 face data were collected. By the way, it is quite creepy when face data of the same person is collected so much I also wrote the Face recognition ancestor that I used as a reference, but when making face recognition, I definitely choose the person I like I think it's good w (By the way, OpenCV seems to be difficult to detect when the face is slanted, and there seems to be a method to detect it while rotating the image little by little, but for some reason Mr. Zuckerberg is almost beautiful, almost the face Thank you Zuckerberg!)

3. Have Bill Gates and Elon Musk also participate

Work so far and notice a fundamental mistake.

** "If only Zuckerberg's face is used, Zuckerberg's face may be identifiable, but I think there is an AI that can identify" human face "=" Zuckerberg ". In the first place, the identification that I want to do this time outputs the label that seems to correspond to the input image and classifies it (described later), but there is only one type of label and there is no identification option. Then all the identification results will be Zuckerberg ... "** (At the beginning of the move, the identification mechanism was not well understood ... orz)

So, I wanted to divide it into at least three labels, and Microsoft founder ** Bill Gates ** and SpaceX, Tesla founder, making self-driving cars and human beings I decided to have ** Mr. Ilone Mask **, who is currently active and planning to send it to Mars, participate in the war. I have become a spectacular member. (At first, I chose Steve Jobs instead of Ilone Mask, but Jobs gave up because Mr. Jobs's image is strangely black and white, so it seems to be inconsistent with other data.)

So, it's actually ** "AI that identifies the faces of Zuckerberg, Bill Gates, and Elon Musk" **.

We also collected about 350 face data of Mr. Gates and Mr. Musk in exactly the same way.

As usual, a lot of faces are scary. I can laugh a little.

4. Organize and organize facial images

As a preparation for training the data collected in TensorFlow, we first divide it into two parts: ** training data ** to train TensorFlow and ** verification data ** to check the accuracy of the model after training. First, 100 sheets were assigned to each of the verification data, and the rest was used for learning data. There are now about 300 verification data and 700 learning data.

This learning face data and verification face data are summarized in the following directory structure.

/data
  /train
    /zuckerbuerg
    /elonmusk
    /billgates
  /test
    /zuckerbuerg
    /elonmusk
    /billgates

I want to combine the paths of the face image files for learning and verification into one text file so that these organized face image data can be read at once as input data to TensowFlow. At this time, also attach a label indicating who's face each image corresponds to. This time, the labels are `` `Zuckerberg: 0 Elon Musk: 1 Bill Gates: 2```.

data.txt

./data/train/zuckerberg/cutted_zuck0.jpg 0
./data/train/zuckerberg/cutted_zuck1.jpg 0
./data/train/zuckerberg/cutted_zuck10.jpg 0
~abridgement~
./data/train/elonmusk/cutted_elon0.jpg 1
./data/train/elonmusk/cutted_elon1.jpg 1
./data/train/elonmusk/cutted_elon10.jpg 1
~abridgement~
./data/train/billgates/cutted_gates0.jpg 2
./data/train/billgates/cutted_gates10.jpg 2
./data/train/billgates/cutted_gates100.jpg 2

A simple script that outputs a text file. (I'm sorry for ruby again .. ruby will not come out anymore.)

generate_csv.rb

require 'fileutils'

train_data_path = "./data/train/data.txt"
test_data_path = "./data/test/data.txt"

FileUtils.touch(train_data_path) unless FileTest.exist?(train_data_path)
FileUtils.touch(test_data_path) unless FileTest.exist?(test_data_path)


test_zuck_data_paths = Dir.glob("./data/test/zuckerberg/*.jpg ")
test_elon_data_paths = Dir.glob("./data/test/elonmusk/*.jpg ")
test_gates_data_paths = Dir.glob("./data/test/billgates/*.jpg ")
train_zuck_data_paths = Dir.glob("./data/train/zuckerberg/*.jpg ")
train_elon_data_paths = Dir.glob("./data/train/elonmusk/*.jpg ")
train_gates_data_paths = Dir.glob("./data/train/billgates/*.jpg ")


File.open(test_data_path, "w") do |f|
  test_zuck_data_paths.each { |path| f.puts("#{path} 0") }
  test_elon_data_paths.each { |path| f.puts("#{path} 1") }
  test_gates_data_paths.each { |path| f.puts("#{path} 2") }
end
File.open(train_data_path, "w") do |f|
  train_zuck_data_paths.each { |path| f.puts("#{path} 0") }
  train_elon_data_paths.each { |path| f.puts("#{path} 1") }
  train_gates_data_paths.each { |path| f.puts("#{path} 2") }
end

At the stage of collecting data, you may have noticed that it is creepy that there are only a large number of human faces. When the image data is ready, it's finally time to start learning with TensorFlow!

Continuation and bonus

Click here for the second part of the continuation: Making AI that identifies Zuckerberg's face by deep learning ② (AI model construction)

** "Creating an AI that identifies Zuckerberg's face with deep learning" ** Part 1: Create AI to identify Zuckerberg's face by deep learning ① (Learning data preparation) Part 2: Making AI that identifies Zuckerberg's face by deep learning ② (AI model construction) Part 3: Create AI to identify Zuckerberg's face by deep learning ③ (Data learning) Part 4: Create AI to identify Zuckerberg's face by deep learning ④ (WEB construction)

GitHub：https://github.com/AkiyoshiOkano/zuckerberg-detect-ai

Bonus images that I tried with the created AI. I was surprised that it hits with more accuracy than I expected.

① If you are facing the front, it will judge even if there is a slight angle.

(2) Even if the face is unexpectedly small, it will be judged properly.

③ You can also identify Mr. Takahashi, a comedian who looks like Zuckerberg.

[^ 1]: At my time, I got the error Import Error: No module named cv2. I forgot which article I referred to, but I googled the error and solved it