I tried to make a simple face recognition Android application using OpenCV

Introduction

Face recognition technology itself has existed for a long time, but recently, smartphone apps that use face recognition such as Snow have appeared, and it is becoming more familiar. How did it happen before? I'm also interested in that, and if I take a face photo myself, I'd like to make an app that replaces the face of the person in the photo with another one! I thought. However, when I actually looked up face recognition and image analysis on the net, many of them focused on machine learning, and I was overwhelmed by the difficulty ... However, when I made a face recognition function part using OpenCV with an Android application this time, it was easier than I imagined, so I would like to introduce it. What I found by actually making it was that it was more difficult to implement the camera function than to recognize the face from the image using OpenCV.

What is OpenCV in the first place?

OpenCV (Open Source Computer Vision Library) is an open source library for processing videos and images. For OpenCV, this article was helpful.

What I wanted to do

Use OpenCV to determine if a face is present in a photo taken with the Android camera function

• This time, we will introduce only the part that recognizes the face of the Bitmap image data after shooting and acquires the result. Once you get the results (face coordinates, size), all you have to do is add another image to the original image. If you actually perform machine learning on your own and perform face recognition using OpenCV, the flow will be as follows.

1. Perform machine learning to acquire features
2. Implement the camera function in the Android app
3. Implemented face recognition function for image data taken with a camera ← Part to be introduced this time
4. Image processing and judgment using recognition results (face size, position information)

What is used to realize the face recognition function with OpenCV

OpenCV provides interfaces for C, C ++, Python and Java. When embedding in a smartphone app, there is an SDK prepared for iOS and Android, and by importing this SDK into the project, you can easily prepare an environment to use OpenCV. Also, in face recognition, the result of machine learning is required to judge "face", but there are some that OpenCV has already prepared. The result of this machine learning * is enclosed in the SDK package.

• It is enclosed in XML file format. The list of enclosed xml files is introduced in the following article.

• https://qiita.com/hitomatagi/items/04b1b26c1bc2e8081427

Usage environment / library

• Android 6.0
• OpenCV 3.2.0 --The OpenCV SDK can be downloaded from the following site. - http://opencv.org/releases.html

Process flow

Load OpenCV library

--Write the following code in the class that uses OpenCV and load the OpenCV library imported into Android Studio. --Refer to the following site for how to import to Android Studio - https://qiita.com/kodai100/items/6c9e8a34d0714913c017

FaceClassifier.java

  //Load OpenCV library
  static {
    System.loadLibrary("opencv_java3");　
  }

Convert image data to Mat format

--Convert image data to OpenCV Mat format. --In the sample, we converted from Bitmap format to Mat format.

FaceClassifier.java

    //Convert image data (Bitmap Mat file conversion)
    Mat matImg = new Mat();
    Utils.bitmapToMat(image,matImg); 

Creating a CascadeClassifier instance for face recognition

--Use "haarcascade_frontalface_alt.xml" which is a face cascade. Since the feature file is also downloaded when the SDK is downloaded, store the target file under res / raw of the Android project. ――I got stuck here for a moment ... --The CascadeClassifier constructor cannot take an InputStream as an argument, so you must take the path of the xml file as an argument. Therefore, it is necessary to follow the procedure "Read the xml file once, output it with FileOutputStream, and get the path of the output file."

FaceClassifier.java

    //Generate a cascade classifier instance that performs face recognition (write the file once and get the path of the file)
    //Get the xml file once stored under raw
    InputStream inStream = this.activity.getResources().openRawResource(R.raw.haarcascade_frontalface_alt);
    File cascadeDir = this.activity.getDir("cascade", Context.MODE_PRIVATE);
    File cascadeFile = new File(cascadeDir, "haarcascade_frontalface_alt.xml");
    //Output the acquired xml file to a specific directory
    FileOutputStream outStream = new FileOutputStream(cascadeFile);
    byte[] buf = new byte[2048];
    int rdBytes;
    while ((rdBytes = inStream.read(buf)) != -1) {
      outStream.write(buf, 0, rdBytes);
    }
    outStream.close();
    inStream.close();
    //Use the path of the output xml file as an argument of CascadeClassifier
    CascadeClassifier faceDetetcor = new CascadeClassifier(cascadeFile.getAbsolutePath());
    //When the CascadeClassifier instance is created, the output file is unnecessary, so delete it.
    if (faceDetetcor.empty()) {
      faceDetetcor = null;
    } else {
      cascadeDir.delete();
      cascadeFile.delete();
    }

Face recognition by giving image data to Cascade Classifier

--Mat type image data and MatOfRect instance are passed to CascadeClassifier.detectMultiScale method. The authentication result is stored in MatOfRect.

FaceClassifier.java

    //Face recognition by giving image data to the cascade classifier
    MatOfRect faceRects = new MatOfRect();
    faceDetetcor.detectMultiScale(matImg, faceRects);

Get the number of faces recognized in the image and the position of the face from the result stored in the MatOfRect instance.

--MatOfRect instance can be changed to Rect type. --The sample source is only log output, but if you want to replace the face in the image with another one, you can do it by overlaying the images according to the coordinates of the face and the size of the face.

FaceClassifier.java

    //Confirmation of face recognition results
    Log.i(TAG ,"Number of recognized faces:" + faceRects.toArray().length); 
    if (faceRects.toArray().length > 0) {
      for (Rect face : faceRects.toArray()) {
        Log.i(TAG ,"Vertical width of the face" + face.height);
        Log.i(TAG ,"Width of face" + face.width);
        Log.i(TAG ,"Face position (Y coordinate)" + face.y);
        Log.i(TAG ,"Face position (X coordinate)" + face.x);
      }
      return true;
    } else {
      Log.i(TAG ,"No face was detected");
      return false;
    }

Source code

FaceClassifier.java

import org.opencv.android.Utils;
import org.opencv.core.Mat;
import org.opencv.core.MatOfRect;
import org.opencv.core.Rect;
import org.opencv.objdetect.CascadeClassifier;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.InputStream;

public class FaceClassifier {
  //Load OpenCV library
  static {
    System.loadLibrary("opencv_java3");　
  }
  private Activity activity;
  public FaceClassifier (Activity activity) {
    this.activity = activity;
  }
  public boolean checkFaceExistence (Bitmap image) throws IOException {
    //Convert image data (Bitmap Mat file conversion)
    Mat matImg = new Mat();
    Utils.bitmapToMat(image,matImg); 
    //Generate a cascade classifier instance that performs face recognition (write the file once and get the path of the file)
    //Get the xml file once stored under raw
    InputStream inStream = this.activity.getResources().openRawResource(R.raw.haarcascade_frontalface_alt);
    File cascadeDir = this.activity.getDir("cascade", Context.MODE_PRIVATE);
    File cascadeFile = new File(cascadeDir, "haarcascade_frontalface_alt.xml");
    //Output the acquired xml file to a specific directory
    FileOutputStream outStream = new FileOutputStream(cascadeFile);
    byte[] buf = new byte[2048];
    int rdBytes;
    while ((rdBytes = inStream.read(buf)) != -1) {
      outStream.write(buf, 0, rdBytes);
    }
    outStream.close();
    inStream.close();
    //Use the path of the output xml file as an argument of CascadeClassifier
    CascadeClassifier faceDetetcor = new CascadeClassifier(cascadeFile.getAbsolutePath());
    //When the CascadeClassifier instance is created, the output file is unnecessary, so delete it.
    if (faceDetetcor.empty()) {
      faceDetetcor = null;
    } else {
      cascadeDir.delete();
      cascadeFile.delete();
    }
    //Face recognition by giving image data to the cascade classifier
    MatOfRect faceRects = new MatOfRect();
    faceDetetcor.detectMultiScale(matImg, faceRects);
    //Confirmation of face recognition results
    Log.i(TAG ,"Number of recognized faces:" + faceRects.toArray().length); 
    if (faceRects.toArray().length > 0) {
      for (Rect face : faceRects.toArray()) {
        Log.i(TAG ,"Vertical width of the face" + face.height);
        Log.i(TAG ,"Width of face" + face.width);
        Log.i(TAG ,"Face position (Y coordinate)" + face.y);
        Log.i(TAG ,"Face position (X coordinate)" + face.x);
      }
      return true;
    } else {
      Log.i(TAG ,"No face was detected");
      return false;
    }
  }
}

Summary

The OpenCV SDK is much easier to use, so it's easier than you think it would be. If you want to recognize something other than the face, you need to check machine learning using OpenCV, but since there are various published results (XML files) of machine learning for OpenCV, once It is recommended to check. If you want to make an app that uses face recognition technology like me, please use OpenCV.