Face recognition app with OpenCV + Android Studio

Thing you want to do

Creating a face recognition app using OpenCV ·Target ・ For the time being, learn how to install and use Android Studio (beginner) ・ Run OpenCV on Android ・ Both images taken with the camera and images saved in the gallery can be used.

reference

I tried to make a simple face recognition Android application using OpenCV https://qiita.com/Kuroakira/items/094ecb236da89949d702 Supplement on how to call the camera with an intent (mainly for Xperia 2.1 problem) https://gabu.hatenablog.com/entry/20101125/1290681748

What I used

Android Studio 3.5.3 OpenCV 4.2.0

Rough flow

As for the implementation, I will write it briefly because I refer to the above predecessors.

Press the camera and gallery buttons ↓ Get images from camera, gallery ↓ Convert the acquired image from bitmap to Mat format ↓ Pass Mat format image to OpenCV ↓ Acquires face recognition information and displays it on the screen

result

Coordinate information like that came out.

result1.png

Things that can be used in the future

·camera If you want to acquire the image taken by the camera in high resolution, write it to the URL once and acquire the image from the URL. getImg = (Bitmap) data.getExtras (). get ("data"); can only get low resolution images for thumbnails. (The image for image recognition is obtained by getImg = (Bitmap) data.getExtras (). Get ("data") ;, but it seems to be inconvenient if the number of people increases or if it is a drawing image)

MainActivity.java


   getImg = (Bitmap) data.getExtras().get("data");

MainActivity.java


    /**Abbreviation**/
    imageView.setImageURI(mImageUri);
    /**Abbreviation**/

    /**
     *For photography
     *Export the taken photo to URL once, and store the URL of the export destination in mImageUri
     */
    protected void takePhoto(){
        String filename = System.currentTimeMillis() + ".jpg ";

        ContentValues values = new ContentValues();
        values.put(MediaStore.Images.Media.TITLE,filename);
        values.put(MediaStore.Images.Media.MIME_TYPE, "image/jpeg");
        mImageUri = getContentResolver().insert(
                MediaStore.Images.Media.EXTERNAL_CONTENT_URI, values
        );

        Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
        intent.putExtra(MediaStore.EXTRA_OUTPUT, mImageUri);
        startActivityForResult(intent, RESULT_CAMERA);
    }

What I want to achieve in the future

・ Class division only for face recognition Initially, I tried to implement it by classifying it, but I gave up because I did not know how to classify the following parts. (Can't access res / raw with the same description as MainActivity.java from another class?)

MainActivity.java


        File cascadeDir = getDir("cscade", Context.MODE_PRIVATE);
        File cascadeFile = new File(cascadeDir, "lbpcascade_frontalface.xml");

・ Image acquisition from ImageView By acquiring images from ImageVIew, in the case of cameras and galleries, implementation in each case becomes unnecessary, so it may be possible to reduce the code.

・ Implementation in Kotlin I implemented it in Java, which I was familiar with, because I wanted to get used to it for the time being, but I would like to use Kotlin when developing for Android in the future.

Source code

MainActivity.java


package com.example.myapplication;

import androidx.appcompat.app.AppCompatActivity;

import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import  java.io.InputStream;

import android.content.ContentValues;
import android.content.Context;
import android.content.Intent;
import android.graphics.Bitmap;
import android.graphics.Canvas;
import android.graphics.BitmapFactory;
import android.graphics.Color;
import android.graphics.Paint;
import android.net.Uri;
import android.os.Bundle;
import android.provider.MediaStore;
import android.util.Log;
import android.view.View;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.TextView;
import android.widget.Toast;

import org.opencv.android.Utils;
import org.opencv.core.Mat;
import org.opencv.core.MatOfRect;
import org.opencv.core.Rect;
import org.opencv.objdetect.CascadeClassifier;

public class MainActivity extends AppCompatActivity {
    private final static int RESULT_CAMERA = 1001; //For camera
    private final static int REQUEST_GALLERY = 1000; //For gallery
    private Uri mImageUri; //Instance variable to store the URL of the image
    private Bitmap getImg = null; //Images from the camera or gallery
    private ImageView imageView;//Image view declaration

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);

        imageView = findViewById(R.id.image_view);//First, associate ImageView with the layout view ID

        Button cameraButton = findViewById(R.id.camera_button);
        cameraButton.setOnClickListener(new View.OnClickListener() {//Implementation using a normal inner class
            @Override
            public void onClick(View v) {
                takePhoto();
            }
        });

        Button galleyButton = findViewById(R.id.gallery_button);
        galleyButton.setOnClickListener(new View.OnClickListener() {//Implementation using a normal inner class
            @Override
            public void onClick(View v) {
                Intent intent = new Intent();
                intent.setType("image/*");
                intent.setAction(Intent.ACTION_GET_CONTENT);
                startActivityForResult(intent, REQUEST_GALLERY);
            }
        });
    }

    //I will paste the photo taken in ImageView from now on.
    @Override
    protected void onActivityResult(int requestCode, int resultCode, Intent data) {
        if(requestCode == RESULT_CAMERA || requestCode == REQUEST_GALLERY) {
            TextView textView = findViewById(R.id.text1); //Text view(View to store the text to display)
            String outputText = ""; //Text to display
            MatOfRect faceRects; //Store face recognition data

            //When selecting an image from the camera
            if (requestCode == RESULT_CAMERA) {
                //Including canceled cases
                if (data.getExtras() == null) {
                    Log.d("debug", "cancel ?");
                    return;
                } else {
                    imageView.setImageURI(mImageUri);
                    getImg = (Bitmap) data.getExtras().get("data");
                }
            }
            //When selecting an image from the gallery
            if (requestCode == REQUEST_GALLERY && resultCode == RESULT_OK) {
                try {
                    InputStream in = getContentResolver().openInputStream(data.getData());
                    getImg = BitmapFactory.decodeStream(in);
                    in.close();
                    //Display the selected image
                    imageView.setImageBitmap(getImg);
                } catch (Exception e) {
                    //When there is no file
                    textView.setText("File not found");
                }
            }
            //Image pasted on imageView(Obtained from camera, gallery)Face recognition from
            try {
                faceRects = checkFaceExistence(getImg);
                outputText = makeOutputText(faceRects);\
                //Display the selected image
                imageView.setImageBitmap(getImg);
            } catch (IOException e) {
                outputText = "Face recognition error";
                e.printStackTrace();
            }

            Toast.makeText(this,outputText,Toast.LENGTH_LONG).show();
            textView.setText(outputText);
        }
    }

    /**
     *For photography
     *Export the taken photo to URL once, and store the URL of the export destination in mImageUri
     */
    protected void takePhoto(){
        String filename = System.currentTimeMillis() + ".jpg ";

        ContentValues values = new ContentValues();
        values.put(MediaStore.Images.Media.TITLE,filename);
        values.put(MediaStore.Images.Media.MIME_TYPE, "image/jpeg");
        mImageUri = getContentResolver().insert(
                MediaStore.Images.Media.EXTERNAL_CONTENT_URI, values
        );

        Intent intent = new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
        intent.putExtra(MediaStore.EXTRA_OUTPUT, mImageUri);
        startActivityForResult(intent, RESULT_CAMERA);
    }

    /**
     *Perform face recognition based on the passed image
     * @param bitmap Image for face recognition
     * @return text Face recognition information
     * @throws IOException
     */
    protected MatOfRect checkFaceExistence(Bitmap bitmap) throws IOException{
        System.loadLibrary("opencv_java4");

        //Image data sent(bitmap)To Mat format
        Mat matImg = new Mat();
        Utils.bitmapToMat(bitmap, matImg);
        String text =  "";   //Text that stores face recognition information

        //Generating a cascade classifier instance for face recognition(Export the file once and get the file path)
        //Get the xml file once stored under raw
        InputStream inStream = getResources().openRawResource(R.raw.haarcascade_frontalface_alt);
        MatOfRect faceRects = new MatOfRect(); //Store face recognition data

        try {
            //Set the output xml file path as an argument of CascadeClassfle
            CascadeClassifier faceDetetcor = outputCascadeFile(inStream);
            //Face recognition by giving image data to the cascade classifier
            faceDetetcor.detectMultiScale(matImg, faceRects);
        }
        catch (IOException e){
            Toast.makeText(this,"Failed to open the analysis information file.",Toast.LENGTH_SHORT).show();
        }

        return faceRects;
    }

    /**
     *Import the openCV classifier prepared in advance once and make it available for export.
     * @param inStream Original data of classifier
     * @return faceDetector classifier data
     * @throws IOException
     */
    protected CascadeClassifier outputCascadeFile(InputStream inStream) throws IOException {
        File cascadeDir = getDir("cscade", Context.MODE_PRIVATE);
        File cascadeFile = new File(cascadeDir, "lbpcascade_frontalface.xml");

        //Output the acquired xml file to a specific directory
        FileOutputStream outputStream = new FileOutputStream(cascadeFile);

        byte[] buf = new byte[2048];
        int rdBytes;

        while ((rdBytes = inStream.read(buf)) != -1) {
            try {
                outputStream.write(buf, 0, rdBytes);
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

        //Set the output xml file path as an argument of CascadeClassfle
        CascadeClassifier faceDetetcor = new CascadeClassifier((cascadeFile.getAbsolutePath()));

        outputStream.close();

        return faceDetetcor;
    }

    /**
     *Text creation for output
     *Set the coordinate information to text based on the face recognition information
     * @param faceRects
     * @return
     */
    protected String makeOutputText(MatOfRect faceRects){
        String text = "";

        //Return face recognition result as String
        if(faceRects.toArray().length > 0){
            for(Rect face: faceRects.toArray()) {
                text = "Vertical width of the face:" + face.height + "\n" +
                        "Width of face" + face.width + "\n" +
                        "Face position(Y coordinate)" + face.y + "\n" +
                        "Face position(X coordinate)" + face.x;
            }
        }
        else{
            text = "No face was detected.";
        }

        return text;
    }

Recommended Posts

Face recognition app with OpenCV + Android Studio
Face recognition app made with Amazon Rekognition and Java
How to make an app using Tensorflow with Android Studio
Import device images with Android app
Problems with android studio development series
USB serial communication with Android app (usb-serial-for-android)
I made a rock-paper-scissors app with android
Riot (chat app) development (settings) in Android Studio
Android weather app
[Android] Despaired story with App UserId and SharedUserId
Vibrate the wristband device with Bluetooth from the Android app
Compatible with Android 10 (API 29)
Android app Spectrum Analyzer
Notes in Android studio
Defeating Android Studio Part 3-6
Defeating Android Studio Part 1 & Part 2
About Android App Components
How to take a screenshot with the Android Studio emulator
Use Java included with Android Studio to build React Native