[PYTHON] Make OpenCV object detection rotation invariant

The features of haarcascade are not invariant!

I use Haarcascade_frontalface of OpenCV for face recognition, but unlike SIFT and SURF, it doesn't seem to be a feature that is invariant to rotation, and even if I tilt the face a little, the face is not recognized.

When detecting in real time with a camera image, if you put the camera straight, it will be recognized reasonably because ordinary people point their necks straight. If you detect a photo that is tilted like this by itself, it will not be recognized properly.

In this photo, only Reni Takagi, who is facing straight ahead, has his face detected.

Solution

The idea is that if you only recognize a certain angle, you should tilt the image itself, and if you judge all the images whose tilt is changed in small steps, all the tilted faces will be detected. right.

The steps are as follows

1. Load the image
2. Prepare sky images with twice the height and width.
3. Tilt the loaded image only 5 degrees
4. Paste the tilted image in the center of the sky image
5. Judge the face
6. Repeat steps 2-5

It looks like this in the figure

Source code

This is the source code to cut out and save the face photo.

import cv2
import numpy as np
import os
from math import ceil

temp_face_img_path = 'work/temp/face/'

#Specify the xml file used for face judgment.
cascade_path =  os.path.dirname(os.path.abspath(__file__)) + "/haarcascade_frontalface_alt2.xml"

class classifyPhoto:
    def __init__(self):
        print("init")

    #Cut the face from the image, save, return the path
    def crop_face(self, img_path):
        #File name analysis
        base_name = os.path.basename(img_path)
        name,ext = os.path.splitext(base_name)
        if (ext != '.jpg') and (ext != '.jpeg') :
            print('not a jpg image')
            return

        img_src = cv2.imread(img_path, 1)
        #Convert to grayscale
        img_gray = cv2.cvtColor(img_src, cv2.COLOR_BGR2GRAY)
        cascade = cv2.CascadeClassifier(cascade_path)

        org_width = img_src.shape[1]
        org_height = img_src.shape[0]
        i = 0

        for j in range(0,71):
            #Creating a magnified image
            big_img = np.zeros((org_height * 2, org_width * 2 ,3), np.uint8)
            big_img[ceil(org_height/2.0):ceil(org_height/2.0*3.0), ceil(org_width/2.0):ceil(org_width/2.0*3.0)] = img_src

            #Center position of the image
            center = tuple(np.array([big_img.shape[1] * 0.5, big_img.shape[0] * 0.5]))

            #Get image size(side,Vertical)
            size = tuple(np.array([big_img.shape[1], big_img.shape[0]]))

            #The angle you want to rotate
            angle = 5.0 * float(j)
            #Expansion ratio
            scale = 1.0

            #Calculation of rotation transformation matrix
            rotation_matrix = cv2.getRotationMatrix2D(center, angle, scale)

            #Affine transformation
            img_rot = cv2.warpAffine(big_img, rotation_matrix, size, flags=cv2.INTER_CUBIC)
            rot_gray = cv2.cvtColor(img_rot, cv2.COLOR_BGR2GRAY)

            #Face judgment
            faces  =  cascade.detectMultiScale(img_rot, scaleFactor=1.2, minNeighbors=2, minSize=(50, 50))
            #If there is a face
            if len(faces) > 0:
                for (x,y,w,h) in faces:
                    face = img_rot[y:y+h, x:x+w]
                    file_name =  name + "_face_" + str(i) + ext
                    cv2.imwrite(temp_face_img_path + file_name, face )
                    i += 1

            else :
                print('does not have any faces')

        return 

if __name__ == '__main__':
    classifier = classifyPhoto()
    classifier.crop_faceo('image.jpg')

result

• Detects about 40 sheets in total
• 8 for Reni-chan, 9 for Arin, 1 for Momoka and Orin, 0 for Kanako, and other false positives for face parts and clothes.

Consideration

• The number of false positives increased as the number of judgments increased.
• There is a bias depending on the members. There is a possibility that not only the tilt but also the accuracy of face recognition is poor.
• Members detected in multiple sheets are likely to be combined by coordinate transformation + clustering.
• Since each calculation is performed in 5 degree increments, ** the amount of calculation is 72 times **.

So it might be better if you tune it, but it probably won't work.