Introduction

We have summarized the presentations of 56th Computer Vision Study Group @ Kanto held on Saturday, January 19, 2020. You can see the materials of the day from here.

The source code is available on Github below. https://github.com/33taro/gaze_cv

About the procedure for iris detection

Iris detection is a theme I was researching when I was in college, so I thought it would be easy to do it with the evolved OpenCV. The procedure is as follows.

Detection of human faces and facial landmarks from camera images
Cut out the eye area from the landmark on the face
The eye area is binarized to extract the iris area
Iris detection from the extracted iris area

虹彩検出手順.png

Detection of human faces and facial landmarks from camera images

I tried to detect the iris (black eye part) using the landmark detection of the face introduced in Another article before. Please refer to that for details.

Cut out the eye area from the landmark of the face

As introduced in the above article, the landmark of the face is [here](https://ibug.doc.ic.ac.uk/ We are using the trained model in resources / facial-point-annotations /). Therefore, the right eye is "No.37-42" and the left eye is "No.43-48".

目領域.png

This is described in "eye_region_manager.py" in the "tracking_system" directory in the actual source code.

`eye_region_manager.py`


    def detect_eye_region(self, face_landmark):
        """
Get the coordinates of the eye area from the landmark
        :param face_landmark:
        """
        #Right eye cutout
        self._right_eye_region = {'top_x': face_landmark[36][0], 'bottom_x': face_landmark[39][0],
                                  'top_y': face_landmark[37][1]
                                  if face_landmark[37][1] < face_landmark[38][1] else face_landmark[38][1],
                                  'bottom_y': face_landmark[41][1]
                                  if face_landmark[41][1] > face_landmark[40][1] else face_landmark[40][1]}

        #Left eye cutout
        self._left_eye_region = {'top_x': face_landmark[42][0], 'bottom_x': face_landmark[45][0],
                                 'top_y': face_landmark[43][1]
                                 if face_landmark[43][1] < face_landmark[45][1] else face_landmark[45][1],
                                 'bottom_y': face_landmark[47][1]
                                 if face_landmark[47][1] > face_landmark[46][1] else face_landmark[46][1]}

Since the array starts from 0, it will be 0 to 67 and the number will be shifted by one.
The vertical y-coordinates of the eye area are acquired so that they are longer.

Extract the iris area by binarizing the eye area

The P-tile method was used to binarize the eye area. This is a method to specify the percentage of the image area that you want to binarize. As a result, the iris could be acquired regardless of the brightness. (As a rule of thumb, the iris is 40% of the eye area)

目領域の2値化.png

2 Before binarization, it is smoothed with a Gaussian filter to remove noise.

Since the P tile method is not implemented in OpenCV, I made it myself, It is described in "image_utility.py" in the "utility" directory.

`image_utility.py`


# coding:utf-8

import cv2


def p_tile_threshold(img_gry, per):
    """
Binarization process by P tile method
    :param img_gry:Grayscale image to be binarized
    :param per:Percentage of images that are binarized
    :return img_thr:Binarized image
    """

    #Histogram acquisition
    img_hist = cv2.calcHist([img_gry], [0], None, [256], [0, 256])

    #Calculate the number of pixels from the ratio of the binarized object in the image
    all_pic = img_gry.shape[0] * img_gry.shape[1]
    pic_per = all_pic * per

    #Threshold calculation for binarization by P-tile method
    p_tile_thr = 0
    pic_sum = 0

    #Current brightness and sum of brightness(Add in descending order of value)Calculation
    for hist in img_hist:
        pic_sum += hist

        #Processing ends when the total brightness exceeds the specified ratio
        if pic_sum > pic_per:
            break

        p_tile_thr += 1

    #Binarization process with the threshold value obtained by the P tile method
    ret, img_thr = cv2.threshold(img_gry, p_tile_thr, 255, cv2.THRESH_BINARY)

    return img_thr

Iris detection from the extracted iris area

I was able to obtain the iris area, but I cannot extract only the iris cleanly due to the shadows of the eyebrows and eyelids. Therefore, the black area was acquired by contour point tracking, added to each area to approximate the circumscribed circle, and the circle with the largest radius was used as the iris.

However, if the circumscribed circle is too large, it will be excluded from the iris candidates.

虹彩の外接円近似.png

A series of binarization-iris detection is described in "eye_system_manager.py" in the "tracking_system" directory.

`eye_system_manager.py`


    @staticmethod
    def _detect_iris(eye_img):
        #After grayscale, smoothing with Gaussian filter
        eye_img_gry = cv2.cvtColor(eye_img, cv2.COLOR_BGR2GRAY)
        eye_img_gau = cv2.GaussianBlur(eye_img_gry, (5, 5), 0)

        #Binarization by P tile method
        eye_img_thr = p_tile_threshold(eye_img_gau, IRIS_PER)

        cv2.rectangle(eye_img_thr, (0, 0), (eye_img_thr.shape[1] - 1, eye_img_thr.shape[0] - 1), (255, 255, 255), 1)

        #Contour extraction
        contours, hierarchy = cv2.findContours(eye_img_thr, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE)

        #Find the iris from the contour by the smallest circumscribed circle
        iris = {'center': (0, 0), 'radius': 0}
        for i, cnt in enumerate(contours):
            (x, y), radius = cv2.minEnclosingCircle(cnt)
            center = (int(x), int(y))
            radius = int(radius)

            #Exclude from iris candidates if radius is too large
            if eye_img_thr.shape[0] < radius*0.8:
                # #Drawing of iris candidates
                # cv2.circle(eye_img, center, radius, (255, 0, 0))
                continue

            #The circle with the largest radius is recognized as the iris
            if iris['radius'] < radius:
                iris['center'] = center
                iris['radius'] = radius
                iris['num'] = i

        return iris

Iris detection results

As shown in the figure above, the irises of the left and right eyes are firmly acquired. However, there are still issues regarding the top and bottom. However, when a person looks up and down, he often moves his face rather than moving his iris. (Actually looking up and down with just your eyes makes you tired) So I'm wondering if I can somehow detect the orientation of the face from the landmarks on the face.

Reference link

Contour point tracking

** OpenCV-Python Tutorial ~ Outline: First Step ~ ** http://labs.eecs.tottori-u.ac.jp/sd/Member/oyamada/OpenCV/html/py_tutorials/py_imgproc/py_contours/py_contour_features/py_contour_features.html OpenCV tutorial. I used it as a reference for basic usage.
** OpenCV --How to extract contours with findContours ** https://www.pynote.info/entry/opencv-findcontours Details of openCV contour point tracking function findContours () are introduced. I used it as a reference when writing the source code.
** [OpenCV; Python] Summary of findcontours function ** https://qiita.com/anyamaru/items/fd3d894966a98098376c Each function of OpenCV contour point tracking function findContours () is explained in an easy-to-understand manner using figures. I used it as a reference for understanding this technology.

Minimum circumscribed circle

** OpenCV-Python Tutorial-Features of Area (Outline)-**
http://labs.eecs.tottori-u.ac.jp/sd/Member/oyamada/OpenCV/html/py_tutorials/py_imgproc/py_contours/py_contour_features/py_contour_features.html
OpenCV tutorial. I used it as a reference for basic usage.

P tile method

** Imaging Solution-P Tile Method-** https://imagingsolution.net/imaging/p-tile-method/ It explains the P tile method in an easy-to-understand manner. I used it as a reference for understanding and implementing the P-tile method.
** Introduction to image processing / C language sample collection-P tile method-** http://pgsample.info/image/other/PercentileMethod.html It is an implementation of the P tile method in C language. This time I used it as a reference when writing the code for the P tile method in python.

[PYTHON] I tried to detect the iris from the camera image