[PYTHON] Kaggle Facial Keypoints Detection

Overview of the competition

In this competition, for each of the 7049 face image data, the x and y coordinates of the 15 parts that characterize the face are given as training data. (Example: x coordinate of the center of the left eye, y coordinate of the center of the right eye, etc.) It learns that data and gives the coordinates of the parts that characterize the face to the image of the test set.

I will start immediately.

Execution code


import pandas as pd
import numpy as np
import zipfile
import tensorflow as tf

Data given

Training data

Unzip and open the zip file.

Execution code


with zipfile.ZipFile('training.zip') as existing_zip:
    with existing_zip.open('training.csv') as traincsv:
        train=pd.read_csv(traincsv)
train
left_eye_center_x left_eye_center_y right_eye_center_x right_eye_center_y ... mouth_center_top_lip_y mouth_center_bottom_lip_x mouth_center_bottom_lip_y Image
0 6.033564 39.002274 30.227008 36.421678 ... 72.935459 43.130707 84.485774 238 236 237 238 240 240 239 241 241 243 240 23...
1 64.332936 34.970077 29.949277 33.448715 ... 70.266553 45.467915 85.480170 219 215 204 196 204 211 212 200 180 168 178 19...
... ... ... ... ... ... ... ... ... ...
7047 70.965082 39.853666 30.543285 40.772339 ... NaN 50.065186 79.586447 254 254 254 254 254 238 193 145 121 118 119 10...
7048 66.938311 43.424510 31.096059 39.528604 ... NaN 45.900480 82.773096 53 62 67 76 86 91 97 105 105 106 107 108 112 1...

It is the data of 7049 × 31. The last column of the data is the image data, and the other columns are the x and y coordinates that indicate the part with the face. There also seems to be missing values. How many are there?

Execution code


train.isnull().sum()

output


left_eye_center_x              10
left_eye_center_y              10
right_eye_center_x             13
right_eye_center_y             13
left_eye_inner_corner_x      4778
left_eye_inner_corner_y      4778
left_eye_outer_corner_x      4782
left_eye_outer_corner_y      4782
right_eye_inner_corner_x     4781
right_eye_inner_corner_y     4781
right_eye_outer_corner_x     4781
right_eye_outer_corner_y     4781
left_eyebrow_inner_end_x     4779
left_eyebrow_inner_end_y     4779
left_eyebrow_outer_end_x     4824
left_eyebrow_outer_end_y     4824
right_eyebrow_inner_end_x    4779
right_eyebrow_inner_end_y    4779
right_eyebrow_outer_end_x    4813
right_eyebrow_outer_end_y    4813
nose_tip_x                      0
nose_tip_y                      0
mouth_left_corner_x          4780
mouth_left_corner_y          4780
mouth_right_corner_x         4779
mouth_right_corner_y         4779
mouth_center_top_lip_x       4774
mouth_center_top_lip_y       4774
mouth_center_bottom_lip_x      33
mouth_center_bottom_lip_y      33
Image                           0

There is a mess ...

test data

Also unzip the zip file and open it.

Execution code


with zipfile.ZipFile('test.zip') as existing_zip:
    with existing_zip.open('test.csv') as testcsv:
        test=pd.read_csv(testcsv)
test
ImageId Image
0 1 182 183 182 182 180 180 176 169 156 137 124 10...
1 2 76 87 81 72 65 59 64 76 69 42 31 38 49 58 58 4...
2 3 177 176 174 170 169 169 168 166 166 166 161 14...
3 4 176 174 174 175 174 174 176 176 175 171 165 15...
... ... ...
1780 1781 28 28 29 30 31 32 33 34 39 44 46 46 49 54 61 7...
1781 1782 104 95 71 57 46 52 65 70 70 67 76 72 69 69 72 ...
1782 1783 63 61 64 66 66 64 65 70 69 70 77 83 63 34 22 2...

This is 1783 image data.

File format to submit

Two other CSV files are given, but here the format of the file to be submitted is shown.

Execution code


df=pd.read_csv("IdLookupTable.csv")
sample=pd.read_csv("SampleSubmission.csv")

Execution code


df
RowId ImageId FeatureName Location
0 1 1 left_eye_center_x NaN
1 2 1 left_eye_center_y NaN
2 3 1 right_eye_center_x NaN
3 4 1 right_eye_center_y NaN
4 5 1 left_eye_inner_corner_x NaN
... ... ... ... ...
27119 27120 1783 right_eye_center_y NaN
27120 27121 1783 nose_tip_x NaN
27121 27122 1783 nose_tip_y NaN
27122 27123 1783 mouth_center_bottom_lip_x NaN
27123 27124 1783 mouth_center_bottom_lip_y NaN

IdLookupTable.csv shows the coordinates you want to present for each of the 1783 images. The tricky part of this competition is that the training data gives the coordinates of 30 facial parts, but you don't necessarily have to predict all the coordinates for the test data image.

Specifically, check the number of each coordinate included in the submitted file.

Execution code


df["FeatureName"].value_counts()

output


nose_tip_y                   1783
nose_tip_x                   1783
left_eye_center_x            1782
right_eye_center_y           1782
left_eye_center_y            1782
right_eye_center_x           1782
mouth_center_bottom_lip_x    1778
mouth_center_bottom_lip_y    1778
mouth_left_corner_x           590
mouth_center_top_lip_y        590
mouth_left_corner_y           590
mouth_center_top_lip_x        590
left_eye_outer_corner_x       589
left_eye_outer_corner_y       589
right_eye_inner_corner_y      589
right_eye_inner_corner_x      589
left_eye_inner_corner_y       588
right_eye_outer_corner_x      588
right_eye_outer_corner_y      588
left_eye_inner_corner_x       588
mouth_right_corner_x          587
mouth_right_corner_y          587
left_eyebrow_inner_end_x      585
right_eyebrow_inner_end_x     585
left_eyebrow_inner_end_y      585
right_eyebrow_inner_end_y     585
left_eyebrow_outer_end_x      574
left_eyebrow_outer_end_y      574
right_eyebrow_outer_end_x     572
right_eyebrow_outer_end_y     572

In this way, the required coordinates vary irregularly.

Next is a sample submission file.

Execution code


sample
RowId Location
0 1 0
1 2 0
2 3 0
3 4 0
4 5 0
... ... ...
27119 27120 0
27120 27121 0
27121 27122 0
27122 27123 0
27123 27124 0

The data to be submitted is simply the line number and the 27124 coordinate values.

Molding of training data

Needless to say, this is the main part of the competition. First, extract only the image data from train.csv and test.csv and make it into a numpy array. When converting the extracted image data into a numpy array, it is recommended to prepare an empty array and substitute it one by one.

Execution code


x_train=np.empty((7049,96,96,1))
x_test=np.empty((1783,96,96,1))

Execution code


for i in range(7049):
    train0=train["Image"][i].split(" ")     #Extract the i-th image array separated by spaces
    train1=[int(x) for x in train0]         #Convert each to int type and store in list
    train2=np.array(train1,dtype="float")   #Convert list to numpy array
    train3=train2.reshape(96,96,1)          #Mold the array into 96x96x1
    x_train[i]=train3                       #Assign as the i-th element of an empty numpy array

for i in range(1783):
    test0=test["Image"][i].split(" ")
    test1=[int(x) for x in test0]
    test2=np.array(test1,dtype="float")
    test3=test2.reshape(96,96,1)    
    x_test[i]=test3

Execution code


x_train =x_train / 255
x_test = x_test /255

Let y_train be the one with the Image column removed from train.csv.

Execution code


y_train=train.drop(['Image'],axis=1)

Handle the missing values you saw earlier. For the time being, apply a ffill that complements with the value in the previous line.

Execution code


y_train.fillna(method = 'ffill',inplace = True)

Check the missing value again.

Execution code


y_train.isnull().sum()

output


left_eye_center_x            0
left_eye_center_y            0
right_eye_center_x           0
right_eye_center_y           0
left_eye_inner_corner_x      0
left_eye_inner_corner_y      0
left_eye_outer_corner_x      0
left_eye_outer_corner_y      0
right_eye_inner_corner_x     0
right_eye_inner_corner_y     0
right_eye_outer_corner_x     0
right_eye_outer_corner_y     0
left_eyebrow_inner_end_x     0
left_eyebrow_inner_end_y     0
left_eyebrow_outer_end_x     0
left_eyebrow_outer_end_y     0
right_eyebrow_inner_end_x    0
right_eyebrow_inner_end_y    0
right_eyebrow_outer_end_x    0
right_eyebrow_outer_end_y    0
nose_tip_x                   0
nose_tip_y                   0
mouth_left_corner_x          0
mouth_left_corner_y          0
mouth_right_corner_x         0
mouth_right_corner_y         0
mouth_center_top_lip_x       0
mouth_center_top_lip_y       0
mouth_center_bottom_lip_x    0
mouth_center_bottom_lip_y    0
Image                        0

It was confirmed that there were no missing values.

Finally, standardize the data.

Execution code


for columns in train_y.columns:
    mean.append(train_y[columns].mean())
    std.append(train_y[columns].std())
    train_y[columns] = (train_y[columns] - train_y[columns].mean()) / train_y[columns].std()

Learning

Build the model.

Execution code


model=tf.keras.models.Sequential([
    tf.keras.layers.Conv2D(6,(3,3), activation = 'relu', input_shape=(96,96,1)),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Conv2D(12,(3,3), activation = 'relu'),
    tf.keras.layers.MaxPooling2D(2,2),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(512, activation = 'relu'),
    tf.keras.layers.Dense(30,activation='relu')
    ])
model.summary()

output


Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_2 (Conv2D)            (None, 94, 94, 6)         60        
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 47, 47, 6)         0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 45, 45, 12)        660       
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 22, 22, 12)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 5808)              0         
_________________________________________________________________
dense_2 (Dense)              (None, 512)               2974208   
_________________________________________________________________
dense_3 (Dense)              (None, 30)                15390     
=================================================================
Total params: 2,990,318
Trainable params: 2,990,318
Non-trainable params: 0
_________________________________________________________________

Execution code


model.compile(optimizer='adam', loss='mean_squared_error', metrics=['accuracy'])

Let me learn.

Execution code


model.fit(x_train, y_train, epochs = 10)

output


Train on 7049 samples
Epoch 1/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 470.8903 - accuracy: 0.5540
Epoch 2/10
7049/7049 [==============================] - 19s 3ms/sample - loss: 283.7981 - accuracy: 0.6073
Epoch 3/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 152.0215 - accuracy: 0.6493
Epoch 4/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 151.0923 - accuracy: 0.6866
Epoch 5/10
7049/7049 [==============================] - 19s 3ms/sample - loss: 150.6215 - accuracy: 0.7188
Epoch 6/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 149.9657 - accuracy: 0.7289
Epoch 7/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 149.8715 - accuracy: 0.7371
Epoch 8/10
7049/7049 [==============================] - 19s 3ms/sample - loss: 149.7018 - accuracy: 0.7424
Epoch 9/10
7049/7049 [==============================] - 20s 3ms/sample - loss: 149.4364 - accuracy: 0.7451
Epoch 10/10
7049/7049 [==============================] - 19s 3ms/sample - loss: 149.3164 - accuracy: 0.7510

Make predictions for the test model.

Execution code


pred = model.predict(x_test)

Creating a submission file

It's a little annoying from here, but you have to refer to the IdLookupTable.csv mentioned above and extract the coordinates required for each image. Not surprisingly, the pred consists of predictions of the x and y coordinates of 15 features for each of the 1783 images. (1783 × 30)

As an approach,

  1. Get the features to extract for each image from IdLookupTable.csv
  2. Next, encode the features with indexes from 0 to 29, respectively.
  3. Extract the value corresponding to the combination of image ID and feature index from pred It is a flow.

Execution code


#1. IdLookupTable.Extract ImageId and feature name from csv.
lookid_list = list(df['FeatureName'])
imageID = list(df['ImageId']-1)
pre_list = list(pred)

Execution code


#2.Feature name 0~Encode to 29 indexes and store in the list.
feature = []
for f in looked_list:
    feature.append(lookid_list.index(f))

Execution code


#3.The one corresponding to the combination of the image ID and the feature name index is extracted from the prediction result.
#At that time, the value is obtained by performing the reverse conversion of standardization.
preded = []
for x,y in zip(imageID,feature):
    preded.append(pre_list[x][y] * std[y] + mean[y])

Finally, create a submission file.

Execution code


rowid = pd.Series(df['RowId'],name = 'RowId')
loc = pd.Series(preded,name = 'Location')

Execution code


submission = pd.concat([rowid,loc],axis = 1)

Summary

that's all. This will give you a score of around 3.8. スクリーンショット 2020-01-20 3.13.07.png If you apply Augmentation to the image or improve the model, the score will increase at all. I would like to post again as soon as I work on it. I would appreciate it if you could point out any mistakes. Thank you very much.

Recommended Posts

Kaggle Facial Keypoints Detection
Facial expression detection using Yolov5
[Ubuntu] [Python] Facial organ detection using dlib