[PYTHON] I made an app to find out who the members of the Straw Hat Pirates look like

I made an app to find out who the members of the Straw Hat Pirates look like

This time, I made an application that uses image recognition of machine learning. As the title suggests, you can find out how much your face resembles who is a member of the One Piece Straw Hat Pirates.

Click here for git hub ↓↓↓ https://github.com/Sho-cmyk/shiny-winner

First, look at the completed form. When you press the shooting button, It shows how much your face looks like to any member of the Straw Hat Pirates. The top two similar characters are displayed. I found that it was 38% similar to Jinbei and 20% similar to Luffy.

If you are interested in the app, please try it from this URL. ↓↓↓ https://one-piece-camera.herokuapp.com/

Steps to make an app

Next, I will introduce the procedure for making an application. The procedure is roughly divided as follows.

0. File structure (flask)

1. 1. Collect images for learning

2. Image preprocessing

3. 3. Learning

4. Estimate

5. Arrange the output screen with html / css / js

6. File structure in Flask

flask_app
├ one-piece_cnn_aug.h5 ---Save weight
├ static　---Static file location
│ ├ image
│ └ webcam.js
│ └ stylesheet.css
├ templates　---template
│ └ index.html
│ └ photoclick.html
│ └ photoclick_copy.html
├ main.py ---Application execution script (application entrance)
├ requirements.txt 
└ runtime.txt
└ Profile
└ README.md

1. 1. Collect images for learning I use Yahoo image search to scrape images of each member of the Straw Hat Pirates. I scraped 200 sheets each. For the detailed scraping method, refer to the following. Quote: https://www.youtube.com/watch?v=BrCaYYJtC4w

2. Image preprocessing I will leave only the images that can be used next. I will erase the photos that the target character has not moved by itself.

The image is now divided into training data and test data. Training data is inflated using the Image Data Generator.

generatedata.py

from PIL import Image
import os,glob
import numpy as np
from sklearn.model_selection import train_test_split
from keras.preprocessing.image import ImageDataGenerator

classes=["Usopp","Sanji","Jinbei","Zoro","chopper","Nami","Bibi","Frankie","Brook","Luffy","Robin"]
num_classes=len(classes)
image_size=50
num_testdata=30

#Loading images
X_train=[]
X_test=[]
Y_train=[]
Y_test=[]

for index,classlabel in enumerate(classes):
    photos_dir="./"+classlabel
    files=glob.glob(photos_dir+"/*.jpg ")
    for i,file in enumerate(files):
        if i >=200:break
        image= Image.open(file)
        image=image.convert("RGB")
        image=image.resize((image_size,image_size))
        data=np.asarray(image)
        X_train.append(data) #Add to the end of the list
        Y_train.append(index) #Add label
#Standardization
datagen=ImageDataGenerator(samplewise_center=True,samplewise_std_normalization=True)


g=datagen.flow(X_train,Y_train,shuffle=False)
X_batch,y_batch=g.next()

#0 while maintaining the ratio of rgb values-Close to 255 for easy viewing
X_batch *=127.0/max(abs(X_batch.min()),X_batch.max())
X_batch +=127.0
X_batch =X_batch.astype('unit8')

#Put it in an array and divide it into test and training data
X_train=np.array(X_train)
X_test=np.array(X_test)
y_train=np.array(Y_train)
y_test=np.array(Y_test)

#X_train,X_test,y_train,y_test=train_test_split(X,Y)
xy=(X_train,X_test,y_train,y_test)
np.save("./one-piece_aug.npy",xy)#Save code

3. 3. Learning The learning model uses CNN. The code is below.

one-piece_cnn_aug.py

from keras.models import Sequential
from keras.layers import Conv2D,MaxPooling2D
from keras.layers import Activation,Dropout,Flatten,Dense
from keras.utils import np_utils
import keras
import numpy as np
from tensorflow.keras.optimizers import RMSprop

classes=["Usopp","Sanji","Jinbei","Zoro","chopper","Nami","Bibi","Frankie","Brook","Luffy","Robin"]
num_classes=len(classes)
image_size=50
#Define the main function that performs the training
def main():
    X_train,X_test,y_train,y_test=np.load("./one-piece.npy",allow_pickle=True) #Read data from a text file
    X_train=X_train.astype("float")/256 #Normalizes and converges RGB values to 0 to 1 to make neural networks easier to learn
    X_test=X_test.astype("float")/256 #Also
    y_train=np_utils.to_categorical(y_train,num_classes) #one-hot-vector:The correct answer value is 1, and the others are 0. The first argument is the label you want to convert to a vector, and the second argument is the label you can convert to a vector.
    y_test=np_utils.to_categorical(y_test,num_classes) #Also

    model=model_train(X_train,y_train) #Do learning
    model_eval(model,X_test,y_test) #Make an evaluation

#Define a model for CNN
def model_train(X,y):
    model=Sequential() #Create a Sequential model that connects each layer of the neural network in order
    #Take out features
    model.add(Conv2D(32,(3,3),padding='same',input_shape=X.shape[1:])) #32 layers of convolution layer 3x3
    model.add(Activation('relu')) #Activation function relu,Negative → 0, positive → as is
    model.add(Conv2D(32,(3,3))) #2nd convolution layer
    model.add(Activation('relu')) #Activation function relu
    model.add(MaxPooling2D(pool_size=(2,2))) #The pooling layer makes the features stand out and takes out
    model.add(Dropout(0.25)) #Discard 25% of data to reduce data bias

    model.add(Conv2D(64,(3,3),padding='same')) #Next is 64 kernel layers
    model.add(Activation('relu')) #relu activation function
    model.add(Conv2D(64,(3,3))) #Also a convolution layer
    model.add(Activation('relu')) #relu activation function
    model.add(MaxPooling2D(pool_size=(2,2))) #Max pooling

    #Fully join. Classify based on characteristics
    model.add(Flatten()) #First, arrange the data in a row
    model.add(Dense(512)) #Fully join
    model.add(Activation('relu')) #relu activation function
    model.add(Dropout(0.5)) #Discard half
    model.add(Dense(11)) #11 output layer nodes
    model.add(Activation('softmax')) #Use a softmax function that sums up to 1

    opt=RMSprop(lr=0.0001,decay=1e-6) #Definition of optimization method. Update algorithm during training.

    model.compile(loss='categorical_crossentropy',
                    optimizer=opt,metrics=['accuracy']) #Evaluation function. loss:The error between the correct answer and the estimated value in the loss function. opt:Optimization method. metrics:Evaluation value

    model.fit(X,y,batch_size=32,epochs=100) #Parameter optimization. batch_size:The number of data used at one time. epoch: Number of learnings.
#Save model
    model.save('./one-piece_cnn.h5')

    return model

#Make the model usable
def model_eval(model,X,y):
    scores=model.evaluate(X,y,verbose=1) #verbose:Show progress,Put the learning result in score
    print('Test Loss:',scores[0]) #loss
    print('Test Accuracy:',scores[1]) #Accuracy

#Allow this program to be called by other files
if __name__=="__main__":
    main()

The learning results are as follows.

Epoch 100/100
7888/7888 [==============================] - 64s 8ms/step - loss: 0.0200 - acc: 0.9962
330/330 [==============================] - 1s 4ms/step

4. Estimate

main.py

import PIL
import io
import base64
import re
from io import StringIO
from PIL import Image
import os
from flask import Flask, request, redirect, url_for, render_template, flash,jsonify
from werkzeug.utils import secure_filename
from keras.models import Sequential, load_model
from keras.preprocessing import image
import tensorflow as tf
import numpy as np
from datetime import datetime

classes=["Usopp","Sanji","Jinbei","Zoro","chopper","Nami","Bibi","Frankie","Brook","Luffy","Robin"]
classes_img=["static/img/usoppu.jpg ","static/img/sanji.jpg ","static/img/jinbe.jpg ","static/img/zoro.jpg ","static/img/chopper.jpg ","static/img/nami.jpg ","static/img/bibi.jpg ","static/img/franky.jpg ","static/img/bruck.jpg ","static/img/rufi.jpg ","static/img/robin.jpg "]
num_classes=len(classes)
image_size=50

UPLOAD_FOLDER = "./static/image/"
ALLOWED_EXTENSIONS = set(['png', 'jpg', 'jpeg', 'gif'])

app = Flask(__name__)

def allowed_file(filename):
    return '.' in filename and filename.rsplit('.', 1)[1].lower() in ALLOWED_EXTENSIONS

model = load_model('./one-piece_cnn_aug.h5')#Load the trained model

graph = tf.get_default_graph()


app = Flask(__name__)

@app.route('/', methods=['GET', 'POST'])
def upload_file():

    global graph
    with graph.as_default():
        if request.method == 'POST':

                myfile= request.form['snapShot'].split(',')
                imgdata = base64.b64decode(myfile[1])
                image = Image.open(io.BytesIO(imgdata))

                #Save
                basename = datetime.now().strftime("%Y%m%d-%H%M%S")
                image.save(os.path.join(UPLOAD_FOLDER, basename+".png "))
                filepath = os.path.join(UPLOAD_FOLDER, basename+".png ")


                image=image.convert('RGB')
                image=image.resize((image_size,image_size))
                data=np.asarray(image)
                X=[]
                X.append(data)
                X=np.array(X).astype('float32')
                X /=256

                result=model.predict([X])[0]
                result_proba = model.predict_proba(X, verbose=1)[0]
                    
                percentage=(result_proba*100).astype(float)
                array_sort =sorted(list(zip(percentage,classes,classes_img)),reverse=True)
                percentage,array_class,array_img = zip(*array_sort)
                
                
                pred_answer1="label:"+str(array_class[0])+ ",probability:"+str(percentage[0])+"%"
                pred_answer2="label:"+str(array_class[1])+ ",probability:"+str(percentage[1])+"%"
                pred_answer3="label:"+str(array_class[2])+ ",probability:"+str(percentage[2])+"%"
                
                img_src1=array_img[0]
                img_src2=array_img[1]
                img_src3=array_img[2]


                basename = datetime.now().strftime("%Y%m%d-%H%M%S")
                filepath3 = UPLOAD_FOLDER + basename+".png "


                return render_template("index.html",answer1=pred_answer1,img_data1=img_src1,answer2=pred_answer2,img_data2=img_src2,answer3=pred_answer3,img_data3=img_src3)

        return render_template("index.html",answer="")

        

if __name__ == '__main__':
   app.run(debug = True)

Five. Arrange the output screen All you have to do is arrange the output screen with html / css / js.