[PYTHON] About the shortest path to create an image recognition model by machine learning and implement an Android application

Purpose of this article

--Share how to easily create an image recognition model and easily run it as an app

Execution environment

--google colaboratory (runtime: GPU) (tensorflow 2.0) (google Chrome)

What to write in this article

-[Dataset creation method for image recognition model](#Dataset creation method for image recognition model) -[How to create an image recognition model](#How to create an image recognition model) -[How to load and move the created image recognition model in Android Studio](#How to load and move the created image recognition model in android-studio) -[My failure story](# My failure story)

How to create a dataset for an image recognition model

Take the following steps

  1. Create multiple classes you want to recognize. (Here, classes A, B, C)
  2. Collect images of class A, B, C
  3. Separate class A, B, and C images for training and testing

Collect images of classes A, B, C

Skip if you have already collected images If you haven't collected it yet, google_image_download (reference URL) is convenient. It downloads images by specifying keywords, extensions, sizes, number of sheets, etc.

Separate class A, B, C images for training and testing

Divide the images into good feelings for training and testing (80% training: about 20% test?) Specifically, create a folder with the following structure images ├train │└A │ └a01.jpg │ └aslfdjk.png │ ... │└B │ └ba.jpg │ └dskrup.png │ ... │└C │ └ba.jpg │ └sdddrrd.png │ ... │ ├validation │└A │ └fwwqd.jpg │ └qiita.png │ ... │└B │ └sddd.jpg │ └reag.png │ ... │└C │ └vtet.jpg │ └fhyr.png │ ...

How to create an image recognition model

Create the model in the following steps

  1. Use tensorflow tutorial
    1. Replace the data for training the model with your own
  2. Train the model by slightly modifying the shape of the model in consideration of the implementation in Android Studio.
  3. How to save the trained model

Use tensorflow transfer learning tutorial

Focusing on simplicity, the code that diverted the tutorial provided by tensorflow is [(reference URL) * google colaboratory opens suddenly when you press the URL](https://colab.research.google.com/drive/ 1p-MY8C_H238TWffWrRjC8cc20alZACSD) For how to use google colaboratory, refer to Official HP.

Replace the data that trains the model with your own

This part of the source code

from google.colab import drive
drive.mount('/content/drive')
PATH = '/content/drive/'+'Path to where your dataset is on google drive'

If you enter this, do you want to allow access to google drive? You will be asked to certify that. I personally don't care about it, so I authenticated without thinking about it, but if you care about it, you should stop loading your own dataset with google colaboraotry.

Train the model with minor modifications to the model shape for implementation in Android Studio

This part of the source code

After that, you can train the model by executing the following code cells in sequence.


history = model.fit_generator(
    train_data_gen,
    steps_per_epoch=total_train // batch_size,
    epochs=epochs,
    validation_data=val_data_gen,
    validation_steps=total_val // batch_size
)

How to save a trained model

only this

saved_model_dir = base_dir+'Where you want to save'
tf.saved_model.save(model, saved_model_dir)

How to load and move the created image recognition model in Android Studio

Take the following steps

  1. Convert the saved model to a TensorFlow Lite model and save it
  2. Create an Android studio project
  3. Load the model saved in Android studio and run the app

Convert the saved model to a TensorFlow Lite model and save

only this

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()

with open(base_dir+'Model name.tflite', 'wb') as f:
  f.write(tflite_model)

Download the saved model and save it locally.

Create an Android studio project

First, install Android studio (reference URL), As usual, the quick start provided by tensorflow is used. Reference URL If you follow the steps, a folder named example will be created in the local environment, so open the android project in the following path. \examples\lite\examples\image_classification\android

Load the model saved in Android studio and run the app

Follow the steps below

  1. Place the locally saved .tflite file in the asset folder
  2. Place the .txt file containing the class in the asset folder
  3. Partially rewrite the .ja file

Place the locally saved .tflite file in the asset folder

Place it in the following folder. \examples\lite\examples\image_classification\android\app\src\main\assets There are some .tflite in the same place.

Put the .txt file with the class in the asset folder

In this case, write classes A, B, and C as follows and save them in the same location as the .tflite file.

mylabel.text


A
B
C

Partially rewrite the .java file

First, \examples\lite\examples\image_classification\android\app\src\main\java\org\tensorflow\lite\examples\classification\tflite\ClassifierFloatMobileNet.java 56th line

return "mobilenet_v1_1.0_224.tflite";

Is rewritten as follows

return "Model name.tflite";

next, \examples\lite\examples\image_classification\android\app\src\main\java\org\tensorflow\lite\examples\classification\tflite\Classifier.java 110th line

  public static Classifier create(Activity activity, Model model, Device device, int numThreads)
      throws IOException {
    if (model == Model.QUANTIZED) {
      return new ClassifierQuantizedMobileNet(activity, device, numThreads);
    } else {
      return new ClassifierFloatMobileNet(activity, device, numThreads);
    }
  }

Is rewritten as follows

  public static Classifier create(Activity activity, Model model, Device device, int numThreads)
      throws IOException {
    //if (model == Model.QUANTIZED) {
      //return new ClassifierQuantizedMobileNet(activity, device, numThreads);
    //} else {
      return new ClassifierFloatMobileNet(activity, device, numThreads);
    //}
  }

Finally, run a make project in Android Studio. After that, connect the smartphone to the PC and execute Run'app'. Then, an application named TFL Classify will be installed and executed on your smartphone. Confidence of A, B, and C (indicating how likely the class is in%) is displayed at the bottom of the screen.

Writer's failure story

I will share some of the failure (?) Stories that the author stumbled upon that may be helpful.

Failure (?) 1

I tried to convert the pytorch model to the tensorflow LITE model and ended up frustrated. I mainly referred to these two websites, URL1, [URL2](https://heartbeat.fritz.ai/deploying-pytorch-and -keras-models-to-android-with-tensorflow-mobile-a16a1fb83f2) Both of them gave me an unclear error and stopped. When I think about it now, it may have been caused by the existence of both keras and tf.keras. Shall we verify it next time?

This time, I wanted to move my own model with the android application as soon as possible, so I ended up creating a model using the tensorflow tutorial and converted the keras model to the tensorflow LITE model as described above.

As described in [URL1] above (https://qiita.com/lain21/items/9f9f9707ebad4bbc627d)

Meanwhile, TensorFlow's clear advantages over PyTorch include:

Can learn using TPU Rich APIs that facilitate front-end deployment, such as TFLite and TensorFlow.js

It seems that tensorflow is currently more supportive for facilitating conversion to apps.

Failure (?) 2

While collecting various codes, keras and tf.keras were mixed. From the middle, I rewrote it to tf.keras as a man who absolutely erases keras.

Failure (?) 3

I don't notice that the output of the model is one in the default tutorial code, and there is not enough output in android studio! I made an error saying. I thought that the cause of the error was to create a .tflite file from keras using TFLiteConverter, so I spent a lot of time there ... After all, at this URL, while trying to evaluate the self-made model, it became "something wrong with the output shape?" I noticed that the shape of

Failure (?) 4

I tried to quantize the model, but I gave up because it could not be read by android studio. After all, I used the float model. I tried it because the size of the model would be smaller and the processing would be faster if it was quantized, but I couldn't. The cause of this is still unknown.

Failure (?) 5

At first, I hadn't decided what to use for object recognition, so I was thinking of using yolo on darknet, but I didn't know how to use my own model, so I gave up because I didn't know what to do. From an acquaintance, pytorch is easy! It led to failure 1. Don't give up as soon as you think this way. .. ..

Failure (?) 6

You can easily collect images using google_image_download with [Collect images of classes A, B, C](# Collect images of classes A, B, C)! I wrote that, but this is quite difficult. Of course, inappropriate images are also downloaded, so it is difficult to throw them away or crop them, and then the work of classifying is waiting. I understand why annotation is a job in the world, but I don't want to do much because real MP is likely to decrease. .. .. There is an active learning as a related word, and it seems to be a method of automatically selecting an image that increases the recognition rate. It would be convenient if possible. I wonder?

Failure (?) 7

After I started writing this article, I noticed that the contents of the tensorflow tutorial had changed. Actually, I used the transfer learning tutorial, but since it was troublesome to publish my code, I had no choice but to use the image classification tutorial for explanation. I think transfer learning, which is said to improve accuracy with a small number of data, is more convenient than creating a model by yourself. The current transfer learning tutorial is designed to read data from tf's, and it is complicated to change and write it, so I gave up using it for explanation. I want to use my data for transfer learning! If there is a request, you may feel like doing your best. .. ..

Recommended Posts

About the shortest path to create an image recognition model by machine learning and implement an Android application
Create an image recognition application that discriminates the numbers written on the screen on android (PyTorch Mobile) [Android implementation]
Introduction to Deep Learning for the first time (Chainer) Japanese character recognition Chapter 2 [Model generation by machine learning]
Machine Learning: Image Recognition of MNIST by using PCA and Gaussian Native Bayes
I tried to process and transform the image and expand the data for machine learning
Save the graph drawn by pyqtgraph to an image
Artificial intelligence, machine learning, deep learning to implement and understand
I tried to compress the image using machine learning
The procedure from generating and saving a learning model by machine learning, making it an API server, and communicating with JSON from a browser
I tried to verify the yin and yang classification of Hololive members by machine learning
Try to evaluate the performance of machine learning / regression model
Try to evaluate the performance of machine learning / classification model
Machine learning model management to avoid quarreling with the business side
Try to implement and understand the segment tree step by step (python)
[Machine learning] I tried to do something like passing an image
Deep learning image recognition 2 model implementation
[Python Kivy] How to get the file path by dragging and dropping
I tried moving the image to the specified folder by right-clicking and left-clicking
[Python] Create a linebot to write a name and age on an image
Let's make an image recognition model with your own data and play!
Take the free "Introduction to Python for Machine Learning" online until 4/27 application
Create an AI that identifies Zuckerberg's face by deep learning ② (AI model construction)
I tried to visualize the model with the low-code machine learning library "PyCaret"
Let's make Godzilla's image recognition model preprocessing, learning and deployment feel good
Create an application that recognizes images by writing numbers on the screen on android (PyTorch Mobile) [CNN network creation]