This is Ichi Lab from RHEMS Giken.
This time, using TensorFlow's Object Detection API, Detect the object you want to recognize I finally tried it on an Android device.
Many helpful sites have helped me, Still, there were so many things I couldn't solve with just one or two pages.
In this article, I will pick up the parts that I stumbled upon, so I will describe them. I hope that the number of people who have similar difficulties will be reduced as much as possible.
TensorFlow 1.15
MacBook Pro OS : Catalina 10.15 CPU : Intel Core i5 2.3GHz Memory : 8GB 2133MHz
MacBook Pro OS : Catalina 10.15 CPU : Intel Core i7 3.5GHz Memory : 16GB 2133MHz
I changed from 1 to 2 for some reason on the way.
Basically, it is as written in the formula below.
TensorFlow Object Detection - Installation
Get the latest source of Object Detection API with the following command.
$ git clone --depth 1 https://github.com/tensorflow/models.git
Where to set PYTHONPATH, it is explained in pwd, but if it is described in full, it is as follows.
(There is also a container whose root directory is tf instead of tensorflow, so check it.)
$ export PYTHONPATH=$PYTHONPATH:/tensorflow/models/research:/tensorflow/models/research/slim
If you use Docker to run it in a container, there are various types as shown on the official page below,
Be careful if you use latest (latest version).
Because, as of November 2019 at the time of writing, ** Object Detection API is This is because it does not support TensorFlow 2.0 **.
Similarly, when installing with the pip command, it is essential to check the version of TensorFlow.
If you want to check, you can type the following command.
$ pip list | grep tensor
tensorboard 1.15.0
tensorflow 1.15.0rc3
tensorflow-estimator 1.15.1
If you want to change the version from 2.0 to 1.X, run the following command. (Changed to 1.15 in this example)
$ pip install tensorflow==1.15.0rc3
By the way, the difference in this version is quite important, so I will add it, The official TensorFlow has prepared a script that automatically converts even if the version is 2.0. However, it is described as ʻexcept for contrib` (excluding contrib).
Migrate your TensorFlow 1 code to TensorFlow 2
If you run the Object Detection API model_builder_test.py in TensorFlow 2.0,
It will fail as follows.
AttributeError: module 'tensorflow' has no attribute 'contrib'
You're stuck with an error in contrib. This means that automatic conversion scripts cannot be used at this time.
By the way, did you run model_builder_test.py safely and it was displayed as OK?
model_builder_test.py does the following in the models / research directory:
$ python object_detection/builders/model_builder_test.py
If successful, OK will be displayed as shown below. (Since it is long, some parts are omitted)
Running tests under Python 3.6.8: /usr/local/bin/python
[ RUN ] ModelBuilderTest.test_create_faster_rcnn_model_from_config_with_example_miner
...
[ RUN ] ModelBuilderTest.test_unknown_ssd_feature_extractor
[ OK ] ModelBuilderTest.test_unknown_ssd_feature_extractor
----------------------------------------------------------------------
Ran 16 tests in 0.313s
By the way, except for TensorFlow </ b>, I didn't have any effect even if I didn't care about the version, so I think you can install the latest one. That is all for building the environment.
To detect the object you want to recognize, create teacher data. In order to create teacher data, we have to prepare a lot of images that we want to detect.
If you find it difficult to collect by yourself, there is also a handy tool called google-images-download, so
If you are interested, please check it out and use it.
Now, let's explain the teacher data. The teacher data is prepared in the TFRecord format </ b> recommended by TensorFlow. TFRecord is a simple form for storing a series of binary records, The data is serialized (saved as array data) and can be read continuously. There are roughly two ways to create TFRecord format data.
Annotation is so-called tagging. While displaying photos and videos, it is a task to teach "this part is this". I used a tool from Microsoft called VoTT </ b> for this. VoTT
As you can see, it is a tool that has various difficult points to use, but It is recommended in that respect because it eliminates the hassle of converting to TFRecord format.
It is possible to generate a TFRecord format file from an image with the library provided by TensorFlow. However, with the TFRecord generated in the official tutorial below, The TFRecord generated by VoTT had slightly different element names and could not be mixed and trained together. Usage of TFRecords and tf.Example
This document will be more useful. using_your_own_dataset.md
The story around here will be long, so I summarized it in another article, so please have a look if you like. TFRecord file creation memorandum for object detection
The explanation from here is based on the assumption that all directories are based on models / research.
Once you have the TFRecord format file ready, it's time to start learning. Here, let's learn your own original based on the existing learning model called "transfer learning". We will proceed with the method of creating a more customized learning model. Transfer learning uses weights learned in advance on large-scale data. It is expected that sufficient performance can be obtained even with a small amount of training data.
First, download the trained model for transfer learning.
This time, I couldn't find it used in other Japanese articles.
Let's proceed with the new MobileNet v3.
The trained model can be downloaded from the following page.
Tensorflow detection model zoo
In this, download the one called ssd_mobilenet_v3_large_coco.
Please note that this model will only work with the latest sources updated around mid-October 2019.
Place the downloaded model in the ʻobject_detection` directory. If you want to run it with a command, do the following:
$ wget http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v3_large_coco_2019_08_14.tar.gz
$ tar zxvf ssd_mobilenet_v3_large_coco_2019_08_14.tar.gz
The file structure of the learning model is basically as follows.
When starting transfer learning, edit the contents of the file pipeline.config in this and use it.
This section describes the prepared TFRecord file directory. The TFRecord file has separate directories for training (teacher) and verification. Generally, the ratio is training: validation is 8: 2, that is, training is divided into 80% of the data and the remaining 20% is divided into validation. (Book "Practical Machine Learning with scikit-learn and TensorFlow" From P30) Of course, it is possible to study even if you do not divide it properly. Below is the directory structure. (Train stands for training, val stands for validation)
TFRecord file names should be unified with serial numbers instead of separate numbers. For example, here we name it as follows.
How to name the file here is related to the setting of the following config file.
Open pipeline.config in the trained model you downloaded earlier.
Here you can adjust various parameters such as batch size, weighting, and image expansion, but I will explain the items that you will often edit.
(Excerpt from the actual config for explanation)
pipeline.config
model {
ssd {
num_classes: 1
num_classes :
It is a setting of the number to classify.
It is written relatively at the top of the config file.
pipeline.config
train_config: {
batch_size: 32
num_steps: 10000
optimizer {
momentum_optimizer: {
learning_rate: {
cosine_decay_learning_rate {
total_steps: 10000
warmup_steps: 10000
fine_tune_checkpoint: "./object_detection/ssd_mobilenet_v3_large_coco/model.ckpt"
}
batch_size :
When using stochastic gradient descent, to reduce the effects of outliers
Train the dataset in several subsets.
The number of data contained in each subset is called the batch size.
This value is often the value of 2 to the nth power as is customary in the field of machine learning.
And, the larger this value is, the more load is applied during learning, and depending on the environment, the process may die and not learn.
num_steps :
The number of steps to learn.
The number of steps can also be specified in the command when performing learning,
As far as I've tried, the command specification takes precedence.
total_steps and warmup_steps:
I am investigating because it is an item that was not in the config of other models,
total_steps must be greater than or equal to warmup_steps.
(If this condition is not met, an error will occur and learning will not start.)
fine_tune_checkpoint :
Specifies a model for transfer learning.
It is OK if you write the directory containing the downloaded learning model up to "~ .ckpt".
This item was not included at the time of downloading ssd_mobilenet_v3.
That's why I added it myself.
Most models already have this item.
This line is unnecessary if you do not perform transfer learning.
pipeline.config
train_input_reader: {
tf_record_input_reader {
input_path: "./object_detection/test0001/train/hoge????.tfrecord"
}
label_map_path: "./object_detection/test0001/tf_label_map.pbtxt"
}
eval_input_reader: {
tf_record_input_reader {
input_path: "./object_detection/test0001/val/hoge????.tfrecord"
}
label_map_path: "./object_detection/test0001/tf_label_map.pbtxt"
}
input_path :
Specify the directory of the prepared training and verification TFRecord files.
For example, in this case, we named it with a serial number of hoge0000.tfrecord, so write it as hoge ????. Tfrecord.
label_map_path :
Specify the prepared label. There is no problem with the same specification for train and eval.
After writing the config file, it's time to start learning.
For training, use the file model_main.py located in the ʻobject_detection` directory.
A description of the run-time arguments.
--model_dir：
Specify the save destination of the training data.
A file called ".ckpt ~" will be created in the specified directory during learning.
--pipeline_config_path：
Specify the config file to use.
Let's specify the config file you edited earlier.
--num_train_steps：
Specify the number of learnings.
This option is not needed if you want to implement the number specified in config,
If you want to train at a different number of times than config, the number of times specified here will take precedence. (Result of actual trial)
The following is an execution example.
$ python object_detection/model_main.py \
--pipeline_config_path="object_detection/ssd_mobilenet_v3_large_coco/pipeline.config" \
--model_dir="./object_detection/test0001/save" \
--alsologtostderr
Since there are many options and it is troublesome to enter them one by one, it will be easier to execute if you make a shell script.
#! /bin/bash
PIPELINE_CONFIG_PATH="./object_detection/ssd_mobilenet_v3_large_coco.config"
MODEL_DIR="./object_detection/test0001/save"
NUM_TRAIN_STEPS=10000
cd '/tensorflow/models/research'
python object_detection/model_main.py \
--pipeline_config_path=$PIPELINE_CONFIG_PATH \
--model_dir=$MODEL_DIR \
# --num_train_steps=$NUM_TRAIN_STEPS \
--alsologtostderr
"Oh, I can't run the shell I made ...?" Did you forget to change the permissions?
$ chmod 775 hoge.sh
If you think that learning has begun, it may happen as follows.
~ session_manager.py:500] Running local_init_op.
~ session_manager.py:502] Done running local_init_op.
~ basic_session_run_hooks.py:606] Saving checkpoints for 0 into {Save checkpoint The specified directory/file name}.ckpt
Killed
There are several reasons why the process dies, In both cases, it is highly possible that the operating environment is running out of resources.
Here, I will introduce the means that I solved when I experienced this event.
This is a method that may be useful if you are doing it inside a Docker container. First open the settings. Select Preferences → Advanced tab </ b> to increase resources.
This is resolved by setting pipeline.config.
What kind of parameter combination will kill the process? I think it depends on the operating environment and the amount and size of teacher data used. First of all, it may be solved by reducing the batch size value, so If you are worried here, give it a try.
The training data will be saved at any time in the save destination directory specified when model_main.py is executed.
Data will be saved at the timing when Saving is displayed as shown below during learning.
At this stage, you can visualize the situation during learning with the Tensorboard explained below.
INFO:tensorflow:Saving 'checkpoint_path' summary for global step 500: object_detection/test0001/save/model.ckpt-500
I1012 08:29:56.544728 139877141301056 estimator.py:2109] Saving 'checkpoint_path' summary for global step 500: object_detection/test0001/save/model.ckpt-500
Visualization of training data is essential, and Tensorboard is very helpful for that.
You can start tensorboard with the following command.
$ tensorboard --logdir=object_detection/test0001/save
--logdir :
Specify the directory where the learning data is saved, which was specified at the start of learning.
If you specify the save destination of the learning results performed in the past, you can view it again even after the learning is completed.To see Tensorboard in your web browser, go to localhost. By default, the port number is 6006.
http://localhost:6006/
If you are running in a Docker container environment, make sure you have access to -p 6006: 6006 when you do docker run.
I have a port setting in docker-compose.yml.
Even if you can visualize learning, it's painful if you don't understand the point of view at all. I have summarized the contents of my research, so I hope you can refer to them. GRAPHS The graph is automatically generated from the processing of the source code.
| item | Description |
|---|---|
| Run | You can switch the subdirectory where the log is located |
| Upload | Upload Tensorflow model file |
| Trace inputs | Being able to chase node dependencies |
| Color | Choose a color coding method - Structure :Model (network) configuration - Device :Processed device (CPU vs GPU) - Compute time :processing time - Memory :memory usage - TPU Compatibility :Run on tensor processing unit |
SCALARS
| item | Description |
|---|---|
| Show data download links | Display a link where you can save the graph. You can choose CSV or JSON format. |
| ignore outliners in chart scaling | Whether to scale the graph to avoid outliers (check to avoid) |
| Tooltip sorting method | Tooltip order - default :Alphabetical order - descending :In descending order of value - ascending :In ascending order of value - nearest :Close to mouse cursor |
| Smoothing | Graph smoothing |
| Horizontal Axis | Specifying the horizontal axis (X axis) of the line graph - STEP :Step (number of executions) - RELATIVE :Execution time (difference from the first time) - WALL :Times of Day - Runs :Show / hide graph |
Reference: Visualization of learning with TensorBoard Next, I will explain each of the main graphs. Before that, I will give a description of the words that you need to know for explanation.
Reference: Meaning of IoU (evaluation index) and strictness of value Reference: [For beginners] Explanation of evaluation indicators for classification problems in machine learning (correct answer rate, precision rate, recall rate, etc.)
The following is an excerpt with reference to the explanation of the items described in ʻobject_detection / metrics / coco_tools.py`.
| item | Description |
|---|---|
| Precision/mAP | Average accuracy of the class averaging IOU thresholds in the range 5-95 in 5 increments. |
| Precision/[email protected] | Average accuracy of 50% IOU |
| Precision/[email protected] | Average accuracy of 75% IOU |
| Precision/mAP (small) | Average accuracy of small objects (less than 32 x 32 px) |
| Precision/mAP (medium) | Average accuracy of medium-sized objects(32×32 px〜96×96 px) |
| Precision/mAP (large) | Average accuracy of large objects(96×96 px〜10000×10000 px) |
| Recall/AR@1 | Percentage of averages found correctly in a single detection |
| Recall/AR@10 | Percentage of averages found correctly in 10 detections |
| Recall/AR@100 | Percentage of averages found correctly in 100 detections |
| Recall/AR@100 (small) | Average recall of small objects detected 100 times |
| Recall/AR@100 (medium) | Average recall of medium objects detected 100 times |
IMAGES
Display image data. You can check the difference between the correct answer of the verification data and the inference result in .ckpt file (checkpoint) units.
When the training is finished, it is finally time to convert the data into an inference graph. Specifically, it converts the .ckpt file created by learning into a .pb file.
There are multiple sources for conversion in the ʻobject_detection` directory.
export_inference_graph.pyexport_tflite_ssd_graph.py** To try it on Android, it will be converted to ** .tflite format, so ** use the latter **.
A description of the run-time arguments.
--input_type :
For the inference graph, specify one of the following three depending on the user
image_tensor :ʻEncoded_image_string_tensor`: 1D tensor [None] Includes encoded PNG or JPEG images. If multiple images are provided, it is assumed that the image resolutions are the same.
tf_example: 1D string tensor [None]
If multiple images are provided, including serialized TF Example protos, it is assumed that the image resolutions are the same.
--pipeline_config_path :
Specify the config file used during training.
--trained_checkpoint_prefix :
Specify the "model.ckpt-XXXX" file created as the save destination for the training data.
For XXXX, specify the latest (largest) number of learning steps performed.
For example, if it is executed 10000 times, it will be "model.ckpt-10000".
--output_directory :
Specify the directory you want to export.
In the specified directory, a file with exactly the same structure as the file structure of the first downloaded model will be created.
The following is an execution example.
$ python object_detection/export_inference_graph.py \
--input_type image_tensor \
--pipeline_config_path object_detection/ssd_mobilenet_v3_large_coco/pipeline.config \
--trained_checkpoint_prefix object_detection/test0001/save/model.ckpt-10000 \
--output_directory object_detection/test0001/output
A program that converts a trained model to a .tflite compatible model.
The arguments are basically the same as ʻexport_inference_graph.py. (There is no --input_type`)
The following is an execution example.
$ python object_detection/export_tflite_ssd_graph.py \
--pipeline_config_path=object_detection/ssd_mobilenet_v3_large_coco/pipeline.config \
--trained_checkpoint_prefix=object_detection/test0001/save/model.ckpt-10000 \
--output_directory=object_detection/test0001/tflite \
--add_postprocessing_op=true
Two files, tflite_graph.pb and tflite_graph.pbtxt, are created in the directory specified by --output_directory.
Use tflite_convert to convert to tflite format.
This converter should be included from the beginning.
If you type tflite_convert --help, usage will come out.
Since there are many options to add, I summarized it in a shell script as follows.
Change the directory to suit you.
In addition, I recognize that --input_shapes is probably a value that matches ʻimage_resizer` of config at the time of learning, but there is no confirmation ...
The meaning of the numbers is (batch size, input image height, input image width, input image depth (RGB channel)).
Converter command line reference
#! /bin/bash
OUTPUT_FILE="object_detection/test0001/tflite/test.tflite"
GRAPH_DEF_FILE="object_detection/test0001/tflite/tflite_graph.pb"
INTERFACE_TYPE="FLOAT"
INPUT_ARRAY="normalized_input_image_tensor"
OUTPUT_ARRAYS="TFLite_Detection_PostProcess,TFLite_Detection_PostProcess:1,TFLite_Detection_PostProcess:2,TFLite_Detection_PostProcess:3"
INPUT_SHAPES="1,300,300,3"
cd '/tensorflow/models/research'
tflite_convert \
--output_file=$OUTPUT_FILE \
--graph_def_file=$GRAPH_DEF_FILE \
--inference_type=$INTERFACE_TYPE \
--input_arrays=$INPUT_ARRAY \
--input_shapes=$INPUT_SHAPES \
--output_arrays=$OUTPUT_ARRAYS \
--default_ranges_min=0 \
--default_ranges_max=6 \
--mean_values=128 \
--std_dev_values=127 \
--allow_custom_ops
In addition, I found multiple instructions to install bazel and toco on other sites, but it was possible only with this command line without them.
(I tried those methods to be exact, but the result was the same)
At this time, if the version of TensorFlow is ** 1.12.0rc0, it failed **, With ** 1.15.0rc3 **, the exact same command ** worked **.
In addition, the official description recommended the Python API. Converter command line reference
It's been a long time, but it's finally in production.
First, let's install "Android Studio". If you find other tools easier to use, you can use them.
Next, download the sample collection from the official.
$ git clone --depth 1 https://github.com/tensorflow/examples.git
After launching Android Studio, open ʻexamples / lite / examples / object_detection / android`.
ʻPlace yourtest.tflite and labelmap.txt in the examples / lite / examples / object_detection / android / app / src / main / assetsdirectory. labelmap.txt` is a text file that lists the tag names you have tagged.
For example, if you prepare two types of tag names, "apple" and "orange", the text file will be as follows.
labelmap.txt
???
apple
orange
The important thing is that the first line should be ???.
It's long, but edit DetectorActivity.java in the ʻexamples / lite / examples / object_detection / android / app / src / main / java / org / tensorflow / lite / examples / detection /` directory.
DetectorActivity.java
private static final boolean TF_OD_API_IS_QUANTIZED = false; //true->false
private static final String TF_OD_API_MODEL_FILE = "test.tflite"; //detect.tflite -> my tflite
private static final String TF_OD_API_LABELS_FILE = "file:///android_asset/labelmap.txt"; //my txt
Then edit build.gradle in the ʻexamples / lite / examples / object_detection / android / app /` directory.
Comment out the following around line 40: (If you do not comment out here, it will be replaced with the default sample data when building)
build.gradle
apply from:'download_model.gradle' //Comment out
Once you've done this, all you have to do is build it, run it on your device, and see! Congratulations if you can try it well.
Below is a reference site for Android
Training and serving a realtime mobile object detector in 30 minutes with Cloud TPUs How to Train Your Own Custom Model with Tensorflow Object Detection API and Deploy It into Android with TF Lite Detecting Pikachu on Android using Tensorflow Object Detection
What did you think? It looks like a summary page on another site, but At the very least, I think we've covered quite a bit of the information we might need to use TensorFlow's Object Detection API.
In particular, TensorFlow has different detailed behavior depending on the version, Since the versions of various reference sites used at that time are different, I think that many people have had the same difficulty.
There are still many parts that I don't understand, but I hope it helps. Thank you for reading the long article.
Recommended Posts