[PYTHON] Machine Learning with Caffe -1-Category images using reference model

※ WIP。

Introduction

• I just added caffe and don't know how to use it
• I want to implement something with machine learning
• I want to deepen my understanding through implementation rather than reading papers
• I've never done python. Indentation is important
• I've done some programming

goal

Classify Caltech images using Caffe's reference model (* the model used in famous papers for which parameter tuning has been completed). The goal is to produce the following output as a concrete visible form.

(Image. You can get on when the implementation is completed successfully)

Rough procedure

** STEP1. ** Download the image dataset to classify ** STEP2. ** Extract features from dataset images ** STEP3. ** Train SVM to classify extracted features by linear SVM. ** STEP4. ** Classify based on features with trained SVM

1. Download the image dataset to classify

Jump to the following page and Download. http://www.vision.caltech.edu/Image_Datasets/Caltech101/#Download

2. Extract features from dataset images

2-1. Download reference model

$ scripts/download_model_binary.py models/bvlc_reference_caffenet

2-2. Implementation of code to extract image features

Extract feature data of images using a reference model. Just as colors are represented by three numbers, RGB, the features of one image in this model are represented by ** 4096 numbers **. In 2-2., Input: jpg data, output: 4096 numerical data, create a script to perform various processing.

Create the following in the caffe root directory.

feature_extraction.py

#! /usr/bin/env python
# -*- coding: utf-8 -*-
import sys, os, os.path, numpy as np, caffe

# path to git-cloned caffe dir
CAFFE_DIR  = os.getenv('CAFFE_ROOT')

MEAN_FILE  = os.path.join(CAFFE_DIR, 'python/caffe/imagenet/ilsvrc_2012_mean.npy')
MODEL_FILE = os.path.join(CAFFE_DIR, 'models/bvlc_reference_caffenet/deploy.prototxt')
PRETRAINED = os.path.join(CAFFE_DIR, 'models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel')

LAYER = 'fc7'
INDEX = 4

class FeatureExtraction:

    def __init__(self):
        net = caffe.Classifier(MODEL_FILE, PRETRAINED)
        caffe.set_mode_cpu()
        net.transformer.set_mean('data', np.load(MEAN_FILE))
        net.transformer.set_raw_scale('data', 255)
        net.transformer.set_channel_swap('data', (2,1,0))
        self.net = net

    def extract_features(self):
        imageDirPath = sys.argv[1]
        previousLabelName = ''
        labelIntValue = 0
        for root, dirs, files in os.walk(imageDirPath):
            for filename in files:
                if filename == '.DS_Store': 
                    continue
                fullPath  = os.path.join(root, filename)
                dirname   = os.path.dirname(fullPath)
                labelName = dirname.split("/")[-1]
                if labelName != previousLabelName:
                    labelIntValue += 1
                    previousLabelName = labelName
                image = caffe.io.load_image(fullPath)
                feat = self.extract_features_from_image(image)
                self.print_feature_with_libsvm_format(labelIntValue, feat)

    def build_test_data(self, imagePaths):
        for fullPath in imagePaths:
            image = caffe.io.load_image(fullPath)
            feat = self.extract_features_from_image(image)
            self.print_feature_with_libsvm_format(-1, feat)

    def extract_features_from_image(self, image):
        self.net.predict([image])
        feat = self.net.blobs[LAYER].data[INDEX].flatten().tolist()
        return feat 

    def print_feature_with_libsvm_format(self, labelIntValue, feat):
        formatted_feat_array = [str(index+1)+':'+str(f_i) for index, f_i in enumerate(feat)]
        print str(labelIntValue) + " " + " ".join(formatted_feat_array)

2-3. Export the feature data of all images downloaded in STEP1 to one file

Prepare the script for the above execution

exec.py

#! /usr/bin/env python
# -*- coding: utf-8 -*-
import sys
from feature_extraction import FeatureExtraction 
FeatureExtraction().extract_features()

Execute the following to create feature data (feature.txt).

python

$ python exec.py path/to/images_dir > feature.txt

On the machine at hand, the first line

(10, 3, 227, 227)

Will be included. This is not feature data, it's like a garbage print in another process, so delete it.

Supplement: About the format of the output feature data

In STEP3., SVM is learned by libsvm. In order to handle with libsvm, it is necessary to write out the feature data in the following format.

...
4 1:0.89 2:0.19 3:0.10 ...  4096:0.77 
1 1:0.01 2:0.99 3:0.11 ...  4096:0.97 
...

1 data,

(label number) 1:Numerical value of the first feature 2:Numerical value of the second feature...

It is expressed in the form of. In feature.txt, there are as many lines as the number of images.

3. Train the SVM to classify the extracted features by linear SVM.

The famous libsvm package is used for SVM. The explanation of libsvm and svm is kind here.

• Master SVM! 8 checkpoints
• Machine learning with libsvm that can be set in 10 seconds
• References for support vector machines found on the Web

3-1. Installation of libsvm

$ brew install libsvm

3-2. Learning

Train SVM. Type the following command.

$ svm-scale -s scale.txt feature.txt > feature.scaled.txt
$ svm-train -c 0.03 feature.scaled.txt caltech101.model

svm-scale is a command to scale with libsvm, and svm-train is a command to learn. The meaning of each file is as follows.

• scale.txt: A file that saves how you scaled.
• feature.txt: Feature data file (created in STEP2)
• feature.scaled.txt: Feature data after scaling
• caltech101.model: File where the trained SVM parameters are stored

4. Category classification by trained SVM

• WIP. Suspicion of misunderstanding at SVM.

4-1. Experiment with the same data as a trial

$ cp feature.txt feature_test.txt
$ svm-scale -r scale.txt feature_test.txt > feature_test.scaled.txt
$ svm-predict feature_test.scaled.txt caltech101.model result.txt

... accuracy is bad! now debugging ...

Next Step Candidate

• Fine tuning
• Make a model from scratch
• Detection

Hopefully all three will be in August. .. ..

Referenced URL list

In the whole flow

• Focusing on Caffe's Deep Learning where it is easy to trip
• A brief description of the general implementation flow of Caffe. If you want to know the whole picture, this is it.
• Easy image classification with Caffe

• This original story. However, as of 2015-08-21, it is not easy to do so, so proceed while looking at other reference URLs.

• Until you run the feature extractor with Caffe's ImageNet

• An arrangement of the original material into the current version.

• I tried "Classification using Caffe as a feature extractor" of "Easy image classification with Caffe" 1

• Not very helpful.

• "Easy image classification with Caffe" cannot be done easily.

• FAQ-like material in case of trouble. Thank you very much.

• Try Caffe, Pylearn2 together

• It will be a reference for the procedure when using the reference model with Caffe.

• I tried feature extraction by Caffe + classification by AROW

• I used the source code of the feature extraction part as a reference.

• Caffe Pukiwiki

• Refer to here for the code to format for libsvm.

libsvm

• Machine learning with libsvm that can be set in 10 seconds
• Tutorial-like material of libsvm.
• Homebrew + Install libsvm in Virtualenv environment
• Libsvm with python. I want to use it. I installed it, but I haven't used it. Postponed.

FAQ

Q1. What is python / caffe / imagenet / ilsvrc_2012_mean.npy?

A. An average image. See below.

http://qiita.com/uchihashi_k/items/8333f80529bb3498e32f

Q2. Is SVM a binary classifier?

A. Multi-value classification is also possible. libsvm casually counts the number of classes of teacher data you enter and does a good job of creating a multi-value classifier if needed ... it wasn't a sweet story.