[PYTHON] Load caffe model with Chainer and classify images

Load the caffe model with Chainer and classify the images. The Chainer sample also has image classification, but I can't tell which image was classified into which category just by outputting the recognition rate. Allows you to output the category name and score as the classification result. You can find the source code at here. (A classified version of the code in this article) If you find it difficult to read the article, please clone it.

Download caffe model

This time, we will use bvlc_googlenet as the model. 1000 categories can be classified. There is a link to the caffemodel file on the bvlc_googlenet page, so download it from there.

Generate a label file

A label file is generated so that the category number of the classification result and the category name can be associated. Below is a script to download imagenet related files. https://github.com/BVLC/caffe/blob/master/data/ilsvrc12/get_ilsvrc_aux.sh A label file is generated by processing the synset_words.txt included in caffe_ilsvrc12.tar.gz described in this.

`synset_words.txt`


n01440764 tench, Tinca tinca
n01443537 goldfish, Carassius auratus
n01484850 great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias
n01491361 tiger shark, Galeocerdo cuvieri

Execute the following command

wget http://dl.caffe.berkeleyvision.org/caffe_ilsvrc12.tar.gz
tar -xf caffe_ilsvrc12.tar.gz
sed -e 's/^[^ ]* //g' synset_words.txt > labels.txt

The label file is created.

`labels.txt`


tench, Tinca tinca
goldfish, Carassius auratus
great white shark, white shark, man-eater, man-eating shark, Carcharodon carcharias
tiger shark, Galeocerdo cuvieri
hammerhead, hammerhead shark

Since there are two lines called "crane", it is confusing, so change the 135th line to "crane (bird)" and the 518th line to "crane (machine)".

Convert image to numpy array

Use Pillow to read the image, resize it, clip it and then convert it to a numpy array

import numpy as np
from PIL import Image

#Definition of input image size
image_shape = (224, 224)

#Read image and convert to RGB format
image = Image.open('sample.png').convert('RGB')

#Image resizing and clipping
image_w, image_h = self.image_shape
w, h = image.size
if w > h:
    shape = (image_w * w / h, image_h)
else:
    shape = (image_w, image_h * h / w)
x = (shape[0] - image_w) / 2
y = (shape[1] - image_h) / 2
image = image.resize(shape)
image = image.crop((x, y, x + image_w, y + image_h))
pixels = np.asarray(image).astype(np.float32)

#pixels are 3D and each axis is[Y coordinate,X coordinate, RGB]Represents
#Input data is 4D[Image index, BGR,Y coordinate,X coordinate]So, do the array conversion
#Convert from RGB to BGR
pixels = pixels[:,:,::-1]

#Swap the axes
pixels = pixels.transpose(2,0,1)

#Draw average image
mean_image = np.ndarray((3, 224, 224), dtype=np.float32)
mean_image[0] = 103.939
mean_image[1] = 116.779
mean_image[2] = 123.68
pixels -= self.mean_image

#Make it 4D
pixels = pixels.reshape((1,) + pixels.shape)

Load caffemodel and classify

Load the caffemodel and use the array you just generated as input data.

import chainer
import chainer.functions as F
from chainer.functions import caffe

#Load caffe model
func = caffe.CaffeFunction('bvlc_googlenet.caffemodel')

#layer'loss3/classifier'Get the output of and apply softmax
x = chainer.Variable(pixels, volatile=True)
y, = func(inputs={'data': x}, outputs=['loss3/classifier'], disable=['loss1/ave_pool', 'loss2/ave_pool'], train=False)
prediction = F.softmax(y)

Output the result

The classification result is output.

#Read label
categories = np.loadtxt('labels.txt', str, delimiter="\n")

#Scores and labels are linked and sorted in descending order of score
result = zip(prediction.data.reshape((prediction.data.size,)), categories)
result = sorted(result, reverse=True)

#View the top 10 results
for i, (score, label) in enumerate(result[:10]):
    print '{:>3d} {:>6.2f}% {}'.format(i + 1, score * 100, label)

Recognition example

When I recognized the landscape image taken in Asakusa, it became as follows. The top category is now a mosque. I would like you to recognize skyscrapers and towers, but they do not seem to be in the category.

  1  38.85% mosque
  2   6.07% fire engine, fire truck
  3   5.15% traffic light, traffic signal, stoplight
  4   3.97% radio, wireless
  5   3.25% cinema, movie theater, movie theatre, movie house, picture palace
  6   2.14% pier
  7   2.01% limousine, limo
  8   1.92% stage
  9   1.89% trolleybus, trolley coach, trackless trolley
 10   1.61% crane (machine)

At the end

There are several trained caffe models available that anyone can use to classify images. This time, only one image was input, but it is possible to input multiple images at the same time. Since it takes time to load the caffemodel, it is better to load the image while keeping the caffemodel loaded.

reference

Import Caffe model using Chainer and let it recognize images on Mac without CUDA