Hello everyone. @best_not_best. This article is a continuation of First Deep Learning ~ Preparation ~. If you haven't read it, please read it first.
This article is a product of personal desires and is not the official view of the organization to which it belongs.
I wanted to use the Google Image Search API, but according to the following article, usage restrictions seem to be strict. Collecting images with Web image search-Throat candy
Use the Bing Search API. I also referred to the above article for the source code.
get_images.py
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import sys
import os
import requests
import urllib
import urllib2
import json
BING_URL = 'https://api.datamarket.azure.com/Bing/Search/Image?'
MS_ACCTKEY = 'hogehoge'
QUERY = 'The name of your favorite celebrity'
OUTPUT_DIR_PATH = './talent/'
opener = urllib2.build_opener()
urllib2.install_opener(opener)
def download_urllist(urllist):
for url in urllist:
try:
print 'url: ' + url
with open(OUTPUT_DIR_PATH + '/' + os.path.basename(url), 'wb') as f:
img = urllib2.urlopen(url, timeout = 5).read()
f.write(img)
except urllib2.URLError:
print('URLError')
except IOError:
print('IOError')
except UnicodeEncodeError:
print('EncodeError')
except OSError:
print('OSError')
if __name__ == "__main__":
query = urllib2.quote(QUERY)
step = 20
num = 50
url_param_dict = {
'Query': "'"+QUERY+"'",
'Market': "'ja-JP'",
}
url_param_base = urllib.urlencode(url_param_dict)
url_param_base = url_param_base + "&$format=json&$top=%d&$skip="%(num)
for skip in range(0, num*step, num):
url_param = url_param_base + str(skip)
url = BING_URL + url_param
print url
response = requests.get(url,
auth = (MS_ACCTKEY, MS_ACCTKEY),
headers = {'User-Agent': 'My API Robot'})
response = response.json()
urllist = [item['MediaUrl'] for item in response['d']['results']]
download_urllist(urllist)
Execute it with the following command.
$ python get_images.py
MS_ACCTKEY is the primary account key of Azure Market Place, QUERY is the character string you want to search, and ʻOUTPUT_DIR_PATH` is the storage directory of the retrieved file, so select it as appropriate.
With this, you can get about 1,000 images for 50 x 20 pages. You did it!
I wanted to use the previous cutout_face.py as it is, but since some of the images collected from the web are irregular, I added exception handling when cropping. By the way, I didn't like the fact that the cut out image file name contained Japanese, so I try to use the serial number as the file name.
cutout_talent_face.py
#!/usr/bin/env python
# -*- coding: UTF-8 -*-
import numpy
import os
import cv2
CASCADE_PATH = '/usr/local/opt/opencv/share/OpenCV/haarcascades/haarcascade_frontalface_alt.xml'
INPUT_DIR_PATH = './talent/'
OUTPUT_DIR_PATH = './talent_cutout/'
OUTPUT_FILE_FMT = '%s%d_%d%s'
count = 1
files = os.listdir(INPUT_DIR_PATH)
for file in files:
input_image_path = INPUT_DIR_PATH + file
#File reading
image = cv2.imread(input_image_path)
#Grayscale conversion
try:
image_gray = cv2.cvtColor(image, cv2.cv.CV_BGR2GRAY)
except cv2.error:
continue
#Acquire the features of the cascade classifier
cascade = cv2.CascadeClassifier(CASCADE_PATH)
#Execution of object recognition (face recognition)
facerect = cascade.detectMultiScale(image_gray, scaleFactor=1.1, minNeighbors=1, minSize=(1, 1))
if len(facerect) > 0:
#Saving recognition results
i = 1
for rect in facerect:
print rect
x = rect[0]
y = rect[1]
w = rect[2]
h = rect[3]
path, ext = os.path.splitext(os.path.basename(file))
output_image_path = OUTPUT_FILE_FMT % (OUTPUT_DIR_PATH, count, i, ext)
i += 1
try:
cv2.imwrite(output_image_path, image[y:y+h, x:x+w])
except cv2.error:
print file
continue
count += 1
Usage is similar to cutout_face.py.
I referred to the following. Chainer's NIN allows you to deep-learn and recognize your own image set --- shi3z's long diary Can Osomatsu-san's six children be identified by deep learning? ~ Implementation ~ --bohemia diary
Clone the source.
[work_dir]$ git clone https://github.com/shi3z/chainer_imagenet_tools.git
The ImageNet source will also be cloned.
[work_dir]$ git clone https://github.com/pfnet/chainer.git
[work_dir]$ cd chainer
[chainer]$ git checkout -b 1.4.1 refs/tags/v1.4.1
[chainer]$ cd ..
Download and unzip Caltech 101
[work_dir]$ cd chainer_imagenet_tools
[chainer_imagenet_tools]$ wget http://www.vision.caltech.edu/Image_Datasets/Caltech101/101_ObjectCategories.tar.gz
[chainer_imagenet_tools]$ tar xzvf 101_ObjectCategories.tar.gz
In the 101_ObjectCategories directory, put the directory containing the image cut out in 4. Also, if there are many directories, it will take time to learn, so some directories have been deleted. The structure of the 101_ObjectCategories directory is as follows.
The panda is cute.
drwxr-xr-x 472 talent_cutout
drwxr-xr-x 438 Faces_easy
drwxr-xr-x 438 Faces
drwxr-xr-x 41 panda
Create train.txt, test.txt, label.txt.
[chainer_imagenet_tools]$ python make_train_data.py 101_ObjectCategories
Set the image size to 256 x 256.
[chainer_imagenet_tools]$ python crop.py images/ images/
Generate mean.npy.
[chainer_imagenet_tools]$ python ../chainer/examples/imagenet/compute_mean.py train.txt
Start learning!
[chainer_imagenet_tools]$ python ../chainer/examples/imagenet/train_imagenet.py -g -1 -E 20000 train.txt test.txt 2>&1 | tee log
...
train 3315 updates (106080 samples) time: 14:22:12.501801 (2.05054842326 images/sec)epoch 110
It doesn't end even after 14 hours (´; ω; `)
That's all for this time. (I will study and come back again.) Continue to the final episode (scheduled to be published later this year)! !!
solved! → First Deep Learning ~ Solution ~ --Qiita
Recommended Posts