Object extraction in images by pattern matching using OpenCV with Python

Introduction

There was a consultation, "Isn't it possible to automatically find the same object in the photo and cut it out according to the angle and position?"

The condition is that what you want to discover is one object, and only one object is shown in the photo.

I had a little trouble to realize it, so I will record it. Also, it seems that adjustment is necessary depending on the image (^ _ ^;)

Operating environment

Environment construction procedure

Please refer to the following URL for the usual front miso.

-Procedure to quickly create a deep learning environment on Mac with TensorFlow and OpenCV

-Procedure to quickly create a machine learning environment on Ubuntu 16.04

Photo used

First sheet (IMG_4754.JPG)

Second piece (IMG_4777.JPG)

Here, the first piece is manually processed and edited as follows. (IMG_4754s.JPG)

Use this as a template image.

Library read

python

import cv2
import math

In addition, declare the following function for displaying the image.

python

from IPython.display import display, Image

def display_cv_image(image, format='.png'):
    decoded_bytes = cv2.imencode(format, image)[1].tobytes()
    display(Image(data=decoded_bytes))

Image reading and feature extraction

I referred to the following article.

Feature matching with OpenCV 3 and Python 3 (A-KAZE, KNN)

python

#Image reading
img1 = cv2.imread("IMG_4777.JPG")
img2 = cv2.imread("IMG_4754s.JPG")

# A-Generation of KAZE detector
detector = cv2.AKAZE_create()

#Feature detection and feature vector calculation
kp1, des1 = detector.detectAndCompute(img1, None)
kp2, des2 = detector.detectAndCompute(img2, None)

# Brute-Generation of Force Matcher
bf = cv2.BFMatcher()

#Brute feature vectors-Matching with Force & KNN
matches = bf.knnMatch(des1, des2, k=2)

#Thin out data
ratio = 0.2
good = []
for m, n in matches:
    if m.distance < ratio * n.distance:
        good.append([m])

#Sort features according to matching status
good = sorted(matches, key = lambda x : x[1].distance)

#Draw corresponding feature points
img3 = cv2.drawMatchesKnn(img1, kp1, img2, kp2, good[:2], None, flags=2)

display_cv_image(img3, '.png')

The following image is displayed, and you can see that the feature points are matched correctly.

Cut out by adjusting the angle, position, and size

python

#Get feature data

q_kp = []
t_kp = []

for p in good[:2]:
    for px in p:
        q_kp.append(kp1[px.queryIdx])
        t_kp.append(kp2[px.trainIdx])

#Calculate the angle and distance between feature points from the image to be processed
q_x1, q_y1 = q_kp[0]
q_x2, q_y2 = q_kp[-1]

q_deg = math.atan2(q_y2 - q_y1, q_x2 - q_x1) * 180 / math.pi
q_len = math.sqrt((q_x2 - q_x1) ** 2 + (q_y2 - q_y1) ** 2)

#Calculate the angle and distance between feature points from the template image
t_x1, t_y1 = t_kp[0]
t_x2, t_y2 = t_kp[-1]

t_deg = math.atan2(t_y2 - t_y1, t_x2 - t_x1) * 180 / math.pi
t_len = math.sqrt((t_x2 - t_x1) ** 2 + (t_y2 - t_y1) ** 2)

#Calculation of cutout position
x1 = q_x1 - t_x1 * (q_len / t_len)
x2 = x1 + img2.shape[1] * (q_len / t_len)

y1 = q_y1 - t_y1 * (q_len / t_len)
y2 = y1 + img2.shape[0] * (q_len / t_len)

#Image size
x, y, c = img1.shape
size = (x, y)

#Center position of rotation
center = (q_x1, q_y1)

#rotation angle
angle = q_deg - t_deg

#Size ratio
scale = 1.0

#Calculation of rotation transformation matrix
rotation_matrix = cv2.getRotationMatrix2D(center, angle, scale)

#Affine transformation
img_rot = cv2.warpAffine(img1, rotation_matrix, size, flags=cv2.INTER_CUBIC)

#Image cropping
img_rot = img_rot[y1:y2, x1:x2]

#Scale adjustment
x, y, c = img2.shape
img_rot = cv2.resize(img_rot, (y, x))

#Result display
display_cv_image(img_rot, '.png')

When executed, the following image was displayed.

did it!

Future tasks

It may be used for things taken at a fixed position, such as a fixed-point camera, but if it is slanted, another process is required.

Since we are doing the same calculation repeatedly, it may be better to make it a function and make it easier to reuse ... rather than that.

... I will find time again and think about it (^-^)

2017.08.20 postscript I did a little different test. Please also see. Try projective transformation of images using OpenCV with Python