[PYTHON] Former image processing engineer tried to solve Saizeriya's spot the difference with OpenCV

Introduction

Since I was making settings for image processing used for industrial machines in my previous job, I was curious about the algorithm of Spot the Difference in Saizeriya, so I tried out it. ..

This theme

As expected, there is no trick to use the same thing, so this time I will search for mistakes with the image below. Copyright Saizeriya Co,. Ltd All rights reserved.

From here, you can find information on the answers to finding mistakes. If you don't like spoilers, try it from Saizeriya's homepage.

Changes ... Before implementing

Confirmation of assumptions

First, let's check Saizeriya's spot the difference. There are two points to watch when looking for mistakes in Saizeriya.

1. Use the normally printed menu
2. Differences in color are included in the answer

1 means that the image file is likely to be converting the resolution from print to display. Therefore, there is no guarantee that the pixels in the same place will be the same color, and some noise countermeasures are required. 2 means that we need a method to clearly describe and detect the difference in color. It seems that better results will be obtained by comparing HSV (hue / brightness / saturation) instead of RGB.

change point

Confirmation before change

First, let's check the current situation. The difference in the initial state is as follows.

Noise removal 1

Try blurring to see if you can remove fine noise.

    #Remove image margins
    img = img_src[:, padding:-padding]
    #Split the image left and right
    height, width, channels = img.shape[:3]
    img1 = img[:, :width//2]
    img2 = img[:, width//2:]

    #Add blur to remove fine noise
    blue_size = (5, 5)
    img1 = cv2.blur(img1, blue_size)
    img2 = cv2.blur(img2, blue_size)
    
    cv2.imshow("img2",img2)

There is no atmosphere that has improved so much. Let's try another method later.

HSV conversion

Get diffs and convert to grayscale

Being aware that it is printed matter, convert it to HSV and then take the difference. Also, the answer should be that one of H, S, and V has changed significantly, so check the maximum value of the HSV difference to see if it is a mistake.

#Since the original target is printed matter, try converting it to HSV.
img1_hsv = cv2.cvtColor(img1, cv2.COLOR_RGB2HSV)
img2_hsv = cv2.cvtColor(img2, cv2.COLOR_RGB2HSV)

#Calculate the difference between two images
img_diff = cv2.absdiff(img2_hsv, img1_hsv)
diff_h = img_diff[:, :, 0]
diff_s = img_diff[:, :, 1]
diff_v = img_diff[:, :, 2]

#Get the biggest change in the HSV difference
diff_glay = np.maximum(diff_h, diff_s, diff_v)

Especially, there is a lot of noise near the characters, but since the characters are black and white, the hue (H) changes greatly with a slight variation in RGB.

Also, I'm worried that the color of the sheep's clothes, which should have been wrong, is quite dark. Consider the later processing, and change it so that the change in brightness is detected as over.

Low-saturation hue measures

Simply avoid seeing changes in hue in desaturated areas. At the same time, the difference between brightness and saturation is standardized so that it is in the range of 0 to 255.

#Calculate the difference between two images
img_diff = cv2.absdiff(img2_hsv, img1_hsv)
diff_h = img_diff[:, :, 0]
diff_s = img_diff[:, :, 1] * 3
diff_v = img_diff[:, :, 2]

#Saturation below a certain level(V)The part of is the shade(H)Do not consider the difference of
H_THRESHOLD = 70
_, diff_h_mask = cv2.threshold(diff_v, H_THRESHOLD, 255, cv2.THRESH_BINARY)
diff_h = np.minimum(diff_h, diff_h_mask)

#Normalize the difference between brightness and saturation
diff_s = cv2.normalize(diff_s, _, 255, 0, cv2.NORM_MINMAX)
diff_v = cv2.normalize(diff_v, _, 255, 0, cv2.NORM_MINMAX)

#Get the biggest change in the HSV difference
diff_glay = np.maximum(diff_h, diff_s, diff_v)

It has become much easier to understand.

Binarization and noise removal

Next, let's make it a little easier to understand what has changed. Binarize and display the difference in white.

Also, since it was full of noise as it was, cancel the blur and use the opening to remove the noise.

#Get candidates for unusual locations by binarization
DIFF_THRESHOLD = 60
_, diff_bin = cv2.threshold(diff_glay, DIFF_THRESHOLD, 255, cv2.THRESH_BINARY)

#Noise removal
diff_bin = cv2.morphologyEx(diff_bin, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3)))

Binarization only

Opening included

By arranging them in this way, the power of noise removal by the opening is obvious.

Display format improvement

In the linked article, I couldn't visually find the color depth. Let's change it so that you can easily see the change between a place where there is no difference and a certain place.

That said, the places where there are already mistakes are shown in black and white. It will be enough to incorporate this in a nice way.

#Image composition
diff_bin_rgb = cv2.cvtColor(diff_bin, cv2.COLOR_GRAY2RGB)
add = np.maximum(np.minimum(img2, diff_bin_rgb), (img2 // 3))
cv2.imshow("add", add)

The final output looks like this: The fireplace is difficult to understand, but I think you can understand it enough by looking at the data at the time of binarization. Or I think that it is easy to see if you use it by expanding (dilating) the binarized data before RGB conversion.

Source code

There were some other parts of the source of the original article that I was interested in, but this time it is not essential, so I will omit it. After putting in my knowledge of image processing, I ended up with a source like this.

import cv2
import numpy as np


# --------------------------------------------------- #
#Image composition#
# --------------------------------------------------- #

def FitImageSize_small(img1, img2):
    # height
    if img1.shape[0] > img2.shape[0]:
        height = img2.shape[0]
        width = img1.shape[1]
        img1 = cv2.resize(img1, (width, height))
    else:
        height = img1.shape[0]
        width = img2.shape[1]
        img2 = cv2.resize(img2, (width, height))

    # width
    if img1.shape[1] > img2.shape[1]:
        height = img1.shape[0]
        width = img2.shape[1]
        img1 = cv2.resize(img1, (width, height))
    else:
        height = img2.shape[0]
        width = img1.shape[1]
        img2 = cv2.resize(img2, (width, height))
    return img1, img2


img = cv2.imread('file name')

if img is None:
    print('Unable to read the file')
    import sys

    sys.exit()

cv2.imshow("img", img)

#Find the appropriate padding width so that the two images best match when the margins are removed
img_src = img
padding_result = []
for padding in range(0, 50):
    #Remove image margins
    # (Consider the possibility of no margins)
    if padding:
        img = img_src[:, padding:-padding]

    #Split the image left and right
    height, width, channels = img.shape[:3]
    img1 = img[:, :width // 2]
    img2 = img[:, width // 2:]

    #Match image size(To the smaller one)
    img1, img2 = FitImageSize_small(img1, img2)

    #Calculate the difference between two images
    img_diff = cv2.absdiff(img2, img1)
    img_diff_sum = np.sum(img_diff)

    padding_result.append((img_diff_sum, padding))

#Choose the one with the least difference
_, padding = min(padding_result, key=lambda x: x[0])

#Remove image margins
if padding:
    img = img_src[:, padding:-padding]

#Split the image left and right
height, width, channels = img.shape[:3]
img1 = img[:, :width // 2]
img2 = img[:, width // 2:]
cv2.imshow("img2", img2)

#Since the original target is printed matter, try converting it to HSV.
img1_hsv = cv2.cvtColor(img1, cv2.COLOR_RGB2HSV)
img2_hsv = cv2.cvtColor(img2, cv2.COLOR_RGB2HSV)

#Calculate the difference between two images
img_diff = cv2.absdiff(img2_hsv, img1_hsv)
diff_h = img_diff[:, :, 0]
diff_s = img_diff[:, :, 1] * 3
diff_v = img_diff[:, :, 2]

#Saturation below a certain level(V)The part of is the shade(H)Do not consider the difference of
H_THRESHOLD = 70
_, diff_h_mask = cv2.threshold(diff_v, H_THRESHOLD, 255, cv2.THRESH_BINARY)
diff_h = np.minimum(diff_h, diff_h_mask)

#Normalize the difference between brightness and saturation
diff_s = cv2.normalize(diff_s, _, 255, 0, cv2.NORM_MINMAX)
diff_v = cv2.normalize(diff_v, _, 255, 0, cv2.NORM_MINMAX)

#Get the biggest change in the HSV difference
diff_glay = np.maximum(diff_h, diff_s, diff_v)

#Get candidates for unusual locations by binarization and opening
DIFF_THRESHOLD = 60
_, diff_bin = cv2.threshold(diff_glay, DIFF_THRESHOLD, 255, cv2.THRESH_BINARY)
diff_bin = cv2.morphologyEx(diff_bin, cv2.MORPH_OPEN, cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3)))

#Match image size(To the smaller one)
img2, diff_bin = FitImageSize_small(img2, diff_bin)
cv2.imshow("img_diff", diff_bin)

#Image composition
diff_bin_rgb = cv2.cvtColor(diff_bin, cv2.COLOR_GRAY2RGB)
add = np.maximum(np.minimum(img2, diff_bin_rgb), (img2 // 3))
cv2.imshow("add", add)

cv2.waitKey(0)
cv2.destroyAllWindows()

Summary

• When performing image processing, think about the processing target
• Consider how to output the final result
• Think of the debuggee and don't write source that squeezes all exceptions