# Introduction

My previous article

So, I used the affine transformation only for rotation, so this time I will show you another way to use it and play with the projection transformation.

# Transform a parallelogram with affine transformation

Affine matrix

\begin{pmatrix}
x' \\
y' \\
1
\end{pmatrix}
=
\begin{pmatrix}
a & b & c \\
d & e & f \\
0 & 0 & 1
\end{pmatrix}
\begin{pmatrix}
x \\
y \\
1
\end{pmatrix}


The origin (0,0) is (c, f), and the unit vectors (1,0) and (0,1) are (a, d) and (b, respectively. , e) , which means that the xy plane changes to the x'y' plane. It means that any parallelogram consisting of two vectors transforms into another parallelogram, and more fundamentally, any triangle transforms into another triangle.

### Create an affine matrix cv.getAffineTransform (src, dst)

--src The apex of the triangle. Must be a numpy array of float32. --Dst 3 points after the change. Like src, it must be a float32 numpy array.

Three points that define the image size of height 300 and width 200, that is pts1 = [(0,0), (200,0), (0,300)], another 3 points Consider converting to pts2 = [(50,50), (250,100), (200,300)]. Since there are three changes in (x, y)(x', y'), all a, b, c, d, e, f are obtained, and the affine matrix is ​​determined. I haven't calculated it myself, but according to cv.getAffineTransform (), the affine matrix is

[[ 1.          0.5        50.        ]
[ 0.25        0.83333333 50.        ]]


Will be. The source is omitted. The deformation of the image due to this is as follows. You can see that (0,0) is (50,50), (200,0) is (250,100), and (0,300) is (200,300). .. At this time, (200,300) becomes(400,350)by the rule of the parallelogram. Since it starts from (0,0), please do not say that the vertex is (199,299) instead of (200,300). By the way, the original image is Macho 2 (vertical photo) (Muscle Plus) who challenges the final battle to save the world.

# Homography

A transformation that transforms an arbitrary quadrangle into another quadrangle with a higher degree of freedom is called a projective transformation. The projection matrix is

\begin{pmatrix}
x' \\
y' \\
1
\end{pmatrix}
=
\begin{pmatrix}
a & b & c \\
d & e & f \\
f & g & 1
\end{pmatrix}
\begin{pmatrix}
x \\
y \\
1
\end{pmatrix}


It is in the form of.

### Create a projection matrix cv.getPerspectiveTransform (src, dst)

--src The vertex of the quadrangle. Must be a numpy array of float32. --dst 4 points after change. Like src, it must be a float32 numpy array.

It's the same as the affine transformation cv.getAffineTransform ().

### Projection conversion of image cv.warpPerspective (src, M, dsize)

This is also the same as the affine transformation cv.warpAffine ().

## Practice

Four points of an image with a height of 300 and a width of 200, that is, pts1 = [(0,0), (0,300), (200,300), (200,0)], another 4 points Consider converting to pts2 = [(50,100), (100,400), (300,300), (200,50)]. I don't feel like solving an 8-element simultaneous equation by myself, but according to cv.getPerspectiveTransform (), the projection matrix is

[[ 8.33333333e-01  6.94444444e-02  5.00000000e+01]
[-2.29166667e-01  6.11111111e-01  1.00000000e+02]
[ 4.16666667e-04 -9.72222222e-04  1.00000000e+00]]


Will be. And when you look at the change in the image, it looks like this. You can see how each vertex is transformed.

### Source

The source is as follows. I am currently using over-spec techniques such as using the anonymous function lambda to find the maximum value of each column from a two-dimensional array, or arranging graphs by sharing axes with matplotlib.pyplot. ..

#### python


import cv2
import matplotlib.pyplot as plt
import numpy as np

h1, w1 = img1.shape[:2]

pts2 = [(50,100), (100,400), (300,300), (200,50)]
w2 = max(pts2, key = lambda x:x[0])[0]
h2 = max(pts2, key = lambda x:x[1])[1]

pts1 = np.float32([(0,0), (0,h1), (w1,h1), (w1,0)])
pts2 = np.float32(pts2)

M = cv2.getPerspectiveTransform(pts1,pts2)
img2 = cv2.warpPerspective(img1, M, (w2,h2), borderValue=(255,255,255))
print (M)

fig, (ax1, ax2) = plt.subplots(1, 2, sharex=True, sharey=True)
ax1.imshow(img1[:,:,::-1])
ax2.imshow(img2[:,:,::-1])
plt.show()


# Application example

Not only can the rectangle be made into an isosceles quadrangle, but elements (paintings, etc.) that have become an isosceles quadrangle when photographed diagonally can be taken out as if they were photographed from the front. Of course, it is necessary to know the size (aspect ratio) of the rectangle you want to take out.

Original image sf5.jpg
Extracted image

Take this monitor image and replace it with another image. The image is Macho (Muscle Plus). Trim with the same aspect ratio of 16: 9 as the game screen.

This is transformed into the isosceles quadrangle specified earlier.

Intermediate image

If you combine it with the original image, you will have a scene of fighting in a super-realistic fighting game.

Image of hacked screen

The intermediate image is well made, but the composite result shows red noise here and there. I don't think it's a bad composition because it was the same even if I composited it on a white background or set the transparent color to another color, but I'm not sure why.

Animated GIF of a series of operations

### Source

In the previous article "Getting mouse events with OpenCV-Making a GUI concentrated line tool-", global variables were exchanged between the main routine and the callback function, but this time cv2. I tried to make it a little more elegant by using param of setMouseCallback (). As you can see in the animation GIF above, not only the cursor position but also the side of the rectangle is displayed (when deciding the last point, it is a closed rectangle), and you can also right-click to "go back one move". I've implemented it, and I'm happy that it's pretty nice code.

#### hack_image.py


import numpy as np
import cv2
import random

def draw_quadrangle(event, x, y, flags, param):
img = param["img"]
pts = param["pts"]
pic = param["pic"]

color = (random.randint(0,255), random.randint(0,255), random.randint(0,255))
img_tmp = img.copy()

if event == cv2.EVENT_MOUSEMOVE and len(pts) <= 3:
h, w = img.shape[:2]
cv2.line(img_tmp, (x,0), (x,h-1), color)
cv2.line(img_tmp, (0,y), (w-1,y), color)
cv2.imshow("image", img_tmp)

if event == cv2.EVENT_LBUTTONDOWN:
pts.append((x, y))
if len(pts) == 4:
h, w = img.shape[:2]
ph, pw = pic.shape[:2]
pts1 = np.float32(pts)
pts2 = np.float32([[0,0],[0,ph],[pw,ph],[pw,0]])

M1 = cv2.getPerspectiveTransform(pts1,pts2)
tmp = cv2.warpPerspective(img,M1,(pw,ph))
#cv2.imwrite("cut_image.jpg ", tmp)

M2 = cv2.getPerspectiveTransform(pts2,pts1)
transparence = (128,128,128)
front = cv2.warpPerspective(pic, M2, (w,h), borderValue=transparence)
img = np.where(front==transparence, img, front)
#cv2.imwrite("front.jpg ", front)
cv2.imshow("image", img)
#cv2.imwrite("image.jpg ", img)

if event == cv2.EVENT_RBUTTONDOWN and len(pts)>0:
del pts[-1]

if 0 < len(pts) <= 3:
for pos in pts:
cv2.circle(img_tmp, pos, 5, (0,0,255), -1)

cv2.line(img_tmp, pts[-1], (x,y), color, 1)
if len(pts)==3:
cv2.line(img_tmp, pts[0], (x,y), color, 1)

isClosed = True if len(pts)==4 else False
cv2.polylines(img_tmp, [np.array(pts)], isClosed, (0,0,255), 1)
cv2.imshow("image", img_tmp)

def main():
pts = []
cv2.imshow("image", img_origin)

cv2.waitKey(0)
cv2.destroyAllWindows()

if __name__ == "__main__":
main()


# gallery

### Fujiko F Fujio Museum

It is a well-known fact that the window frame of the Fujiko F Fujio Museum is designing the manuscript of Doraemon Episode 1 "From the Land of the Future", but check how faithfully it is reproduced. saw.

Exterior of the museum (from Google Street View)
Synthesis result

Well, it's not bad, but it's a shame that the first page is less reproducible.

Comparison

### An international conference that is noisy after being declared war by an evil organization

All your base are belong to us! The original image is here.

### Play Darius on the screen of an outdoor movie

The original image is here. I wanted to make "Play Road Runner with Sony Jumbotron at Tsukuba Science Expo", but I stopped because There was a similar story on Twitter.