Aligning scanned images of animated video paper using OpenCV and Python

Using OpenCV and Python, I made a tool that detects the alignment holes of animation video paper as shown below and removes the misalignment of the scanned image. A practical article for beginners in OpenCV and Python. See here for OpenCV

The problem you want to solve

When creating an animation, it is easy to use 2D keyframe application like 9VAe Kyubee, but if you draw one by one on video paper, it will be several hundred. It will be a lot more than one. If this is input to a personal computer with a scanner with an auto feeder, the position of several dots will inevitably shift, and the image will be blurred when played back.

Therefore, let's create a tool "Peascan.py" that detects holes on the video paper and corrects the misalignment by image processing.

Hole recognition, image rotation, and misalignment correction should be easy with OpenCV.

1. 1. Install OpenCV and Python

--I decided to use Python (x, y), which is easy to install, so that anyone can use it. You can install OpenCV and Python at the same time.

It installs Python 2.7.10, OpenCV 2.4.12, which is a little earlier, but it works fine.

2. Load and display images

2-1. Creating a Python program

Let's create a simple Python + OpenCV program.

Save the following text in "UTF-8" with the name "test.py". If Japanese is included in the path, it may not work, so it is a good idea to create a folder such as c: \ pytest and save it. xx Sample image In place of xx, prepare an appropriate image file and enter the path name (c: /pytest/test.jpg, etc.). Japanese names cannot be used. The path delimiter is "/".

import numpy as np
import cv2
img = cv2.imread('C:/xx sample image xx.jpg')
print img.shape
print img.shape[0], img.shape[1]
cv2.imshow('Title',img)
cv2.waitKey(5000)

Each has the following meanings.

2-2. Program execution

1. Open the folder containing "test.py"
2. Click the right button where there is nothing> click "Open IPython console here"
   This will open the terminal for Python.
3. Type python test.py and press Enter
4. If the image file is specified correctly, the image will be displayed for 5 seconds. If "None" or "Attribute Error" is displayed, (1) Japanese is not included in the location or file name of the image, and (2) the path delimiter is "" ("/". (Must be), (3) Make sure that the file names are correct.

3. 3. Find the center of gravity by binarizing the mark (hole) in the upper left of the image

After displaying the image, rewrite it as follows from the line following "img =" in "test.py". Assuming that there is a mark in the upper left (0,0)-(200,200) range of the image, find the center of gravity. Adjust the values of frmX, toX, frmY, toY according to the size of the image.

import numpy as np
import cv2
img = cv2.imread('C:/xx sample image xx.jpg')

frmX,toX = 0,200 #Range of marks (holes)
frmY,toY = 0,200 #Range of marks (holes)
mark = img[frmY:toY, frmX:toX] #Partial image
gray = cv2.cvtColor(mark, cv2.COLOR_BGR2GRAY) #Monochrome
ret, bin = cv2.threshold(gray, 127, 255, 0) #Binarization
cv2.imshow('out',bin) #Mark range
cv2.waitKey(1000) #Stop for 1 second
contours, hierarchy = cv2.findContours(bin, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) #Extraction of contour
cnt = contours[0] #First contour
M = cv2.moments(cnt) #Moment
cx = int(M['m10']/M['m00']) #Center of gravity X
cy = int(M['m01']/M['m00']) #Center of gravity Y
cv2.circle(img,(cx,cy), 10, (0,0,255), -1)
print cx,cy
cv2.imshow('Title',img)
cv2.waitKey(5000) #5 seconds display

4. Find the center of gravity of the marks (holes) in the upper left and upper right, and transform them to match the center of gravity of the reference image.

Let's rewrite "test.py" as follows. Detects the center of gravity of the mark (hole) in the range of 200x200 in the upper left and upper right of the image, and transforms it to match the position of the reference image.

import numpy as np
import cv2
img = cv2.imread('C:/xx sample image xx.jpg')

frmX,toX = 0,200 #Range of marks (holes)
frmY,toY = 0,200 #Range of marks (holes)

def searchMark(img, left): #Function to find a mark (hole) left==1 is left
	if left==1: #Find the mark (hole) on the left
		mark = img[frmY:toY, frmX:toX]
	else: #Find the mark (hole) on the right
		mark = img[frmY:toY, img.shape[1]-toX:img.shape[1]-frmX]
	gray = cv2.cvtColor(mark, cv2.COLOR_BGR2GRAY) #Monochrome
	ret, bin = cv2.threshold(gray, 127, 255, 0) #Binarization
	cv2.imshow('out',bin) #Display the range of marks (holes)
	cv2.waitKey(1000) #Stop for 1 second
	contours, hierarchy = cv2.findContours(bin, cv2.RETR_TREE, 	cv2.CHAIN_APPROX_SIMPLE) #Extraction of contour
	ax = ay = sum = 0. #Accumulation of the entire center of gravity
	for cnt in contours: #Find the center of gravity of the whole
		M = cv2.moments(cnt)
		ax += M['m10']
		ay += M['m01']
		sum += cv2.contourArea(cnt)
	if left==1:
		cx = ax/sum+frmX
		cy = ay/sum+frmY
	else:
		cx = ax/sum + img.shape[1]-toX
		cy = ay/sum + frmY
	cv2.circle(img,(int(cx),int(cy)), 10, (0,0,255), -1) #Draw a red circle at the center of gravity
	print cx,cy #Display the calculated center of gravity
	return cx,cy #Function return value

#Affine transformation test
cx0,cy0 = searchMark(img,1) #Center of gravity (reference) of the mark (hole) on the upper left
dx0,dy0 = searchMark(img,0) #Center of gravity (reference) of the mark (hole) on the upper right
cx1,cy1 = cx0,cy0
dx1,dy1 = dx0,dy0+10 #Suppose that the mark (hole) on the upper right is shifted downward by 10 dots.
cv2.circle(img,(int(dx1),int(dy1)), 10, (255,0,0), -1) #Draw a blue circle at the offset point
pts2 = np.float32([[cx0,cy0],[dx0,dy0],[cx0-(dy0-cy0),cy0+(dx0-cx0)]])
pts1 = np.float32([[cx1,cy1],[dx1,dy1],[cx1-(dy1-cy1),cy1+(dx1-cx1)]])
height,width,ch = img.shape
M = cv2.getAffineTransform(pts1,pts2)
dst = cv2.warpAffine(img,M,(width,height))
cv2.imshow('Title',img) #Display the image before conversion
cv2.waitKey(5000) #5 seconds display
cv2.imshow('Title',dst) #Display the converted image
cv2.waitKey(5000) #5 seconds display

When executed, the image will be converted so that the intentionally shifted blue circle overlaps the original red circle. This completes the mark (hole) detection and conversion process.

5. Read the images in the folder and output the aligned images to another folder

Now that the basic processing is complete, let's use it as follows. With this, even if there are hundreds of images in the folder, you can easily convert them. This tool is named "peascan.py". (Abbreviation for Position Error correction After SCAN)

Alignment mark (hole) range frmX, toX =, frmY, toY = Adjust the numbers after that according to the actual image.

peascan.py

import numpy as np
import cv2
import sys #Get argv
import os  #File operations

argv = sys.argv  #Get a list of command line arguments
argc = len(argv) #Number of arguments
if argc == 2:     #Check if it is a folder
	if os.path.isdir(argv[1]) != True: #If not a folder
		argc = -1       #Make an error
if argc != 2:     #Show usage
	print 'Usage: Drag Image folder onto this icon.'
	key = raw_input('Hit Enter')
	quit()        #End

#Range to check the alignment mark (hole) ★ Adjust as appropriate ★
frmX,toX = 10,200 #10 from the horizontal edge-200 dots (symmetrical)
frmY,toY = 10,200 #10 from the top-200 dots

def searchMark(img, left): #Function to find a mark (hole) left==1 is left
	if left==1: #Find the mark (hole) on the left
		mark = img[frmY:toY, frmX:toX]
	else: #Find the mark (hole) on the right
		mark = img[frmY:toY, img.shape[1]-toX:img.shape[1]-frmX]
	gray = cv2.cvtColor(mark, cv2.COLOR_BGR2GRAY) #Monochrome
	ret, bin = cv2.threshold(gray, 127, 255, 0) #Binarization
	cv2.imshow('out',bin) #Display the range of marks (holes)
	cv2.waitKey(1000) #Stop for 1 second
	contours, hierarchy = cv2.findContours(bin, cv2.RETR_TREE, 	cv2.CHAIN_APPROX_SIMPLE) #Extraction of contour
	ax = ay = sum = 0. #Accumulation of the entire center of gravity
	for cnt in contours: #Find the center of gravity of the whole
		M = cv2.moments(cnt)
		ax += M['m10']
		ay += M['m01']
		sum += cv2.contourArea(cnt)
	if left==1:
		cx = ax/sum+frmX
		cy = ay/sum+frmY
	else:
		cx = ax/sum + img.shape[1]-toX
		cy = ay/sum + frmY
	cv2.circle(img,(int(cx),int(cy)), 10, (0,0,255), -1) #Draw a red circle at the center of gravity
	print cx,cy #Display the calculated center of gravity
	return cx,cy #Function return value

#Main loop
inpFolder = argv[1]                    #Input image folder
parent = os.path.dirname(argv[1])
outFolder = os.path.join(parent,'out') #Output image folder
if os.path.exists(outFolder) != True:  #If it does not exist
	os.mkdir(outFolder)            #Create output folder
files = [f for f in os.listdir(inpFolder)] #Input image
files.sort(key=os.path.basename)       #Sort by file name
cx0 = -1
for fn in files:
	img = cv2.imread(os.path.join(inpFolder,fn))
	if img is None:    #I couldn't read it, so next
		continue    
	if cx0 == -1: #Remember the first image as it is
		cx0,cy0=searchMark(img,1)
		dx0,dy0=searchMark(img,0)
		cv2.imwrite(os.path.join(outFolder,fn), img)
	else: #The second and subsequent images are affine-transformed to match the first image
		cx1,cy1=searchMark(img,1)
		dx1,dy1=searchMark(img,0)
		pts2 = np.float32([[cx0,cy0],[dx0,dy0],[cx0-(dy0-cy0),cy0+(dx0-cx0)]])
		pts1 = np.float32([[cx1,cy1],[dx1,dy1],[cx1-(dy1-cy1),cy1+(dx1-cx1)]])
		height,width,ch = img.shape
		M = cv2.getAffineTransform(pts1,pts2)
		dst = cv2.warpAffine(img,M,(width,height))
		cv2.imwrite(os.path.join(outFolder,fn), dst) #Writing image
cv2.imshow('Title',dst) #Show last image
cv2.waitKey(5000) #5 seconds display

――The tool created this time only calculates the center of gravity of the marks in the upper left and upper right of the image, so the shape is completely irrelevant. Even if it is a cross mark like a dragonfly, it can be used for alignment if all the images have the same shape. --The program is written in Python, so you can easily modify it. Please use all means.

6. Note the difference between OpenCV 3 and 2.4

Most of the OpenCV information on the net is from OpenCV 3, and OpenCV 2.4 sometimes gave an error.

Related article

1. Let's make an animation with free software (9VAe Kyubee) -Scratch, biscuits, then 9VAe = 3rd programming learning environment
2. How to make stroke order animation
3. 9VAe Kyubee: How to make a long animation -Create APNG for moving LINE stamps: Free software 9VAe Kyubee -GIF animation that can be made without drawing a picture with [9VAe Kyubee] -SVG animation created with 9VAe Kyubee -How to customize animation software 9VAe -Modify 9VAe for Kids Plaza Osaka -9VAe / 9svg data format explanation -Aligning scanned images of animated video paper using OpenCV and Python