[PYTHON] Motion template analysis using OpenCV (recognizing an object and its moving direction in real time)

Introduction

"Motion Template" is an efficient method of motion extraction developed by MIT Media Lab (Link 1, [Link] 2](http://alumni.media.mit.edu/~jdavis/Publications/motiontemplategradient.pdf)). It is suitable for real-time calculation with a small amount of calculation. It has a wide range of applications, and can visually display the movements of players, the trajectory of balls, bats, clubs, and rackets through gesture recognition and sports broadcasting. This time, I will try motion template analysis using OpenCV 3.

OpenCV OpenCV (Open Source Computer Vision Library) is a collection of BSD-licensed video / image processing libraries. There are many algorithms for image filtering, template matching, object recognition, video analysis, machine learning, and more.

■ Example of motion tracking using OpenCV (OpenCV Google Summer of Code 2015) https://www.youtube.com/watch?v=OUbUFn71S4s

■ Click here for installation and easy usage Install OpenCV 3 (core + contrib) in Python 3 environment & Difference between OpenCV 2 and OpenCV 3 & simple operation check ★ Please install core + opencv_contrib to run the motion template.

■ Click here for still image filtering Try edge detection with OpenCV Perform various filters with OpenCV (gradient, high-pass, Laplacian, Gaussian) Extract feature points with OpenCV (AgastFeature, FAST, GFTT, MSER, AKAZE, BRISK, KAZE, ORB, SimpleBlob)

■ Click here for processing video files Try converting videos in real time with OpenCV Try converting webcam / camcorder video in real time with OpenCV Draw optical flow in real time with OpenCV (Shi-Tomasi method, Lucas-Kanade method) Object tracking using OpenCV (tracking feature points specified by mouse by Lucas-Kanade method

easy explanation

"Motion template" can also be easily realized by using OpenCV. I will briefly summarize the OpenCV methods used this time.

This time, we will create a program that uses all of (a), (b), and (c), but it is also possible to select only the necessary ones. For example, if you want only motion images, you don't need (b) (c).

The main functions of this program are as follows. --Detects motion and displays it in grayscale. --Emphasize new motions and gradually dimming old motions. --The direction of motion at each coordinate is displayed as a green line. --The direction of motion on the entire screen is displayed as a yellow line.

program

• Operating environment

  • python: 3.5.1
  • OpenCV(core + contrib): 3.1.0

• Video data The program uses the sample video that comes with OpenCV. 　　OpenCV\opencv\sources\samples\data\768x576.avi

motion.py

import time
import math
import cv2
import numpy as np

#Video data
VIDEO_DATA = "768x576.avi"
#Esc key
ESC_KEY = 0x1b
#Remaining period of motion(sec)
DURATION = 1.0
#The length of the line that displays the overall direction
LINE_LENGTH_ALL = 60
#The length of the line that displays the direction for each coordinate
LINE_LENGTH_GRID = 20
#Interval to calculate the direction for each coordinate
GRID_WIDTH = 40
#Radius of the circle of the line indicating the direction
CIRCLE_RADIUS = 2

#Display window initialization
cv2.namedWindow("motion")
#Loading video data
video = cv2.VideoCapture(VIDEO_DATA)

#Loading the first frame
end_flag, frame_next = video.read()
height, width, channels = frame_next.shape
motion_history = np.zeros((height, width), np.float32)
frame_pre = frame_next.copy()

while(end_flag):
    #Difference calculation between frames
    color_diff = cv2.absdiff(frame_next, frame_pre)

    #Grayscale conversion
    gray_diff = cv2.cvtColor(color_diff, cv2.COLOR_BGR2GRAY)

    #Binarization
    retval, black_diff = cv2.threshold(gray_diff, 30, 1, cv2.THRESH_BINARY)

    #Processor processing time(sec)Get
    proc_time = time.clock()

    #Update motion history image
    cv2.motempl.updateMotionHistory(black_diff, motion_history, proc_time, DURATION)

    #Dimming the display of old motions over time
    hist_color = np.array(np.clip((motion_history - (proc_time - DURATION)) / DURATION, 0, 1) * 255, np.uint8)

    #Grayscale conversion
    hist_gray = cv2.cvtColor(hist_color, cv2.COLOR_GRAY2BGR)

    #Calculation of change direction of motion history image
    #* In orientation, the value (deg) of the changing direction is stored for each coordinate.
    mask, orientation = cv2.motempl.calcMotionGradient(motion_history, 0.25, 0.05, apertureSize = 5)

    #Draw the movement of each coordinate with a green line
    width_i = GRID_WIDTH
    while width_i < width:
        height_i = GRID_WIDTH
        while height_i < height:
            cv2.circle(hist_gray, \
                       (width_i, height_i), \
                       CIRCLE_RADIUS, \
                       (0, 255, 0), \
                       2, \
                       16, \
                       0)
            angle_deg = orientation[height_i - 1][width_i - 1]
            if angle_deg > 0:
                angle_rad = math.radians(angle_deg)
                cv2.line(hist_gray, \
                         (width_i, height_i), \
                         (int(width_i + math.cos(angle_rad) * LINE_LENGTH_GRID), int(height_i + math.sin(angle_rad) * LINE_LENGTH_GRID)), \
                         (0, 255, 0), \
                         2, \
                         16, \
                         0)
            
            height_i += GRID_WIDTH
        
        width_i += GRID_WIDTH


    #Calculate overall motion direction
    angle_deg = cv2.motempl.calcGlobalOrientation(orientation, mask, motion_history, proc_time, DURATION)

    #Draw the whole movement with a yellow line
    cv2.circle(hist_gray, \
               (int(width / 2), int(height / 2)), \
               CIRCLE_RADIUS, \
               (0, 215, 255), \
               2, \
               16, \
               0)
    angle_rad = math.radians(angle_deg)
    cv2.line(hist_gray, \
             (int(width / 2), int(height / 2)), \
             (int(width / 2 + math.cos(angle_rad) * LINE_LENGTH_ALL), int(height / 2 + math.sin(angle_rad) * LINE_LENGTH_ALL)), \
             (0, 215, 255), \
             2, \
             16, \
             0)

    #Display motion image
    cv2.imshow("motion", hist_gray)

    #Finish by pressing the Esc key
    if cv2.waitKey(20) == ESC_KEY:
        break

    #Load the next frame
    frame_pre = frame_next.copy()
    end_flag, frame_next = video.read()

#End processing
cv2.destroyAllWindows()
video.release()

Execution result

--The walking figure is displayed in the motion template. --The green line shows which direction each grid is moving. ――The yellow line shows which direction the screen is moving in.

It is a little difficult to understand the overall direction, but if it is a fixed frame, it shows which direction the object on the screen is moving as a whole. In cases where the entire frame is moving, it represents the direction in which the frame is moving.

Motion analysis using direction information

As an example of how to analyze movement using a motion template, let's analyze the moment of Kei Nishikori's shot. Performs analysis using the direction information calculated from the motion template information. Let's run the motion template program on the original image. In the motion image, you can see the trajectory of the ball for each frame (motion image 1, motion image 2). Counting the number of balls, the Motion 2 image is 3 frames ahead of the Motion 1 image.

By analyzing the direction of each grid for motion image 1 and motion image 2, the following movements can be read.

--The right shoulder, which was stopped in the Motion 1 image, is moving to the upper right in the Motion 2 image. --The right calf, which was stopped in the Motion 1 image, is moving downward in the Motion 2 image. --The left knee, which was stopped in the motion 1 image, is moving upward in the motion 2 image. --The waist that was moving backward in the Motion 1 image is moving to the upper right in the Motion 2 image. ――From the shin of the left foot, the center of the left and right shoulder blades, and the head are stopped in both Motion 1 and Motion 2 images. ――Overall, the left half of the body is fixed as the axis, and the right half of the body is stretched to hit the ball back.

When you watch it on TV, it looks like you are jumping with your whole body and hitting the ball, but at the moment you actually hit the ball in this video, after making a solid axis with the left half of the body, use the spring of the right half of the body You can see that you are hitting.

** Original image **

** Motion 1 image **

** Motion 2 image **

For those who play sports, it would be interesting to analyze the movements of professionals and the difference between the movements when they succeed and when they fail.