Track baseball balls with Python + OpenCV
When I saw the OpenCV site for the first time in a while, it was new, so I touched it for a moment.
https://opencv.org/
I used to track baseball balls in Java a long time ago, so this time I tried it in Python. Tracking is a combination of background subtraction and template matching.
Environment
The environment is as follows.
- Python : 3.8.0
- OpentCV : 4.2.0
I am creating an environment with venv. It doesn't matter if it's not venv.
$ python -m venv pythonOpevCV
$ cd pythonOpenCV
$ Script\activate
$ python -m pip install --upgrade pip
$ pip install opencv-python
Successfully installed numpy-1.18.2 opencv-python-4.2.0.34
numpy will also be installed.
What to prepare
Please prepare the following items.
--Video showing a baseball ball --Cut image of baseball ball (template image)
The image of the cut out ball should look like the one below. Below is a sample.
procedure
Follow the steps below to track.
- Save as an image for each frame of the video
- Grayscale frame and template images
- Binarize frame image and template image
- Perform background subtraction between the previous and next frame images
- Perform template matching from the image resulting from background subtraction.
- Draw tracking results
Even if template matching is performed on the original frame image suddenly, clouds and buildings in the background will be detected by mistake as a ball, and the accuracy is not good, so grayscale → binarization will make a black and white image. If background subtraction is performed on the previous and next frames in the black-and-white image state, the clouds and background will hardly move in about 0.1 seconds, so the moving ball can be clearly detected.
The grayscale image looks like this:
Detects balls in the area surrounded by the red frame. This image is binarized into a black and white image.
The ball can be detected in white, but only the ball cannot be detected because the background and the ball are the same white. Here, the background subtraction between the previous and next frame images is as follows.
The white background is not working and is not detected by background subtraction, only the ball and other noise are detected in white. In this state, template matching is performed with the binarized image of the following template image.
This will allow the ball to be detected without being affected by luminosity or background.
Source code
The source code is as follows. Place the target video in VIDEOPATH and the template image in TEMPLATEPATH.
main.py
import glob
import re
import cv2
VIDEOPATH = "media/video/video.mp4"
IMAGEPATH = "media/image/"
TEMPLATEPATH = "template.jpeg "
def save_frames(video_path, image_dir):
"""
Extract frame images from videos
"""
cap = cv2.VideoCapture(video_path)
digit = len(str(int(cap.get(cv2.CAP_PROP_FRAME_COUNT))))
n = 0
while True:
ret, frame = cap.read()
if ret:
cv2.imwrite("{}original/frame_{}.{}".format(IMAGEPATH, n, "jpeg"), frame)
n += 1
else:
return
def do_grayscale(image_path):
"""
Grayscale the image
"""
img = cv2.imread(image_path)
gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)
save_image(image_path, "gray", gray)
def do_binarization(image_path):
"""
Image binarized
"""
img = cv2.imread(image_path)
ret, img_thresh = cv2.threshold(img, 100, 255, cv2.THRESH_BINARY)
save_image(image_path, "binary", img_thresh)
def do_backgroundsub():
"""
Perform background subtraction
"""
img_list = glob.glob(IMAGEPATH + "binary/frame*.jpeg ")
num = lambda val: int(re.sub("\D","",val))
sorted(img_list,key=(num))
source = img_list[0]
for path in img_list:
diff = cv2.absdiff(cv2.imread(source),cv2.imread(path))
source = path
save_image(path, "bgsub", diff)
def do_template_matching():
"""
Perform template matching between template image and frame image
"""
template_img = cv2.imread(IMAGEPATH + "binary/" + TEMPLATEPATH)
img_list = glob.glob(IMAGEPATH + "bgsub/frame*.jpeg ")
num = lambda val: int(re.sub("\D","",val))
sorted(img_list,key=(num))
location_list = []
for path in img_list:
result = cv2.matchTemplate(cv2.imread(path), template_img, cv2.TM_CCOEFF)
minVal, maxVal, minLoc, maxLoc = cv2.minMaxLoc(result)
location_list.append(maxLoc)
return location_list
def draw_rectangle(location_list):
"""
Draw the matching result on the image
"""
source = cv2.imread(IMAGEPATH + "original/frame_0.jpeg ")
cv2.imwrite(IMAGEPATH + "result.jpeg ",source)
source = cv2.imread(IMAGEPATH + "result.jpeg ")
for loc in location_list:
lx, ly, rx, ry = loc[0] - 10, loc[1] - 10, loc[0] + 10, loc[1] + 10
img = cv2.rectangle(source, (lx, ly), (rx, ry), (0, 255, 0), 3)
cv2.imwrite(IMAGEPATH + "result.jpeg ",img)
def save_image(img_path, dir, img):
"""
Save the image
img_path :Image path
dir :Directory name
img :image data
"""
file_name = img_path.replace("\\","/").split(".")[0].split("/")[-1]
cv2.imwrite("{}{}/{}.{}".format(IMAGEPATH, dir, file_name,"jpeg"), img)
if __name__=="__main__":
#① Divide the video into frames
save_frames(VIDEOPATH,IMAGEPATH)
#(2) Grayscale the template image and frame image
do_grayscale(IMAGEPATH + TEMPLATEPATH)
for path in glob.glob(IMAGEPATH + "original/*.jpeg "):
do_grayscale(path)
#③ Binarization of template image and frame image
for path in glob.glob(IMAGEPATH + "gray/*.jpeg "):
do_binarization(path)
#④ Perform background subtraction
do_backgroundsub()
#⑤ Perform template matching
location_list = do_template_matching()
#⑥ Project the matched coordinates
draw_rectangle(location_list)
result
I tried to detect the ball in the pitching video. Although it can be detected in general, it also detects parts other than the trajectory of the ball.
There was a way to correct noise and outliers, but I forgot, so I remember it again.
Summary
Baseball ball tracking was performed using background subtraction and template matching in Python + OpenCV.
If you use YOLO, it seems that you can detect a baseball ball in the image, so I would like to try this area as well.
Recommended Posts