[PYTHON] How to save the feature point information of an image in a file and use it for matching

Introduction

There are many opportunities to use OpenCV feature point extraction and matching. When using multiple images, it often took time to extract feature points. Therefore, this time, I made a program that outputs the feature point information to a file and uses it for matching.

This time, as an example, let's find a girl with red glasses (target image) from the girls (12 source images)!

Source image ⬇︎

Target image ⬇︎

Development environment

This time, we developed in the following environment. OpenCV 4.1.2 Python 3.8.1

Constitution

Below is the flow of the program. I will explain step by step from the next section.

1. Get the feature point information of the source image and save it in a file
2. Get feature point information of target image
3. Get the feature point information of the source image from the file
4. Matching

1. Get the feature point information of the source image and save it in a file

First, get the feature point information of the source image (girls) in save_features.py and save it in a file. This time, I used AKAZE, a feature point descriptor implemented in OpenCV. To save the keypoint as a file, you need to access and list the cv :: KeyPoint.

save_features.py

#List keypoint
keypoint = []
for p in features[0]:
    temp = (p.pt, p.size, p.angle, p.response, p.octave, p.class_id)
    keypoint.append(temp)

In addition, this time, the feature point information was converted to byte type in order to reduce memory consumption.

save_features.py

#Convert keypoint to bytes
map(bytes, keypoints)

Below is the entire source code.

save_features.py

import cv2 as cv
import pickle

SOURCE_FILES = [
    "youngwoman_37.jpg ",
    "youngwoman_38.jpg ",
    "youngwoman_39.jpg ",
    "youngwoman_40.jpg ",
    "youngwoman_41.jpg ",
    "youngwoman_42.jpg ",
    "youngwoman_43.jpg ",
    "youngwoman_44.jpg ",
    "youngwoman_45.jpg ",
    "youngwoman_46.jpg ",
    "youngwoman_47.jpg ",
    "youngwoman_48.jpg ",
]


def get_features(img_file_name):
    """Get features of master images

        Args:
            img_file_name(list): Master image

        Returns:
            keypoints, descriptors, img

    """
    img = cv.imread("images/" + img_file_name)

    #Feature point information extraction
    akaze = cv.AKAZE_create()
    kp, des = akaze.detectAndCompute(img, None)

    features = [kp, des]

    return features


sources = {}
for item in SOURCE_FILES:
    features = get_features(item)
    #List keypoint
    keypoints = []
    for p in features[0]:
        temp = (p.pt, p.size, p.angle, p.response, p.octave, p.class_id)
        keypoints.append(temp)

    #Convert keypoints to bytes
    map(bytes, keypoints)
    #Dictionary of feature point information
    sources[item] = {
        "src": item,
        "keypoint": keypoints,
        "descriptor": features[1],
    }

#Write feature point information to a file
with open("sources_data.pickle", mode="wb") as f:
    pickle.dump(sources, f)

2. Get feature point information of target image

From step 2, get_features_from_file.py is used for processing. As with the source image, feature point information is acquired using AKAZE, which is a feature point descriptor.

get_features_from_file.py

#Load target image
target_img = cv.imread("images/target_girl.jpg ")
#Feature acquisition
akaze = cv.AKAZE_create()
target_keypoint, target_descriptor = akaze.detectAndCompute(target_img, None)

3. Get the feature point information of the source image from the file

Get feature point information from the file with get_sources (). Since keypoints was converted to bytes in order to make it pickle, it is converted to list and returned to the original structure.

get_features_from_file.py

def get_sources():
    """Get source's features from file

        Returns:
            sources(list): source's keypoints, descriptors,and img
        """
    #Get feature point information from a file
    with open("sources_data.pickle", mode="rb") as f:
        sources = pickle.load(f)

    for n in sources:
        items = sources[n]
        #Change keypoints from bytes to list
        list(map(list, items["keypoint"]))
        #Restore keypoints to original structure
        keypoints = []
        for p in items["keypoint"]:
            temp = cv.KeyPoint(
                x=p[0][0],
                y=p[0][1],
                _size=p[1],
                _angle=p[2],
                _response=p[3],
                _octave=p[4],
                _class_id=p[5],
            )
            keypoints.append(temp)
        items["keypoint"] = keypoints

    return sources

4. Matching

Matches with the feature point information of the target image for each source image. Data is thinned out, and if the number of matched feature points is equal to or greater than the set threshold (set to 20 this time), matching is successful.

get_features_from_file.py

for n in sources:
    source = sources[n]
    source_img = cv.imread("images/" + source["src"])
    matches = matcher.knnMatch(source["descriptor"], target_des, k=2)
    #Thin out data
    ratio = 0.5
    matched_keypoints = []
    for m, n in matches:
        if m.distance < ratio * n.distance:
            matched_keypoints.append([m])

    #Output the result image when there are more good than any threshold
    if len(matched_keypoints) > 20:
        out = cv.drawMatchesKnn(
            source_img,
            source["keypoint"],
            target_img,
            target_kp,
            matched_keypoints,
            None,
            flags=2,
        )

Below is the entire source code.

get_features_from_file.py

import cv2 as cv
import pickle


def get_sources():
    """Get source's features from file

        Returns:
            sources(list): source's keypoints, descriptors,and img
        """
    #Get feature point information from a file
    with open("sources_data.pickle", mode="rb") as f:
        sources = pickle.load(f)

    for n in sources:
        items = sources[n]
        #Change keypoints from bytes to list
        list(map(list, items["keypoint"]))
        #Restore keypoints to original structure
        keypoints = []
        for p in items["keypoint"]:
            temp = cv.KeyPoint(
                x=p[0][0],
                y=p[0][1],
                _size=p[1],
                _angle=p[2],
                _response=p[3],
                _octave=p[4],
                _class_id=p[5],
            )
            keypoints.append(temp)
        items["keypoint"] = keypoints

    return sources


matcher = cv.BFMatcher()

#Load target image
target_img = cv.imread("images/target_girl.jpg ")
#Feature acquisition
akaze = cv.AKAZE_create()
target_kp, target_des = akaze.detectAndCompute(target_img, None)
#Read the feature point information of the source image from the file
sources = get_sources()

for n in sources:
    source = sources[n]
    source_img = cv.imread("images/" + source["src"])
    matches = matcher.knnMatch(source["descriptor"], target_des, k=2)
    #Thin out data
    ratio = 0.5
    matched_keypoints = []
    for m, n in matches:
        if m.distance < ratio * n.distance:
            matched_keypoints.append([m])

    #Output the result image when there are more good than any threshold
    if len(matched_keypoints) > 20:
        out = cv.drawMatchesKnn(
            source_img,
            source["keypoint"],
            target_img,
            target_kp,
            matched_keypoints,
            None,
            flags=2,
        )

cv.imwrite("images/result.jpg ", out)
cv.waitKey()

result

In the result image below, the feature points matched between the source image and the target image are drawn. I was able to find the target girl safely!

At the end

The key to saving feature points in a file is Accessing cv :: KeyPoint Make a list based on the accessed information is. If you have a chance to process images with OpenCV, please give it a try.

Reference page

Export KeyPoint of OpenCV for Python3 to a file

[Feature point matching with python3, opencv3 (AKAZE, KNN)] (https://techtech-sorae.com/python3opencv3%E3%81%A7%E7%89%B9%E5%BE%B4%E7%82%B9%E3%83%9E%E3%83%83%E3%83%81%E3%83%B3%E3%82%B0akaze-knn/)