This article is a relay article of "2020 New Year Advent Calendar TechConnect!" of Link Information Systems. is. TechConnect! Is a self-starting Advent calendar that is relayed by a self-made group called engineer.hanzomon. (From Facebook here of Link Information Systems)

Since the theme was free, it will be a free article. (The technology used is serious.) I made full use of the tutorial and did my best to recognize the image.

What was made

A program to judge Win / Lose of Shadowverse (Steam version) has been created. ダウンロード.gif (The gif screen is drawn to confirm the Lose judgment. The judgment can be done without drawing.)

As you can see, it's just a comparison of the Win / Lose image and the capture screen. "Determine in real time whether a specified image appears in the capture of a specific application" I think it will be a reference for those who want to do something.

What I used

Python 3.7.4
pip 19.3.1
numpy 1.17.3
opencv-contrib-python 4.1.1.26
Pillow 6.2.0

Please refer to the reference article below for the installation method. I could just install it with pip from one end, so there was nothing particularly clogged up.

manner

Screen capture the game screen
Compare the captured image with the Win / Lose image
Cut off the feature points with Distance, and if the feature points above a certain level match, it is judged as Win / Lose.
If Win / Lose judgment is continuous more than a certain number of times, it is judged as Win screen / Lose screen.

Screen capture the game screen

I captured using PIL referring to the following article. [Python] [Windows] Screen capture with Python

from PIL import ImageGrab
import numpy as np

TARGET_NAME = 'Shadowverse'
handle = win32gui.FindWindow(None, TARGET_NAME)

while True:
    rect = win32gui.GetWindowRect(handle)
    img = ImageGrab.grab(rect)
    ocv_im = np.asarray(img)

    #Color conversion for OpenCV
    ocv_im = cv2.cvtColor(ocv_im, cv2.COLOR_BGR2RGB)

    #Draw on screen (for confirmation)
    cv2.imshow("images", ocv_im, )

    #Wait by an appropriate method (securing drawing time)
    cv2.waitKey(10)

Compare the captured image with the Win / Lose image

(Win / Lose image is cut out from the screenshot in advance.)

Refer to the following article and perform feature matching with AKAZE. Feature matching with OpenCV 3 and Python 3 (A-KAZE, KNN)

def MatchResultCheck(ocv_img):
    win_img = cv2.imread(WIN_IMAGE_PATH)
    lose_img = cv2.imread(LOSE_IMAGE_PATH)

    akaze = cv2.AKAZE_create()

    kp2, des2 = akaze.detectAndCompute(ocv_img, None)
    kp1_l, des1_l = akaze.detectAndCompute(lose_img, None)
    kp1_w, des1_w = akaze.detectAndCompute(win_img, None)

    is_win = False
    is_lose = False

    bf = cv2.BFMatcher()
    if not des2 is None :
        #Feature matching between Lose image and captured image
        matches_l = bf.knnMatch(des1_l,des2, k=2)
        #Feature matching between Win image and captured image
        matches_w = bf.knnMatch(des1_w,des2, k=2)

        #Lose judgment
        good_l = []
        for match1, match2 in matches_l:
            if match1.distance < 0.75*match2.distance:
                good_l.append([match1])
        #Win judgment
        good_w = []
        for match1, match2 in matches_w:
            if match1.distance < 0.75*match2.distance:
                good_w.append([match1])

        #Create an image for confirmation
        akaze_matches = cv2.drawMatchesKnn(lose_img,kp1_l,ocv_img,kp2,good_l,None,flags=2) 
        #Draw on screen (for confirmation)
        cv2.imshow("match", akaze_matches, )

        if len(good_l) > 20 :
            print("is lose")
            is_lose = True

        if len(good_w) > 20 :
            print("is win")
            is_win = True

    return is_win, is_lose

Win / Lose judgment

Since Distance (how much it matches) is stored in the matching result, cut off with Distance.

        #Lose judgment
        good_l = []
        for match1, match2 in matches_l:
            #Extract only feature points with a certain distance or more
            if match1.distance < 0.75*match2.distance:
                good_l.append([match1])

If the feature points remain above a certain level after cutting the foot, it is judged that the Win / Lose image exists.

        if len(good_l) > 20 :
            print("is lose")
            is_lose = True

Win screen / Lose screen judgment

In order to suppress false positives, the Win / Lose screen is judged only when it is judged as a Win / Lose image more than a certain number of times in a row. (Since I'm using python, I think I can write it more simply ...)

    is_win, is_lose = MatchResultCheck(ocv_im)

    if is_lose :
        cnt_lose_match += 1
        if cnt_lose_match >= 4:
            print("is lose match")
            cnt_lose_match = 0
            cv2.waitKey(1000)
    else :
        cnt_lose_match = 0

What I couldn't do

Let's judge the first attack / second attack in the same way! I thought I couldn't do it well ... Unlike Win / Lose, the judgment image is too small, and I wonder if the kanji is painful. We will continue to devote ourselves.

Summary

I was able to determine if a particular image was present in the screen capture. It would be interesting to be able to draw the judgment results in real time.

Next time is @ rysk001.

[PYTHON] Judging the victory or defeat of Shadowverse by image recognition