Automate sushi making with Python

• This is my first time using qiita. Please note that there may be some points that cannot be reached. The code is also messed up and dirty, but I'll fix it when I feel like it ()

Hello. Suddenly everyone Sushi hitting Are you doing? Most people who have practiced typing should have played this game. Sushi hitting is a browser game. So, I thought that it would be easy to automate if I got HTML with Python, so I decided to make it.

I was trying to get the html Actually, sushi making uses a technology called OpenGL, It seems that the characters are displayed as an image. Therefore, it is a policy change. Let's recognize the image.

First, press the start button and the recommended button.

import pyautogui
import time
x,y=pyautogui.locateCenterOnScreen("susida.png ")
print(x)
print(y)
pyautogui.click(x, y+100)
print("I pressed the start button.")
time.sleep(1)
pyautogui.click(x, y+100)
print("I pressed the recommendation.")
time.sleep(1)
pyautogui.typewrite(" ")
print("start.")
time.sleep(2)

If you specify it in absolute coordinates, it will change depending on the window size etc. I decided to use the relative coordinates from susida.png. ←susida.png Then click on that coordinate with pyautogui. Installation of pyautogui

pip install pyautogui

is.

Next is the image recognition part.

import pyocr
import pyocr.builders
from PIL import Image
import sys
import cv2

i=1
result=0
tools=pyocr.get_available_tools()

if len(tools) == 0:
    print("The OCR tool was not found.")
    sys.exit(1)

tool=tools[0]

while(i<201):
    sc=pyautogui.screenshot(region=(x-110, y+85, 210, 25))

    sc.save("original.png ")

    original=cv2.imread("original.png ",0)
    threshold=100

    ret, img_thresh = cv2.threshold(original, threshold, 255, cv2.THRESH_BINARY)
    cv2.imwrite("gray.png ", img_thresh)

    hanten = cv2.imread("gray.png ")
    hanten2=cv2.bitwise_not(hanten)
    cv2.imwrite("sushida_sc.png ", hanten2)


    img_org=Image.open("sushida_sc.png ")

    builder = pyocr.builders.TextBuilder()
    tmp=result
    result=tool.image_to_string(img_org, lang="eng", builder=builder)
    if(tmp != result):
        print(i,"Dish:",result)
        pyautogui.typewrite(result)
    i=i+1
    time.sleep(0.3)

Pyocr is used for image recognition. Please see here for the introduction method ↓ https://gammasoft.jp/blog/ocr-by-python/ First of all, I usually take a screenshot of the Roman alphabet. However, this makes the background transparent and reduces the recognition accuracy. So, it will be binarized. Then invert it to make the text black. If you type in what you recognize with pyautogui, the automation of sushi making is complete. I stopped at 200 because if I eat too much sushi, I get an error. The full text is below.

import pyautogui
import time
import pyocr
import pyocr.builders
from PIL import Image
import sys
import cv2

i=1
result=0
tools=pyocr.get_available_tools()

if len(tools) == 0:
    print("The OCR tool was not found.")
    sys.exit(1)

tool=tools[0]

x,y=pyautogui.locateCenterOnScreen("susida.png ")
print(x)
print(y)
pyautogui.click(x, y+100)
print("I pressed the start button.")
time.sleep(1)
pyautogui.click(x, y+100)
print("I pressed the recommendation.")
time.sleep(1)
pyautogui.typewrite(" ")
print("start.")
time.sleep(2)
while(i<201):
    sc=pyautogui.screenshot(region=(x-110, y+85, 210, 25))

    sc.save("original.png ")

    original=cv2.imread("original.png ",0)
    threshold=100

    ret, img_thresh = cv2.threshold(original, threshold, 255, cv2.THRESH_BINARY)
    cv2.imwrite("gray.png ", img_thresh)

    hanten = cv2.imread("gray.png ")
    hanten2=cv2.bitwise_not(hanten)
    cv2.imwrite("sushida_sc.png ", hanten2)


    img_org=Image.open("sushida_sc.png ")

    builder = pyocr.builders.TextBuilder()
    tmp=result
    result=tool.image_to_string(img_org, lang="eng", builder=builder)
    if(tmp != result):
        print(i,"Dish:",result)
        pyautogui.typewrite(result)
    i=i+1
    time.sleep(0.3)

YouTube:https://www.youtube.com/watch?v=kZ0MKTQXgj0