Automate keyboard and mouse operations with python to streamline daily work [RPA]

What you can learn this time

Automat keyboard and mouse operations with python. Whether it's "Kan * re" or "Castle Pu *", the laps can be automated. Daily work can be streamlined without introducing RPA tools at a high license fee. (I thought I'd actually try to automate the laps, but I'm afraid of the rules, so I'll stop quietly) ~~ You can stay logged in even if you leave your seat by remote work, so you won't notice ... ~~

Get the size of the screen you want to operate

Speaking of packages for automating keyboard and mouse operations in python, pyautogui.

import pyautogui as pa
pa.size()

Size(width=1920, height=1200)

For ease of confirmation, use time to temporarily stop before operation. Since there are times when the screen processing and the browser cannot keep up with the reaction because the input of automatic operation is too fast, use time at your convenience.

Moving the mouse from the origin

Position the upper left corner of the screen as the coordinates (0,0). Use moveTo to move the mouse from (0,0) to the specified position. Specify how many seconds it takes to move with duration.

import time
time.sleep(1)

pa.moveTo(100,100,duration=1)

Know your current location

pa.position()

Point(x=1735, y=157)

Move from current position

MoveRel moves from the current position to the specified position

time.sleep(1)
pa.moveRel(100,100,duration=1)

Click / special click / double click

pa.click(x=1839, y=293, button='left')

pa.leftClick()
pa.rightClick()
pa.doubleClick()

Pushing / releasing the mouse

pa.mouseDown(x=1735, y=157)
pa.mouseUp()

You can drag and drop it by combining it with moveRel.

But there is a special one for drag and drop

pa.dragTo()
pa.dragRel()

screenshot

ss =pa.screenshot()

Recognizes the image and returns the location

Prepare what you want to recognize with screenshots and snipping tool

This time, save the trash can as gomi.png.

Returns the position that matches the image from the screen

list(pa.locateAllOnScreen('gomi.png'))

[Box(left=16, top=4, width=65, height=81)]

Returns the center position of what was recognized in the image

pa.center(pa.locateAllOnScreen('gomi.png'))

Once you recognize the center, you can execute the shortcut by combining it with a click.

Type in letters

Typewrite after specifying the position of the search bar or notepad in image recognition

pa.leftClick()
pa.typewrite('HHH',1)

Schedule execution

import schedule
import time

def job():
    print("Put click or processing in the function")

Perform job function operations every 10 minutes

schedule.every(10).minutes.do(job)

Other specifications

#Every hour
schedule.every().hour.do(job)
#10 daily:When it reaches 30
schedule.every().day.at("10:30").do(job)
#When it's monday
schedule.every().monday.do(job)
#Specify day and time
schedule.every().wednesday.at("13:15").do(job)

Stop schedule execution

schedule.run_pending()

that's all

You can also press and hold shift or press the windows button as a hotkey

PyAutoGUI’s documentation schedule doc