[Automation] Manipulate mouse and keyboard with Python

When do you want to automate mouse / keyboard operations?

With Python, it is relatively easy to directly operate "files" such as Excel and "apps" such as web browsers.

So, do you ever want to automate mouse operations and keyboard input? You may think that.

There are many apps and software in the world. If you include free software, there are as many stars as there are. As expected, Python does not support all of those apps. There are many apps ** that cannot be operated directly with Python.

If you want to automate the work with such an app, you can do so by ** automating mouse and keyboard operations **.

In the first place, when humans operate apps, they operate with the mouse and keyboard, so if you can operate the mouse and keyboard with Python, you should be able to operate anything automatically in theory (probably).

So, in this article, I'll show you how to operate the mouse and keyboard in Python.

Automate with PyAuto GUI

To operate the mouse and keyboard in Python, use a library called ** PyAutoGUI **.

pyautogui should not be installed by default in Anaconda, so let's install it with the pip command first.

pip install pyautogui

If you can install it successfully, import it.

python

import pyautogui

Mouse operation

click

Left-click the mouse with pyautogui.click ().

At this time, the question is where to click, but you can specify the click position with the following three patterns.

I will explain in detail.

pyautogui.click()

Click the current mouse cursor position. In this case, use the commands pyautogui.moveTo () and pyautogui.move () to move the mouse cursor, which will be described later, to first move the mouse cursor to the desired position and then pyautogui.click (). I think it will be.

python

# (100, 200)Move the mouse cursor to the position of
pyautogui.moveTo(100,200)

#click
pyautogui.click()

pyautogui.click(x, y)

Click the position of the coordinates (x, y) on the monitor. What are these coordinates (x, y)? For example, if your monitor has a resolution of 1920 x 1080, the upper left is (0,0) and the lower right is (1919, 1079).

0,0       X increases -->
+---------------------------+
|                           | Y increases
|                           |     |
|   1920 x 1080 screen      |     |
|                           |     V
|                           |
|                           |
+---------------------------+ 1919, 1079

(Reprinted from PyAutoGui official documentation<https://pyautogui.readthedocs.io/en/latest/mouse.html>)

You can get the coordinates of the current mouse cursor with pyautogui.position (), so It is easy to move the mouse to the position you want to click and execute pyautogui.position () to check the coordinates.

python

#Check the coordinates of the position you want to click
print(pyautogui.position())

Execution result

Point(x=207, y=528)

python

#Click on the coordinates found above
pyautogui.click(207, 528)

pyautogui.click('button.png')

Specify the image ('button.png') of the button you want to click and click it. Depending on the size and position of the app window, the coordinates of the place you want to click may not always be the same.

In such a case, you can directly click the position by specifying ** the image of the place (button etc.) you want to click **.

Suppose you want to click the ** "☆" shape ** in the Windows "Paint" app.

In this case, cut out the ☆ part as an image ('star.png').

python

#Click the ☆ part
pyautogui.click('star.png')

The method of specifying this image is very convenient, but in some cases the location of the image may not be recognized. It should be noted that it is not in 100 shots.

Click type

pyautogui.click () was a normal ** left click **. If you want to make another click such as right click or double click, replace it with the corresponding function as shown below.

Move mouse cursor

There are two types of mouse cursor movement. The difference is whether the destination is specified as an absolute position or a relative position.

pyautogui.moveTo(x,y) Specify the coordinates (x, y) on the monitor and move to it. It doesn't matter where you are.

You can also specify ** time to move ** as the third argument of moveTo (). If you do not specify the time, it will move instantly, but you can move slowly by specifying the time.

python

# (100,100)Move to position
pyautogui.moveTo(100,100)

# (100,100)Move to the position of
pyautogui.moveTo(100,100,2)

pyautogui.move(x,y) Moves x horizontally and y vertically from the current mouse cursor position. move () can also specify the time to move.

python

#From the current position(100,100)Just move
pyautogui.move(100,100)

#From the current position(100,100)Just move over 2 seconds
pyautogui.move(100,100,2)

drag

There are two types of drag depending on whether it is the absolute position or the relative position of the destination. For dragging, you can also specify ** time to drag ** in the third argument.

python

#From the current position(100,100)Drag to the position of
pyautogui.dragTo(100,100,2)

Keyboard operation

Character input

Use pyautogui.write () to enter characters from the keyboard.

Actually, click the text box etc. where you want to enter characters once, I think you'll have to activate the character cursor and then run pyautogui.write ().

python

#Click where you want to enter characters
pyautogui.click(100, 100)

#Character input
pyautogui.write('Hello world!')

If for some reason you do not want to input at a very high speed, you can specify the number of seconds between character inputs with ʻinterval = xx` in the argument, and you can also space the character input.

python

#0 for each character.Input at 25 second intervals
pyautogui.write('Hello world!', interval=0.25)

Please note that this method ** cannot input Japanese **! So, if you want to enter Japanese, use another method.

The method is ** Copy and paste the character string you want to enter to the clipboard. ** **

Use ** pyperclip ** to manipulate the clipboard. For details on how to use pyperclip, see the following, but here only the code is described.

[Automation] Operate the clipboard with Python and paste the table into Excel https://qiita.com/konitech913/items/83975332e395a387eace

python

import pyperclip

#Copy the entered character string to the clipboard
pyperclip.copy('Hello World!')

# Ctrl+Paste with v
pyautogui.hotkey('ctrl', 'v')

Key input

Use pyautogui.press () if you just want to press a key instead of typing.

python

#Press "Enter"
pyautogui.press('enter')

#Press the "left cursor key" four times
pyautogui.press(['left', 'left', 'left', 'left'])

For ** hotkeys ** such as "Ctrl + C (paste)" and "Ctrl + Z (undo)", specify them with pyautogui.hotkey ().

python

# 「Ctrl+Z(Undo)」
pyautogui.hotkey('ctrl', 'z')

Automate painting

Finally, let's use the knowledge so far to draw with "Paint" that comes with Windows.

Change the size of the ☆ figure, stack 5 figures, and finally enter characters as text.

python

#First grasp the coordinates
print(pyautogui.position()) #Where you want to paste the text
print(pyautogui.position()) #☆ drag start position

Execution result

Point(x=300, y=285)
Point(x=476, y=566)

Once you know the coordinates, it's time to start drawing.

python

import pyautogui
import pyperclip

#Click once to activate the paint window
pyautogui.click(732,22)

text_x=300
text_y=285

star_x=476
star_y=566

#☆ drawing
stride = 50 #Increase ☆ by stride
star_num = 5 #Number of ☆

pyautogui.click('star.png')

#☆ start_Draw as many as num
for i in range(star_num):
    pyautogui.moveTo(star_x-stride*i, star_y-stride*i, 1)
    pyautogui.drag(50+stride*i*2, 50+stride*i*2, 1)
    
#Enter text
pyautogui.click('text.png')

pyautogui.click(text_x, text_y)
pyperclip.copy('It's like a star matryoshka ...') #Copy Japanese to clipboard once
pyautogui.hotkey('ctrl', 'v') #pasting

Execution result

how was it? If you can master pyautogui, it seems that the application will be effective for automating various applications.

reference

Here are other automation series I wrote. If you are interested, please!

[Automation] Extract the table in PDF with Python https://qiita.com/konitech913/items/4ef70e1f7753c824b40f

[Automation] Read a Word document with Python https://qiita.com/konitech913/items/c30236bdf47775535e2f

[Automation] Convert Python code into an exe file https://qiita.com/konitech913/items/6259f13e057bc25ebc23

[Automation] Send Outlook email with Python https://qiita.com/konitech913/items/51867dbe24a2a4272bb6

[Automation] Read Outlook emails with Python https://qiita.com/konitech913/items/8a285522b0c118d5f905

[Automation] Read mail (msg file) with Python https://qiita.com/konitech913/items/fa0cf66aad27d16258c0

[Automation] Operate the clipboard with Python and paste the table into Excel https://qiita.com/konitech913/items/83975332e395a387eace