Mouse operation using Windows API in Python

Trigger

When I saw a video introducing an anti-recoil macro of a certain FPS game on Twitter at the end of last year, I thought that it would be difficult to make steady adjustments to the macro ~~ It was close to a fraudulent tool, but at the same time it was set in advance. I was wondering if I could do something interesting if I could perform the movements in the game.

trial and error

In Python, there are libraries such as PyAutoGUI and pynput that operate the mouse and keyboard, and you can easily operate and automate them by using these. However, the game I was playing did not accept any operation. Is DirectX involved? After various searches, I found a substitute called Serpent.AI that lets AI play games. Serpent.AI seems to operate the game with Win32 API, and the game I'm playing could be operated with Win32 API.

environment

• Windows 10 64bit
• Python 3.7.4

Write code

There are two ways to operate the mouse with the Win32 API.

1. Use SetCursorPos
2. Use SendInput

1. Use SetCursorPos

Use the standard library ctypes to call the Win32 API from Python.

input_1.py

import ctypes

#SetCursorPos(x, y)
ctypes.windll.user32.SetCursorPos(200, 300)

Just specify with x, y (horizontal, vertical). The value of (0,0) increases toward the lower right at the upper left of the screen. This code moves the cursor to x: 200 y: 300. However, SetCursorPos cannot operate the game.

2. Use SendInput

You can move the mouse cursor, click, and enter keys. For details on how to use it, see here. The code is this and [this](https://stackoverflow.com/questions/49973060/simpl -union-in-python-to-send-keyboard-events-in-windows) was used as a reference.

Declare the INPUT structure required for SendInput and the MOUSEINPUT structure that contains mouse operation information.

input_2.py

import ctypes

ULONG_PTR = ctypes.POINTER(ctypes.c_ulong)

#Mouse event information
class MOUSEINPUT(ctypes.Structure):
    _fields_ = [("dx", ctypes.c_long),
                ("dy", ctypes.c_long),
                ("mouseData", ctypes.c_ulong),
                ("dwFlags", ctypes.c_ulong),
                ("time", ctypes.c_ulong),
                ("dwExtraInfo", ULONG_PTR)]


class INPUT(ctypes.Structure):
    _fields_ = [("type", ctypes.c_ulong),
                ("mi", MOUSEINPUT)]

Specifies the SendInput argument and return type.

input_2.py

LPINPUT = ctypes.POINTER(INPUT)

SendInput = ctypes.windll.user32.SendInput
SendInput.argtypes = (ctypes.c_uint, LPINPUT, ctypes.c_int)
SendInput.restype = ctypes.c_uint

If you convert the coordinates and plunge into SendInput, the mouse cursor will move.

input_2.py

x, y = 200, 300
x = x * 65536 // 1920
y = y * 65536 // 1080
_mi = MOUSEINPUT(x, y, 0, (0x0001 | 0x8000), 0, None)
SendInput(1, INPUT(0, _mi), ctypes.sizeof(INPUT))

The detailed explanation will be long, In these two lines, the coordinates of the cursor (200,300) are converted to the coordinates of 0 to 65,535 (6826,18204). The divisor changes depending on the screen resolution. Division is rounded down, but I don't care because it's not much different from rounding. As we will see later, it is only needed to move in absolute coordinates.

input_2.py

x = x * 65536 // 1920 # x * 65536 //Screen resolution(side)
y = y * 65536 // 1080 # y * 65536 //Screen resolution(Vertical)

The amount of rotation of the wheel is 120, which is one notch of a standard mouse. Integers other than multiples of 120 are fine. You can also scroll horizontally. Multiple mouse events can be set with the bitwise | operator. I have summarized the frequently used items in the table. Click here for details (https://docs.microsoft.com/ja-jp/windows/win32/api/winuser/ns-winuser-mouseinput)

One thing to note here. The format of the coordinates changes depending on whether you use absolute coordinates or relative coordinates. If you use absolute coordinates, you need to convert to coordinates from 0 to 65,535, but if you use relative coordinates, you do not need to convert the coordinates. In this case, it remains (200,300).

input_2.py

# MOUSEINPUT(x coordinate,y coordinate,Wheel rotation amount,Mouse event, 0, None)
_mi = MOUSEINPUT(x, y, 0, (0x0001 | 0x8000), 0, None)

There are three types of input, as shown in the table below.

input_2.py

# SendInput(1, INPUT(Input type, _mi), ctypes.sizeof(INPUT))
SendInput(1, INPUT(0, _mi), ctypes.sizeof(INPUT))

Summary

In SendInput, you can operate the keyboard in addition to the mouse operation, but for the sake of simplicity, we have focused on the mouse operation. Next time, I will explain it including keyboard operation.

Finally

I don't think it's relevant to anyone reading this article, but the program itself was completed long ago. When I was writing the program, I was doing it without thinking deeply about Yoshi if I moved, so when I looked back on it later, it was supposed to be Tondemonai. I put a comment. So, while organizing, I wrote an article about the main points of the program. If you write an article like this, you will not forget it.