Easily create homemade RPA using Python

This article is the 24th day article of RPA (Robotic Process Automation) Advent Calendar 2020.

Summary of 3 lines in this article

--Create your own simple RPA using Python's pywinauto library --As a sample scenario, automate the operation of the Windows calculator --Finally, I will mention the technical specifications of pywinauto.

Introduction

Of course, you can automate your business by using the RPA tools that are distributed in the streets. However, with programming languages ​​and their libraries, you can casually automate your business for free without using RPA tools. This time, I will use a programming language called Python and a library called pywinauto to automate the operation of applications on Windows.

Target

--Those who are interested in business automation using Python --Those who do not have RPA tools at hand but want to automate their work somehow --Those who want to easily try "automation" before introducing RPA tools

What to prepare

--PC with Windows 10 installed

Use pywinauto to automate calculator operations

1. Installation of various libraries, etc.

Python First, install Python. You can install it from the Official Page of Python, but it is difficult to understand, so install it from the following page.

Unofficial Python Download Link

For details on how to install, see This site. Don't forget to check "Add Python 3.x to PATH ".

After the installation is complete, launch a command prompt and type python --version. If you see the version of Python you have installed, the installation is successful.

> python --version
Python 3.8.5

pipenv Install pipenv, Python's package management system. Launch a command prompt and install with the following command.

> pip install pipenv

Initializes pipenv. First, create a working folder. It doesn't matter where you are. After that, move to the relevant folder at the command prompt and hit the following command. A file called Pipfile will be created in the created folder.

> pipenv --python 3

pywinauto Install piwinauto, which is the library that actually automates the operation of the application. Install with the following command.

> pipenv install pywinauto

2. Write a program that automates the operation of the calculator

The following program allows you to "start the Windows calculator and calculate 1 + 2 ". Save it with a file name of your choice (eg calc.py).

from time import sleep
from pywinauto import Desktop, Application

app = Application(backend="uia")
app.start("calc.exe")

dlg = Desktop(backend="uia")["calculator"]

dlg['1'].click()
sleep(1)
dlg['plus'].click()
sleep(1)
dlg['2'].click()
sleep(1)
dlg['equal sign'].click()
sleep(1)

dlg.close()

3. Run the program

You can execute the program with the following command.

> pipenv run python calc.py

Hopefully the calculator will start up automatically and calculate 1 + 2.

Technical background of pywinauto

Here are some technical backgrounds you should know about using pywinauto. The following information is often found in the Official Documentation (https://pywinauto.readthedocs.io/en/latest/). Please check the official documentation for a detailed explanation.

About element identification technology

With pywinauto, you can use two element identification technologies, "Win32 API" and "UI Automation". Which one you use depends on the application you want to automate. In some cases, it is easier to identify the element using "Win32 API", and in other cases, it is easier to identify the element using "UI Automation".

If you want to use Win32 API, launch the application as follows.

from pywinauto import Application
app = Application(backend="win32").start("notepad.exe")

On the other hand, if you want to use UI Automation, launch the application as follows.

from pywinauto import Application
app = Application(backend="uia").start("notepad.exe")

The default is Win32 API.

Personally, when automating recently created modern desktop applications, I feel that it is often easier to identify the elements using "UI Automation". The following Qiita article is detailed about element identification using UI Automation.

I tried disassembling the screens of various applications with the screen element decomposition function of RPA https://qiita.com/Okura_/items/4406e3de8a6582948526

As an aside, I was wondering if Win32 API and MSAA (the predecessor technology of UI Automation) are the same, but it seems that they are different.

MSAA is not the same as backend="win32" in pywinauto. https://github.com/pywinauto/pywinauto/issues/268#issuecomment-261468161

About Attribute Resolution Magic

pywinauto uses a technique called attribute resolution magic to identify the element to be manipulated. Take the calculator as an example. In order to identify the "1" button, the above program made the following description.

dlg['1']

This description is very intuitive, but you can also identify the element in the following ways:

dlg['1Button']

Alternatively, the following description is also possible. The number 24 is thought to mean that the "1" button is the 24th button in the internal structure of the calculator application.

dlg['Button24']

Believe it or not, you can also catch the "1" button below.

dlg['11']

As you can see, in pywinauto, the way elements are identified is not unique. The official document states that "we use a matching algorithm that is resistant to typos and typographical errors" (the contents of the algorithm cannot be read).

But fortunately pywinauto uses “best match” algorithm to make a lookup resistant to typos and small variations. https://pywinauto.readthedocs.io/en/latest/getting_started.html#attribute-resolution-magic

To find out what words can be used to identify an element, use the print_control_identifiers () method. The following is the return value when the method is applied to the calculator dialog.

>>> dlg.print_control_identifiers()
Control Identifiers:

Dialog - 'calculator'    (L2780, T141, R3200, B816)
['calculator', 'Dialog', 'calculatorDialog', 'calculator0', 'calculator1', 'Dialog0', 'Dialog1', 'calculatorDialog0', 'calculatorDialog1']
child_window(title="calculator", control_type="Window")
   |
   | Dialog - 'calculator'    (L3004, T142, R3192, B174)
   | ['Calculator 2', 'Dialog2', 'Calculator Dialog2']
   | child_window(title="calculator", auto_id="TitleBar", control_type="Window")
   |    |
   |    | Menu - 'system'    (L0, T0, R0, B0)
   |    | ['System Menu', 'system', 'Menu', 'system0', 'system1']
   |    | child_window(title="system", auto_id="SystemMenuBar", control_type="MenuBar")
   |    |    |
   |    |    | MenuItem - 'system'    (L0, T0, R0, B0)
   |    |    | ['MenuItem', 'System 2', 'System MenuItem']
   |    |    | child_window(title="system", control_type="MenuItem")
   |    |
   |    | Button - 'Minimize the calculator'    (L3054, T142, R3100, B174)
   |    | ['Minimize the calculator', 'Button', 'Minimize the calculatorButton', 'Button0', 'Button1']
   |    | child_window(title="Minimize the calculator", auto_id="Minimize", control_type="Button")
   |    |
   |    | Button - 'Maximize the calculator'    (L3100, T142, R3146, B174)
   |    | ['Button that maximizes the calculator', 'Maximize the calculator', 'Button2']
   |    | child_window(title="Maximize the calculator", auto_id="Maximize", control_type="Button")
   |    |
   |    | Button - 'Close the calculator'    (L3146, T142, R3192, B174)
   |    | ['Close the calculator', 'Close the calculatorButton', 'Button3']
   |    | child_window(title="Close the calculator", auto_id="Close", control_type="Button")

...

   |    |    |    | Button - '1'    (L2792, T665, R2889, B733)
   |    |    |    | ['1', '1Button', 'Button24']
   |    |    |    | child_window(title="1", auto_id="num1Button", control_type="Button")

...

From this output result, it can be read that the "1" button can be identified by the notation such as 1, 1Button, Button24.

About the class structure of the pywinauto library

Let's verify what class the target operated by pywinauto belongs to.

Take the calculator application described above as an example. With the calculator running, you can check which class the object corresponding to the calculator's "1" button belongs to on pywinauto by performing the following operations with the Python REPL. I will.

>>> dlg = Desktop(backend="uia")["calculator"]
>>> dlg["1"].wrapper_object() #Check the actual condition of the "1" button on the calculator
<uia_controls.ButtonWrapper - '1', Button, -2595060702488467549>
>>> dlg["1"].wrapper_object().__class__
<class 'pywinauto.controls.uia_controls.ButtonWrapper'>

Apparently, you can see that the "1" button is defined within pywinauto as an instance of the uia_controls.ButtonWrapper class.

Tracing the ancestors of this class, we get the following output:

>>> dlg["1"].wrapper_object().__class__.mro()
[<class 'pywinauto.controls.uia_controls.ButtonWrapper'>, <class 'pywinauto.controls.uiawrapper.UIAWrapper'>, <class 'pywinauto.base_wrapper.BaseWrapper'>, <class 'object'>]

In conclusion, the structure of the class that appears in pywinauto is roughly as follows. Note that this figure does not represent all classes.

By knowing which class the operation target belongs to, you can understand what methods are available for the operation target. You can check what methods are available for the instances belonging to each class from this document.