Originally it seems that the plan was to have WinActor used in the field, I had a contract for a while, but I couldn't recover the investment, so I cut the entire budget of RPA including WinActor. As a result, we decided to proceed with RPA with the necessary cost of 0 yen. This time, I will write about the struggle to realize the automatic operation of GUI.
There are various types of automation of GUI operations. The first thing I arrived at was image authentication using pyautogui + keyboard / mouse operation. This will definitely operate the keyboard and mouse, but I'm not sure if they really operate the screen.
If you add image authentication to it, the accuracy will increase considerably. However, this also has its drawbacks. ・ Slow because the operation is to search for the specified image on the screen display. ・ It does not respond if the background image of the desktop is changed. ・ The operation changes with one image to be searched.
I searched for something more stable and found it. It's "UI Automation" Rejected for the following reasons ・ I don't like coding personally ・ The amount of information is small (I personally felt that ...)
I gathered various information from there and arrived at it. "Windows App Driver" (WAD because it is long) The following are the reasons for adoption -Coding looks similar to Selenium └ Anyone can use it if they remember Selenium. And vice versa ・ The information is relatively new, and you can easily find it if you look it up.
It's similar to VB.net or JQuery It is a type that supplements the target and operates by specifying the ID of the operation action and other attributes.
Basically, it will work if you do the contents written in the following URL. https://arakan-pgm-ai.hatenablog.com/entry/2019/08/19/000000
First, you need to give the execution environment "Developer Mode" permissions. Therefore, the option of choosing a personal computer as the execution environment was quickly eliminated.
I chose to use Windows 10 in a virtual environment (If you can't prepare a virtual environment, you can use a PC that has been withdrawn as a substitute for a PC.)
Another purpose is to prevent humans from competing with resources such as clipboards, mice, and keyboards. It also makes maintenance easier.
If there are 10 workers, it may be necessary to install it on 10 PCs. If multiple workers gather, some people may not follow the precautions. Even if there is only one worker, there are various things such as changing the person in charge. We take the method of creating and operating an environment that workers cannot touch.
How to run an automated program? This is easiest to use with a task scheduler or something. There are various measures around here, so I will come back later.
As automation using WAD becomes more widespread, other programmers will start using this environment. Then, even those who do not understand well will start using it by imitating the appearance. When that happened, I closed the black screen ... It ’s something that you do In that case, even if you set the startup with the task scheduler etc. It doesn't move because WAD is stopped.
The simplest countermeasure would be to start it so that the black screen is not shown. The following code should be placed in Shell: startup
wad.vbs
dim ws
Set ws = CreateObject("Wscript.Shell")
ws.Run """C:\Program Files (x86)\Windows Application Driver\WinAppDriver.exe""", 0
Since it cannot be operated, let's install the application to be operated locally. As you can see from the GUI research tool, when trying to operate through VNC or Xen You can only get information about the campus part of the VNC or Xen screen.
Input the character string to be input to the clipboard using pyperclip Let's paste by sending Ctrl + V to the text box. You can use the standard input mechanism, but It behaves like hitting the keyboard, so when entering long sentences The operating speed will be slower.
It ’s a premise, The code below works like launching the calculator and supplementing the calculator's app.
desired_caps["app"] = "Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"
for that reason You cannot supplement the app by specifying the batch path in desired_caps ["app"].
subprocess.Popen(['start',Batch shortcut PATH], shell=True)
#Certainly, explorer should have been good as well as start.
-Change the following part of the Python code
desired_caps["app"] = "Microsoft.WindowsCalculator_8wekyb3d8bbwe!App"
↓
desired_caps["app"] = "Root"
This complements the desktop itself.
If the above two conditions are met Is the application with the wind title "●●●" running on the desktop? You will be able to confirm that. So ● Wait for a second, and if you can't find it, you can make an error / start, and you can continue working.
You can also operate after checking the attribute value in the same way as a text box etc. Some are quite complicated.
As a countermeasure, After checking the wind title This can be easily achieved by combining it with the keyboard operation of pyautogui.
I think that most things can be done by doing this much. Next time, let's summarize the area around automatic operation of the WEB.
Recommended Posts