Continuation of UI Automation in Python This time I will click the button.
ControlPattern Controls that support UI Automation provide an operable interface in the form of Control Pattern. In the case of buttons, InvokePattern is supported, so if you get this and call Invoke (), which is a method of InvokePattern, you can realize button click. You can find out if an AutomationElement supports InvokePattern by looking at the element's IsInvokePatternAvailable.
def click_button(element):
"""IUIAutomationInvokePattern for the specified element.Invoke()Click with
If the IsInvokePatternAvailable property of the specified element is False, do nothing
"""
isClickable = element.GetCurrentPropertyValue(UIA_IsInvokePatternAvailablePropertyId)
if isClickable == True:
ptn = element.GetCurrentPattern(UIA_InvokePatternId)
ptn.QueryInterface(IUIAutomationInvokePattern).Invoke()
Details are below. UI Automation Control Patterns Overview http://msdn.microsoft.com/en-us/library/ee671194(v=vs.85).aspx#controlpatterninterfaces
The .NET version has a Japanese translation (although it is a machine translation). UI Automation Control Pattern Overview http://msdn.microsoft.com/ja-jp/library/ms752362(v=vs.110).aspx
Before actually clicking the button, put it together, including the previous ones.
uiauti.py
# -*- coding: utf-8 -*-
"""
uiautil : UI Automation utility
It only works on Windows.
"""
import comtypes
from comtypes import CoCreateInstance
import comtypes.client
from comtypes.gen.UIAutomationClient import *
__uia = None
__root_element = None
def __init():
global __uia, __root_element
__uia = CoCreateInstance(CUIAutomation._reg_clsid_,
interface=IUIAutomation,
clsctx=comtypes.CLSCTX_INPROC_SERVER)
__root_element = __uia.GetRootElement()
def get_window_element(title):
"""Gets the AutomationElement of the window specified in title
If the title is duplicated on the desktop, return the first one found
"""
win_element = __root_element.FindFirst(TreeScope_Children,
__uia.CreatePropertyCondition(
UIA_NamePropertyId, title))
return win_element
def find_control(base_element, ctltype):
"""Has the specified control type ID from the specified base element subtree
Returns a sequence of elements
"""
condition = __uia.CreatePropertyCondition(UIA_ControlTypePropertyId, ctltype)
ctl_elements = base_element.FindAll(TreeScope_Subtree, condition)
return [ ctl_elements.GetElement(i) for i in range(ctl_elements.Length) ]
def lookup_by_name(elements, name):
"""Elements with the specified Name property from the specified sequence of elements
Returns the first of them.
Returns None if there is no hit
"""
for element in elements:
if element.CurrentName == name:
return element
return None
def lookup_by_automationid(elements, id):
"""Has the specified AutomationId property from the specified sequence of elements
Returns the first of the elements.
Returns None if there is no hit
"""
for element in elements:
if element.CurrentAutomationId == id:
return element
return None
def click_button(element):
"""IUIAutomationInvokePattern for the specified element.Invoke()Click with
If the IsInvokePatternAvailable property of the specified element is False, do nothing
"""
isClickable = element.GetCurrentPropertyValue(UIA_IsInvokePatternAvailablePropertyId)
if isClickable == True:
ptn = element.GetCurrentPattern(UIA_InvokePatternId)
ptn.QueryInterface(IUIAutomationInvokePattern).Invoke()
if __name__ == '__main__':
__init()
Test by running uiautil.py on IDLE. Start the calculator and decide the property that identifies each button. This time I will use AutomationId.
>>>
>>> win = get_window_element('calculator')
>>> btns = find_control(win, UIA_ButtonControlTypeId)
>>> for b in btns: print btns.index(b), b.CurrentName, b.CurrentAutomationId
0 Clear memory 122
1 backspace 83
2 7 137
3 4 134
4 1 131
5 0 130
<Omission>
21 subtraction 94
22 Addition 93
23 Memory subtraction 126
24 square root 110
25 percent 118
26 Reciprocal 114
27 equal sign 121
28 Minimize
29 Maximize
30 close
>>> def one_plus_one():
btn_1 = lookup_by_automationid(btns, "131")
btn_plus = lookup_by_automationid(btns, "93")
btn_equal = lookup_by_automationid(btns, "121")
click_button(btn_1)
click_button(btn_plus)
click_button(btn_1)
click_button(btn_equal)
>>> one_plus_one()
>>>
2 was displayed on the calculator!
This time I am running on Windows 8.1 Pro (x64), but as shown below, it seems that AutomaionId is not guaranteed between builds and releases, so AutomaionId may be different on other OS such as Windows 7. Also, there are cases where AutomationId is not assigned as in minimization and maximization.
From the Description of UIA_AutomationIdPropertyId in http://msdn.microsoft.com/en-us/library/ee684017(v=vs.85).aspx.
Although support for AutomationId is always recommended for better automated testing support, this property is not mandatory. Where it is supported, AutomationId is useful for creating a test automation script that runs regardless of the UI language. Clients should make no assumptions regarding the AutomationId values exposed by other applications. AutomationId is not guaranteed to be stable across different releases or builds of an application.
It seems that the basis for specifying AutomationElement depends on the creation of the application to be operated. Therefore, you will want to find out in advance what properties the Automation Elements of the operation target application have. As in the example above, the method of actually hitting AutomationElement one by one is troublesome, so I searched for a tool that could be easily investigated. there were.
It's also in the Windows SDK from the beginning.
You can use Inspect.exe and Visual UI Automation Verify included in the Windows SDK to check the properties of each element in the AutomationElement tree under the desktop.
Windows Software Development Kit (SDK) for Windows 8.1 http://msdn.microsoft.com/en-us/windows/desktop/bg162891.aspx
If you install the Windows SDK, you will see the following (xxx is x86, x64, arm. The version may differ depending on the version of the SDK you have installed). Visual Studio is not required.
%ProgramFiles%\Windows Kits\8.1\bin\xxx
For details on how to use the tool, see below. Inspect http://msdn.microsoft.com/en-us/library/windows/desktop/dd318521(v=vs.85).aspx
Visual UI Automation Verify http://msdn.microsoft.com/en-us/library/windows/desktop/jj160544(v=vs.85).aspx
Recommended Posts