[PYTHON] Flash Selenium support

Purpose

• I want to automatically test the UI of a site created with Adobe Flash.
  • http://www.adobe.com/jp/products/flashplayer.html
• Mainly, Click, Drag & Drop, Double Click, etc.

problem

• Automatic test of Web Browser = Select Selenium Python with the simple idea of using Selenium.
• However, since Flash is regarded as one object in HTML, Selenium has no way to specify its child element.

solution

• Selenium Web Driver API says move_to_element_with_offset (to_element, xoffset, yoffset) There is a Method. Is this available?

procedure

• The implementation method will be explained using the Flash game Kantai Collection as an example.
• This time, it is just for example, and you should not go to the actual game.

1. Get a screen shot of the screen with Selenium.
2. Obtain the target coordinate position (x, y) from the Reference image prepared in advance using Template Matcing using OpenCV.
3. Specify the standard HTML Element (this time the company logo)
4. Compare the distance from the specified HTML Element to the target coordinate position, determine the offset value, and execute using the previous Method.

Implementation

• This time we will implement only a simple Click operation.

• Since Drag & Drop and Double Click are also implemented in the Selenium Python API, it may be possible to put it to practical use by applying this method. I will not do it because it is troublesome.

• This time, the process of starting Browser and Template Matching by OpenCV is omitted, and the implementation is only for step 3-4. For other implementation methods, please check each website.

  • Selenium Python
  • Template Matching

click.py

from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains

class Browser(object):
    ...

    def click(self, element, x, y):
        """
            element : Target HTML Element
            x : offset_x
            y : offset_y
        """
        self.driver = webdriver.Firefox()
        target = self.driver.find_element_by_id(element)
        off_x = int(target.size["width"]) / 2
        off_y = int(target.size["height"]) / 2
        actions = ActionChains(self.driver)
        actions.move_to_element(target)
        actions.move_by_offset(x - off_x, y - off_y)
        actions.click()
        actions.move_to_element(target)
        actions.perform()

    ...

• A series of actions are registered and executed using ActionChains.
• When Element is specified, it seems that the specification is to click the center of Element, so make fine adjustments.
• The reason why we move to the last specified Element again is that if we leave it as it is, it may get a MouseOver judgment.

Conclusion

• I think there is a way to make it easier.