How to implement UI automated test using image comparison in Selenium

Introduction

UL Systems Advent Calendar 2020-Day 19

At the final stage of system development, I think that E2E tests will be used to confirm that the results are as expected, including the UI and backend logic. However, in the E2E test, there are many cases where manual operation is performed and visual confirmation is performed. In such a case, even if you modify some logic and test again, it cannot be done automatically. In addition, in the test using Selenium, the E2E test can be performed automatically, but there are cases where it is not adopted due to the man-hours required for test code creation.

Therefore, I think that the man-hours for creating test code can be reduced by using image comparison, so I will introduce a UI test using Selenium and image comparison.

Image Comparison used this time is a library for comparing images.

What is Image Comparison?

It is a library that can visually show the difference by comparing two images of the same size and drawing rectangles in different parts.

Library property description

A property that can be specified using the library.

Property Explanation
threshold Maximum distance threshold between pixels that are judged to be unequal (default: 5)
rectangleLineWidth Rectangle line width (default: 1)
destination Where to save the comparison result file
minimalRectangleSize Minimum rectangular size (default: 1)
maximalRectangleCount Maximum number of rectangles drawn (default:-1[Unlimited])
pixelToleranceLevel Pixel tolerance level (default: 0.1[10%])
excludedAreas Lists that are ignored when comparing images
drawExcludedRectangles Whether to draw an excluded rectangle
fillExcludedRectangles Whether to fill the excluded rectangle
percentOpacityExcludedRectangles Excluded rectangle opacity
fillDifferenceRectangles Whether the difference rectangle is possible
percentOpacityDifferenceRectangles Diff rectangle opacity
allowingPercentOfDifferentPixels Percentage of pixels ignored (default:-1[Unlimited])

Example using Selenium

This time, we will use Selenium to perform an automatic test that compares the search result screen searched for Ul Systems Co., Ltd. on Google with the search result screen searched forUr Systems. If the test result is false, it will generate a comparison image with different parts surrounded by rectangles.

Take a screen capture of the current situation

First of all, get the screen capture image to be the comparison source. After starting chrome using Selenium, search for Ul Systems Co., Ltd. on Google and save the captured image of the screen displaying the search results.

public class ExpectedImage {
    public static void main(String[] args) throws InterruptedException {
        // Chrome
        System.setProperty("webdriver.chrome.driver", "selenium/webdriver/chrome/87.0.4280.88/win32/chromedriver.exe");
        //Configure Chrome launch options
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless");
        WebDriver driver = new ChromeDriver(options);

        driver.get("https://www.google.co.jp/");
        WebElement element = driver.findElement(By.name("q"));
        element.sendKeys("Ul Systems Co., Ltd.");
        element.submit();
        driver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS);
        Thread.sleep(5000L);
        driver.manage().window().setSize(new Dimension(Integer.parseInt("1500"), Integer.parseInt("3000")));
        File screenFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
        Files.copy(screenFile.toPath(), Paths.get("src/test/resources/screenshot/expected/Ul Systems Co., Ltd..png "), StandardCopyOption.REPLACE_EXISTING);

        driver.quit();
    }
}

Comparison source search result screen

This is a screen capture image that serves as a comparison source. search_uls.png

UI automated test of search results

Use Selenium and Image Comparison to write UI autotest code and compare with the screen capture image of the comparison source. If the comparison results are different, an image with different parts covered with rectangles will be output.

This time, I would like to search for Ul Systems on Google after starting chrome using Selenium and compare the images on the search result screen.

public class SearchULSTest {
    private WebDriver driver;

    @Before
    public void setUp() {
        // Chrome
        System.setProperty("webdriver.chrome.driver", "selenium/webdriver/chrome/87.0.4280.88/win32/chromedriver.exe");
        //Configure Chrome launch options
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless");
        this.driver = new ChromeDriver(options);
    }

    @After
    public void closeDriver() {
        this.driver.quit();
    }

    @Test
    public void test0001() throws IOException, InterruptedException {
        driver.get("https://www.google.co.jp/");
        WebElement element = driver.findElement(By.name("q"));
        element.sendKeys("Ur Systems");
        element.submit();
        driver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS);
        Thread.sleep(5000L);
        driver.manage().window().setSize(new Dimension(Integer.parseInt("1500"), Integer.parseInt("3000")));

        File screenFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
        Files.copy(screenFile.toPath(), Paths.get("src/test/resources/screenshot/actual/Ur Systems.png "), StandardCopyOption.REPLACE_EXISTING);
        Thread.sleep(5000L);

        boolean result = compareImage("screenshot/expected/Ul Systems Co., Ltd..png ", "screenshot/actual/Ur Systems.png ");
        Assert.assertTrue(result);
    }

    private static boolean compareImage(String expected, String actual) {
        //Loading images to compare
        BufferedImage expectedImage = ImageComparisonUtil.readImageFromResources(expected);
        BufferedImage actualImage = ImageComparisonUtil.readImageFromResources(actual);
        //Where to save the comparison result file
        File resultDestination = new File("compareimage.png ");
        //Generation of image comparison object
        ImageComparison imageComparison = new ImageComparison(expectedImage, actualImage, resultDestination);

        //Maximum distance between pixels to determine image comparison
        imageComparison.setThreshold(10);
        //Difference rectangle line width
        imageComparison.setRectangleLineWidth(2);
        //Specifying whether to fill the inside of the difference rectangle and transparency
        imageComparison.setDifferenceRectangleFilling(true, 10.0);
        //Specifying whether to fill the inside of the exclusion rectangle and transparency
        imageComparison.setExcludedRectangleFilling(true, 10.0);
        //Maximum number of rectangles drawn
        imageComparison.setMaximalRectangleCount(100);
        //Minimum rectangular size
        imageComparison.setMinimalRectangleSize(10);
        //Pixel tolerance level
        imageComparison.setPixelToleranceLevel(0.2);

        //Image comparison
        ImageComparisonResult imageComparisonResult = imageComparison.compareImages();
        if (ImageComparisonState.MATCH == imageComparisonResult.getImageComparisonState()) return true;

        //Save the image of the comparison result
        ImageComparisonState imageComparisonState = imageComparisonResult.getImageComparisonState();
        BufferedImage resultImage = imageComparisonResult.getResult();
        ImageComparisonUtil.saveImage(resultDestination, resultImage);
        return false;
    }
}

Search result screen for comparison

It is a captured image of the screen that displayed the search results by searching for Ul Systems on Google for comparison.

search_uls.png

Image of comparison result

It is an image that is compared with the original image and the different parts are marked with red rectangles. search_uls_compareimage.png

Summary

This time, I used Image Comparison to perform an automatic test by comparing images. It is very difficult to write verification code for each item on the screen, but by using image comparison, you can find out at once what is different if it is the same or different. As a caveat, a positive screen image is required in advance, and since it is a comparison verification using a static screen image, it cannot be applied to screen operation verification.

How about trying to reduce the man-hours for creating test code by using image comparison in this way?

Recommended Posts

How to implement UI automated test using image comparison in Selenium
How to implement image posting using rails
How to implement image posting function using Active Storage in Ruby on Rails
How to test file upload screen in Spring + Selenium
How to implement a circular profile image in Rails using CarrierWave and R Magick
[Swift] How to implement the Twitter login function using Firebase UI ②
Add files to jar files
Extend Selenium Grid (Hub side)
How to add ActionText function
4 Add println to the interpreter
[Rails] Add column to devise
JMX support for Selenium Grid
How to add HDD to Ubuntu
[JQuery] How to preview the selected image immediately + Add image posting gem
How to implement UI automated test using image comparison in Selenium
How to implement search functionality in Rails
How to implement date calculation in Java
How to implement Kalman filter in Java
How to filter JUnit Test in Gradle
How to implement coding conventions in Java
How to implement ranking functionality in Rails
[Swift5] How to implement animation using "lottie-ios"
How to implement asynchronous processing in Outsystems
How to implement a slideshow using slick in Rails (one by one & multiple by one)
[Swift] How to set an image in the background without using UIImageView.
How to use "sign_in" in integration test (RSpec)
How to implement the breadcrumb function using gretel
Flow to implement image posting function using ActiveStorage
How to implement a like feature in Rails
How to implement optimistic locking in REST API
How to implement Pagination in GraphQL (for ruby)
How to make an image partially transparent in Processing
Continued ・ Flow to implement image posting function using ActiveStorage
How to implement guest login in 5 minutes in rails portfolio
How to unit test with JVM with source using RxAndroid
Implement test doubles in JUnit without using external libraries
How to install geckodriver (Selenium WebDriver) automatically using WebDriverManager
How to write a date comparison search in Rails
[Rails 6] How to set a background image in Rails [CSS]
Summary of how to implement default arguments in Java
Rails learning How to implement search function using ActiveModel
How to launch Swagger UI and Swagger Editor in Docker
[Rails] How to display an image in the view
How to dynamically write iterative test cases using test / unit (Test :: Unit)
How to test including images when using ActiveStorage and Faker
Change driver to Rack :: Test if driver wants to POST in Selenium test
How to easily implement in-app purchase using itemstore <Implementation: Android>
How to implement infinite scrolling (page nate) in Swift TableView
How to deploy jQuery in your Rails app using Webpacker
How to implement one-line display of TextView in Android development
How to erase test image after running Rspec test with CarrierWave
How to monitor application information in real time using JConsole
How to change BackgroundColor etc. of NavigationBar in Swift UI
How to control transactions in Spring Boot without using @Transactional
[Rails] How to implement scraping
[Java] How to implement multithreading
How to authorize using graphql-ruby
How to POST JSON in Java-Method using OkHttp3 and method using HttpUrlConnection-
How to implement a job that uses Java API in JobScheduler
How to make an image posted using css look like an icon
How to test file upload screen in Spring + Selenium
Add image selector to Selenium Grid
How to dynamically write iterative test cases using test / unit (Test :: Unit)
[Behavior confirmed in December 2020] How to implement the alert display function
How to solve the unknown error when using slf4j in Java
[rails6.0.0] How to save images using Active Storage in wizard format
How to use Lombok in Spring
Implement Table Driven Test in Java 14
How to find May'n in XPath
Image upload using CarrierWave ~ Rspec test ~
How to hide scrollbars in WebView