How to implement UI automated test using image comparison in Selenium

Introduction

UL Systems Advent Calendar 2020-Day 19

At the final stage of system development, I think that E2E tests will be used to confirm that the results are as expected, including the UI and backend logic. However, in the E2E test, there are many cases where manual operation is performed and visual confirmation is performed. In such a case, even if you modify some logic and test again, it cannot be done automatically. In addition, in the test using Selenium, the E2E test can be performed automatically, but there are cases where it is not adopted due to the man-hours required for test code creation.

Therefore, I think that the man-hours for creating test code can be reduced by using image comparison, so I will introduce a UI test using Selenium and image comparison.

Image Comparison used this time is a library for comparing images.

What is Image Comparison?

It is a library that can visually show the difference by comparing two images of the same size and drawing rectangles in different parts.

Library property description

A property that can be specified using the library.

Example using Selenium

This time, we will use Selenium to perform an automatic test that compares the search result screen searched for Ul Systems Co., Ltd. on Google with the search result screen searched forUr Systems. If the test result is false, it will generate a comparison image with different parts surrounded by rectangles.

Take a screen capture of the current situation

First of all, get the screen capture image to be the comparison source. After starting chrome using Selenium, search for Ul Systems Co., Ltd. on Google and save the captured image of the screen displaying the search results.

public class ExpectedImage {
    public static void main(String[] args) throws InterruptedException {
        // Chrome
        System.setProperty("webdriver.chrome.driver", "selenium/webdriver/chrome/87.0.4280.88/win32/chromedriver.exe");
        //Configure Chrome launch options
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless");
        WebDriver driver = new ChromeDriver(options);

        driver.get("https://www.google.co.jp/");
        WebElement element = driver.findElement(By.name("q"));
        element.sendKeys("Ul Systems Co., Ltd.");
        element.submit();
        driver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS);
        Thread.sleep(5000L);
        driver.manage().window().setSize(new Dimension(Integer.parseInt("1500"), Integer.parseInt("3000")));
        File screenFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
        Files.copy(screenFile.toPath(), Paths.get("src/test/resources/screenshot/expected/Ul Systems Co., Ltd..png "), StandardCopyOption.REPLACE_EXISTING);

        driver.quit();
    }
}

Comparison source search result screen

This is a screen capture image that serves as a comparison source.

UI automated test of search results

Use Selenium and Image Comparison to write UI autotest code and compare with the screen capture image of the comparison source. If the comparison results are different, an image with different parts covered with rectangles will be output.

This time, I would like to search for Ul Systems on Google after starting chrome using Selenium and compare the images on the search result screen.

public class SearchULSTest {
    private WebDriver driver;

    @Before
    public void setUp() {
        // Chrome
        System.setProperty("webdriver.chrome.driver", "selenium/webdriver/chrome/87.0.4280.88/win32/chromedriver.exe");
        //Configure Chrome launch options
        ChromeOptions options = new ChromeOptions();
        options.addArguments("--headless");
        this.driver = new ChromeDriver(options);
    }

    @After
    public void closeDriver() {
        this.driver.quit();
    }

    @Test
    public void test0001() throws IOException, InterruptedException {
        driver.get("https://www.google.co.jp/");
        WebElement element = driver.findElement(By.name("q"));
        element.sendKeys("Ur Systems");
        element.submit();
        driver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS);
        Thread.sleep(5000L);
        driver.manage().window().setSize(new Dimension(Integer.parseInt("1500"), Integer.parseInt("3000")));

        File screenFile = ((TakesScreenshot) driver).getScreenshotAs(OutputType.FILE);
        Files.copy(screenFile.toPath(), Paths.get("src/test/resources/screenshot/actual/Ur Systems.png "), StandardCopyOption.REPLACE_EXISTING);
        Thread.sleep(5000L);

        boolean result = compareImage("screenshot/expected/Ul Systems Co., Ltd..png ", "screenshot/actual/Ur Systems.png ");
        Assert.assertTrue(result);
    }

    private static boolean compareImage(String expected, String actual) {
        //Loading images to compare
        BufferedImage expectedImage = ImageComparisonUtil.readImageFromResources(expected);
        BufferedImage actualImage = ImageComparisonUtil.readImageFromResources(actual);
        //Where to save the comparison result file
        File resultDestination = new File("compareimage.png ");
        //Generation of image comparison object
        ImageComparison imageComparison = new ImageComparison(expectedImage, actualImage, resultDestination);

        //Maximum distance between pixels to determine image comparison
        imageComparison.setThreshold(10);
        //Difference rectangle line width
        imageComparison.setRectangleLineWidth(2);
        //Specifying whether to fill the inside of the difference rectangle and transparency
        imageComparison.setDifferenceRectangleFilling(true, 10.0);
        //Specifying whether to fill the inside of the exclusion rectangle and transparency
        imageComparison.setExcludedRectangleFilling(true, 10.0);
        //Maximum number of rectangles drawn
        imageComparison.setMaximalRectangleCount(100);
        //Minimum rectangular size
        imageComparison.setMinimalRectangleSize(10);
        //Pixel tolerance level
        imageComparison.setPixelToleranceLevel(0.2);

        //Image comparison
        ImageComparisonResult imageComparisonResult = imageComparison.compareImages();
        if (ImageComparisonState.MATCH == imageComparisonResult.getImageComparisonState()) return true;

        //Save the image of the comparison result
        ImageComparisonState imageComparisonState = imageComparisonResult.getImageComparisonState();
        BufferedImage resultImage = imageComparisonResult.getResult();
        ImageComparisonUtil.saveImage(resultDestination, resultImage);
        return false;
    }
}

Search result screen for comparison

It is a captured image of the screen that displayed the search results by searching for Ul Systems on Google for comparison.

Image of comparison result

It is an image that is compared with the original image and the different parts are marked with red rectangles.

Summary

This time, I used Image Comparison to perform an automatic test by comparing images. It is very difficult to write verification code for each item on the screen, but by using image comparison, you can find out at once what is different if it is the same or different. As a caveat, a positive screen image is required in advance, and since it is a comparison verification using a static screen image, it cannot be applied to screen operation verification.

How about trying to reduce the man-hours for creating test code by using image comparison in this way?