I checked the automatic unit test creation tool (end of 2019 version)

I checked the automatic unit test creation tool (end of 2019 version)

Hello. 2019 is almost over. This article is the 25th day article of Software Testing Advent Calendar 2019.

The previous day's article was written by my colleague @ozhiro san, "What is the purpose of your automated testing? ](Https://qiita.com/ozhiro/items/5aa95c6360a8930df325). __be moved! __ Well, today I examined the current status of automatic unit test creation tools.

Why did you decide to look it up?

Unit testing is important, isn't it? As software is developed and modified on a daily basis, unit testing is a test tool that ensures the quality of the developed software. In addition, test execution can be automated, reducing test man-hours.

However, test code development is required to execute unit tests, and development man-hours are required accordingly. For more comprehensiveness, it can take as long as developing the software under test.

Against this background, there are not enough development man-hours and time periods, so it is quite common to do manual tests without creating unit tests. (It's different now, but the project of the previous job and the previous job was like that.) Also, I already have a huge amount of source code and I want to create a unit test from now on, but where should I start? I think there are many situations where you don't know.

I thought there. In such a world where AI and RPA (Robotic Process Automation) are booming now, I think that tools that analyze source code and automatically create unit tests already exist in this world. Isn't it like this in 2020? When.

We in 2020 (image):

Find out what tools are available

I will immediately search the Internet for what kind of tools are available. The search condition is "generate unit test automatically", and since I am an engineer who mainly uses Java, I also add "Java" to the condition.

Surprisingly, there are few search results and old articles can be seen here and there, so I felt a little uneasy ... (´ ・ ω ・ ｀), but I picked up about four good ones.

• TestMe
• Squaretest
• EvoSuite
• Randoop

I couldn't understand the usability just by looking at the outline, so I created a test code from the actual source code and evaluated it. We have prepared the following two types of source code.

1. [Class whose processing is closed only in the class and has only stateless methods](https://github.com/policeman-kh/generate_test/blob/master/src/main/java/test/NumberUtils. java)
2. Spring MVC service class

I wanted to add some branching to method 1 and add the test coverage rate to the evaluation. 2 is a class with a DI (Dependency Injection) instance. I want you to replace the DI instance with a mock of Mockito etc. in the test code. I have a feeling.

The concrete source code was prepared on GitHub as one project together with the created test code. There are links to GitHub in some places, so please see that for details.

Tool introduction and evaluation

Now, I would like to introduce each tool and evaluate the generated test code.

TestMe

• https://plugins.jetbrains.com/plugin/9471-testme/
• IntelliJ plugin.
• Template type, automatically generate test code from source code according to fixed rules.
• The test code framework can be selected from JUnit 4, 5, etc.

The generated test code (partial) is below. The DI instance is mocked using Mockito. __Sounds good! __ Since the test code is only generated according to the template, there is almost no coverage rate, and it is necessary to develop it separately based on the generated test code. However, the speed of automatic generation is a few seconds, and it was not bad compared to creating test code from scratch.

Rating: Speed: ★★★★★ (several seconds) coverage: Customization: ★★ (Test framework can be selected)

Click here for test code and how to generate

class ZipCodeServiceTest {
    @Mock
    ZipCodeApi zipCodeApi;
    @InjectMocks
    ZipCodeService zipCodeService;

    @BeforeEach
    void setUp() {
        MockitoAnnotations.initMocks(this);
    }

    @Test
    void testSearch() throws IOException {
        when(zipCodeApi.search(anyString())).thenReturn(null);

        List<ZipCodeData> result = zipCodeService.search("zipCode");
        Assertions.assertEquals(
                Arrays.<ZipCodeData>asList(
                        new ZipCodeData("zipcode", "prefcode", "address1", "address2", "address3", "kana1",
                                        "kana2", "kana3")), result);
    }
}

Squaretest

• https://squaretest.com/
• IntelliJ plugin.
• Like TestMe, it is a template type, and test code is automatically generated according to the rules decided from the source code.
• The test code framework can be selected from JUnit 4, 5, etc.
• __ Paid license __. However, a 30-day trial is available.

Is it almost the same as TestMe? It's an impression, but you can edit the template on the setting screen. I want to use another library such as AssertJ or Hamcrest for assertions, so this area is easy to use. If it is not a paid license __, I would like to use it.

Rating: Speed: ★★★★★ (several seconds) coverage: Customization: ★★★★ (Select test framework, edit template)

Click here for test code and how to generate

class ZipCodeServiceTest {

    @Mock
    private ZipCodeApi mockZipCodeApi;

    private ZipCodeService zipCodeServiceUnderTest;

    @BeforeEach
    void setUp() {
        initMocks(this);
        zipCodeServiceUnderTest = new ZipCodeService(mockZipCodeApi);
    }

    @Test
    void testSearch() throws Exception {
        // Setup
        final List<ZipCodeData> expectedResult = Arrays.asList(
                new ZipCodeData("zipcode", "prefcode", "address1", "address2", "address3", "kana1", "kana2",
                                "kana3"));

        // Run the test
        final List<ZipCodeData> result = zipCodeServiceUnderTest.search("zipCode");

        // Verify the results
        assertEquals(expectedResult, result);
    }
}

EvoSuite

• http://www.evosuite.org/
• IntelliJ plugin. Test code can be automatically generated even by command line execution.
• However, a separate Jar file is required to generate the test code.
• Also, a separate runtime Jar is required to execute the test code.
• The test code is automatically generated by actually executing the software so that the coverage rate is high, not the template type.
• The test code framework is JUnit4.

Since the test code created from 2 was not good enough, the test code (part) of 1 is described below.

It takes 3 minutes to generate the test code. However, this area can be changed with the argument at the time of generation. Looking at the coverage rate of the test code ... __100% __ Amazing!

However, the generated test code is not readable and I don't feel like maintaining it manually later. Many annotations that I do not understand are given, and the point that it inherits the base class of EvoSuite is also a little unsatisfactory. For mock, take a look at 1 test code However, it was not set properly. This Issue says that it can be adjusted with arguments when generating test code, but an error occurs at runtime and it ends up being a mock. It was not possible to generate some test code. (Abandoned on the way)

However, the coverage rate is very good, so when creating a stateless and public API, it would be better to use it for finding bugs. I feel like.

Rating: Speed: ★★ (3 minutes) Coverage: ★★★★★ (100%!) Customization: ★★ (Some can be specified by arguments when generating test code)

Click here for test code and how to generate

@RunWith(EvoRunner.class) @EvoRunnerParameters(mockJVMNonDeterminism = true, useVFS = true, useVNET = true, resetStaticState = true, separateClassLoader = true, useJEE = true) 
public class NumberUtils_ESTest extends NumberUtils_ESTest_scaffolding {

  @Test(timeout = 4000)
  public void test00()  throws Throwable  {
      boolean boolean0 = NumberUtils.isOddNumberBetween1And50(50);
      assertFalse(boolean0);
  }
// omit other testcases.

Randoop

• https://randoop.github.io/randoop/
• Java library. Run it on the command line to generate test code.
• Similar to EvoSuite, the test code is automatically generated by actually executing the software so that the coverage rate is high.
• The test code framework is JUnit4.

What you can do is almost the same as Evosuite. Coverage was 100% as well. __ Good. __ It's not easy to use because there is only command line execution to generate test code, but I think it's good that runtime Jar is not required to execute the generated test code.

Rating: Speed: ★★ (a few minutes) Coverage: ★★★★★ (100%!) Customization: ★★ (Some can be specified by arguments when generating test code)

Click here for test code and how to generate

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class NumberUtilsTest {

    public static boolean debug = false;

    @Test
    public void test01() throws Throwable {
        if (debug)
            System.out.format("%n%s%n", "RegressionTest0.test01");
        // The following exception was thrown during execution in test generation
        try {
            boolean boolean1 = test.NumberUtils.isEvenNumberBetween1And10WithBadCode((-1));
            org.junit.Assert.fail("Expected exception of type java.lang.IllegalArgumentException; message: input should be between 1 and 10.");
        } catch (java.lang.IllegalArgumentException e) {
        // Expected exception.
        }
    }

Summary

So far, we have investigated and evaluated the four unit test automatic creation tools. To be honest, I feel that it is far from the image I expected, but I think it was good to know the current situation.

The source code, test code, library set, and HowTo ReadMe used in this study are stored on the following GitHub. If you would like to create test code using these tools, we would appreciate it if you could refer to it.

Investigation of auto generation unit tests with Java

Well, the long Advent calendar is over today. Thank you to everyone, both readers and writers! __ Have a nice year!