[PYTHON] The right way of thinking for testing, dedicated to projects where testing doesn't work

Who is the target of this article

--I'm writing a test on a project. (I have written) ――I know that testing seems to be important, but I haven't realized the benefits of testing so much. --After all, we are fixing bugs that rely on manual testing.

Introduction

I had a lot of knowledge about test design methods and implementations, but what I didn't know was the idea of testing.

I think many people know that testing seems to be important. However, I think there are some people who have not actually realized the benefits. In fact, some people say *** testing is important, while others know that *** testing seems important. The latter person can write a test for the time being. However, I feel that I often rely on finding bugs in manual tests in the end, despite the time I spend on testing.

It is natural to write a test in the world, and the test is important! Why can't we realize that testing is important? ?? How can I make a project that makes effective use of tests?

In this article, how do you think about such problems in practice rather than in a tricky way? I would like to explain the thought circuit.

For example, consider the following case.

Case Study

class SampleController
  def index
    # do something
  end
end

In this action, 10 methods A, B, C, D, E, F, G, H, I, J are called, each returning (0, 1), and there are 1024 behaviors. To do.

Here, there is a report that it does not behave normally in the following situations.

Report 1

A -> 0 
B -> 1 
C -> null
D -> 1 
E -> 1 
F -> 1 
G -> 0 
H -> 0 
I -> 0 
J -> - 2

Report 2

A -> 0 
B -> 1 
C -> 0 
D -> 1 
E -> 1 
F -> 1 
G -> 0 
H -> 0 
I -> 0 
J -> 1

What tests should I have done to prevent the bug from recurring? ??

Here, in a project where only unit tests are written, it may be possible to prevent the bug in Report 1, but it may be difficult to prevent the bug in Report 2. What kind of test case should I write based on this case? I would like to explain that.

Premise

What is the purpose of the test in the first place, rather than suddenly entering the methodology? What is the ideal test? What are the problems with the ideal test? I would like to unify the interpretation in that respect.

--The purpose of the test ――What should you do to prevent bugs? ――What is the ideal test? --Ideal test problems

Purpose of the test

The purpose of the test is

1. Finding a bug
2. Guarantee quality
3. Improving quality

It is said that. The most important purpose of this is *** 1. Finding bugs ***. So, to the extreme, a bug-free application is ideal, and testing is one of the best ways to get closer to that ideal.

What should I do to prevent bugs?

I think it can be divided into three areas: specifications, program design, and testing. If the specifications and program design are bad, the test will be difficult. So refactor your spaghetti code before testing. But this time it's about testing.

What is an ideal test?

So what is the ideal test? It is a test that proves that *** all use cases work properly ***.

In other words, if you have 100 different uses, you can find the bug perfectly by testing if the 100 responses are correct. I will explain using the previous case.

Case Study

class SampleController
  def index
    # do something
  end
end

Within this action, 10 methods A, B, C, D, E, F, G, H, I, J are called and there are 1024 use cases. Also assume that each method returns (0, 1).

Here, I got a report that a bug occurred in the following situations.

A -> 0 
B -> 1 
C -> 0 
D -> 1 
E -> 1 
F -> 1 
G -> 0 
H -> 0 
I -> 0 
J -> 1

Answer (ideal position)

First of all, I don't know if the combination is causing a bug or if a single method is causing a bug. However, this time, I will ignore such a situation and consider the test case from an ideal position.

In an ideal position, test that all combinations are normal. In other words, 2 ^ 10 = 1024 tests will help you find the bug perfectly.

Problems in an ideal position

A function with 1024 use cases is a monster, but in reality it is impossible to test hundreds of functions for one action (function) of an ordinary application. Also, when multiple actions (functions) are combined, an infinite number of test cases will be required. The problem here is that it is man-hours impossible to test all use cases. (Of course)

Realistic test design method

The more you aim for the ideal test, the more man-hours will increase exponentially. So in the world of software testing, *** How can you write a cost-effective test? *** *** That becomes a proposition.

So how do you write a cost-effective test? It's *** writing solid tests for buggy code ***. To put it the other way around, *** decimate non-essential conditions from the ideal test ***. Now, I will explain three ways to thin out the ideal test case to a realistic level.

--Method 1: Thin out independent elements --Method 2: Do not test less important patterns --Method 3: Narrow down combinations based on statistical information --Summary

Method 1: Thin out independent elements

This technique is the most basic idea that makes testing realistic.

Case Study

class SampleController
  def index
    # do something
  end
end

Within this action, 10 methods A, B, C, D, E, F, G, H, I, J are called and there are 1024 use cases. Also assume that each method returns (0, 1).

Proposition 1:If "1024 use cases are all normal", then "10 methods are all normal"
Proposition 2:If "10 methods are all normal", then "1024 use cases are all normal"

Are Proposition 1 and Proposition 2 true or false respectively?

Answer

Proposition 1 is "true" and Proposition 2 is "false". (It seems that you can write a program that is true even in the latter if you use a functional language)

The former requires 2 ^ 10 = 1024 tests, The latter requires 2 * 10 = 20 tests.

These propositions prove that *** unit tests alone are not enough to find bugs ***. However, in reality, thinning out may be sufficient.

For example, if the D and H methods are completely independent (does not affect other methods) out of the 10 methods, it can even be proved that D and H return 0 or 1. There is no problem if you do. In addition, E and F also have narrow scopes of influence, and it can be judged that they are not necessary for combination testing.

This means that the use case index is reduced from 10 to 6 and the unit tests are increased by 4. 2 ^ 6 + 2 * 4 = 70 ways

In this way, we will thin out independent combinations.

point

Unit tests are sufficient for independent methods and are subtracted from the exponent of the combination.

Method 2: Do not test less important patterns

This cannot be determined theoretically. On a case-by-case basis, the error will be within an acceptable range just by knowing the idea.

Case Study

class SampleController
  def index
    # do something
  end
end

Within this action, 10 methods A, B, C, D, E, F, G, H, I, J are called and there are 1024 use cases.

Also, the method A is the authentication function provided by the library. 1 for authentication Returns 0 for non-authentication. The method F returns the values'red','green' and changes the color of the button. The method H returns the Plan A object and Plan B object from the information entered by the user.

In this, there should be a hierarchy of cases that should be tested and cases that are allowed without testing. So how should you prioritize your tests? ??

Answer

Judgment is based on the matrix of (importance) x (risk of bug contamination).

• 3rd: Narrowing down test items: Be sure to express the weighting numerically In the article, software tycoons give their opinions.

Importance has factors such as importance to the user and a wide range of influence. The risk of bugs is higher in places where the source code is more complicated by design. (Places with a high degree of coupling) The risk of bug contamination can be determined quantitatively to some extent by performing metric analysis. (I will talk about metric analysis later)

So, in this case, A is provided by a trusted library and the risk of bugs is low. F The range of influence is narrow, and it is not so important for users. H is the value of the application itself, and the logic is complicated. In other words

With this kind of matrix, we can determine that A does not need to be unit tested and F does not need to be in a combination pattern.

point

Considering (importance) × (risk of bug contamination), thin out the parts that do not need to be included in the code and combination patterns that do not require unit tests.

Method 3: Narrow down combinations based on statistical information

What kind of test cases should be thinned out so far? I learned the basic idea. However, with human power, even dozens of test cases are strict for one function. So when can test engineers get bugs? I made that into statistical information. Then, problems that occur between *** 2 parameters account for 70 to 90%. The statistical result of *** appeared. A test design method based on this data is called the *** pairwise method ***.

Case Study

class SampleController
  def index
    # do something
  end
end

Within this action, three methods A, B, C and D are called and there are eight use cases.

The pairwise method is as follows.

In the all-pair (pairwise) method, which is one of the combination test techniques, a test pattern is created so that at least the combination of values between two parameters is covered for all parameters.

All pair (pairwise) method

Is written.

In other words, it's okay if all the combinations of the two methods are covered. For example, if you consider a test case using the pairwise method for the methods A, B, C, and D, it will be as follows.

A       B       C       D
1       0       0       0
1       1       1       1
0       1       0       1
0       0       1       1
0       1       1       0

It seems that there is a tool to make pairwise, so you may try using it.

Combination test case generation tool "PictMaster" and the topic of software testing

Summary

Method 1, find an independent pattern. Method 2, think in the matrix if you really should test. Method 3, wisely reduce the number of combinations based on statistics.

I think that would be a realistic combination.

Good knowledge to know when writing tests in practice

With the above, I was able to acquire the basic idea. However, the cases so far have been case studies based on some idealized models. In previous cases, the combination was very clear. However, in the field code, the number of combinations varies from person to person. Also, what kind of code is buggy in some people? There will be variations in the interpretation.

However, when developing a project, we want to have a common understanding as much as possible. Therefore, I would like to explain my knowledge about the following items. (Skip the items you know.)

--Coverage rate --Equivalence division and boundary value analysis --Metric analysis

Coverage rate is a method of counting route combinations. Equivalence partitioning and boundary value analysis are methods of extracting patterns of values. Metrics analysis is a method of quantitatively measuring what kind of code has a bug.

By knowing these, there will be some variation in combination selection and priority among projects.

What is coverage rate?

An index that shows how much the branch in the program is covered. The coverage rate has levels C0, C1, and C2.

Let's take an example of this code.

def my_method
  if type1 == "A"
    print("Process 1")
  else
    print("Process 2")
  end

  if type == "B"
    print("Process 3")
  end
end

C0: Instruction coverage

At the C0 level, it determines whether the instructions are covered. In other words, it is okay if you pass process 1 to process 3 once.

def my_method
  if type1 == "A"
    print("Process 1")
  else
    print("Process 2")
  end

  if type == "B"
    print("Process 3")
  end
end

The following two test cases can be said to have a C0 level coverage rate of 100%.

type == "A" TRUE,  type == "B" TRUE
type == "A" FALSE, type == "B" TRUE

C1: Branch coverage

At the C1 level, determine if the branch is exhaustive. In other words, it's okay to go through all the branches once.

def my_method
  if type1 == "A"
    print("Process 1")
  else
    print("Process 2")
  end

  if type == "B"
    print("Process 3")
  end
end

The following four test cases can be said to have a C1 level coverage rate of 100%. 2 + 2 = 4 ways

type == "A" TRUE,  type == "B" TRUE
type == "A" FALSE, type == "B" TRUE
type == "A" TRUE,  type == "B" FALSE
type == "A" FALSE, type == "B" FALSE

C2: Coverage of conditions

At the C2 level, determine if all branch combinations are covered. This is hard because it's pretty close to the ideal test.

def my_method
  if type1 == "A"
    print("Process 1")
  else
    print("Process 2")
  end

  if type == "B"
    print("Process 3")
  end
end

There are four test cases below, but in reality, the number of test cases continues to increase exponentially as the conditions increase. 2 * 2 = 4 ways

type == "A" TRUE  && type == "B" TRUE
type == "A" FALSE && type == "B" TRUE
type == "A" TRUE  && type == "B" FALSE
type == "A" FALSE && type == "B" FALSE

Equivalence split and boundary value analysis

By the way, we have found that the combinations are counted according to the number of conditional branches. However, there is still another combination of variables. That is the range of values that can be taken. (Range)

Case Study

Consider the following code testing strategy.

#Possible values for var are String, Int,One of nill
def my_method(var)
  var.your_method
end

For coverage, one time is enough. However, it is not enough to consider only the coverage rate. In this case, you'll want to at least test the String, Int, and nill cases.

*** If coverage is route coverage, this time it is value coverage. *** *** However, the coverage of values is endless. (Natural numbers alone) So how do you cover it? ??

Equivalence split

As usual, I will leave the explanation to other easy-to-understand articles.

What is equivalence division?

Boundary value analysis

As the name implies, it's about testing boundary values.

What is metric analysis?

It statically analyzes software and quantitatively evaluates quality from various perspectives. By knowing this idea, *** What kind of source code has many bugs? You can see that *** to some extent. I will put a reference link below for a detailed explanation, but I will explain only the index called cyclomatic complexity.

What is cyclomatic complexity?

A way to show complexity by counting the number of routes in your source code. However, this article is overwhelmingly easy to understand, so please take a look.

\ It's a bug! / What is the comfort and cyclomatic complexity from the perspective of Mr. Bug, who nests in the system? \ It's a bug! /

Reference link

What is metric analysis? Ruby on Rails | metric analysis with metric_fu What is the static code analysis tool MetricFu looking at

Metrics analysis tool for each language First Software Metrics (Part 1): Quantify and Confirm Software Quality

Read when you want to choose a method for selecting test cases.

[Read "Software Testing Learned from Zero Knowledge" [From White Box Testing to Exploratory Testing]](http://qiita.com/shinofumijp@github/items/6121105c608fc0cf11d4#%E5%90%8C%E5%80 % A4% E5% 88% 86% E5% 89% B2% E6% B3% 95% E3% 81% A8% E5% A2% 83% E7% 95% 8C% E5% 88% 86% E6% 9E% 90 % E6% B3% 95)

Q&A

I wrote a lot. However, it is difficult for all project members to maintain the quality of the test. Therefore, I would like to write the issues that have come up so far in a Q & A format.

Question 1: How should I use the unit test and the E2E test properly?

Ideally, unit test the behavior of all functions and methods, All combinations of them are to be tested with the E2E test. But in reality it's tough. Therefore, we do not unit test functions and methods with low complexity. Independent combinations are decimated from the test case. Test design based on the idea.

Updated from time to time