[PYTHON] A story about trying to improve the testing process of a system written in C language for 20 years

Introduction

A while ago, I tried to improve various tests of a system written in C language for 20 years, so I will write a little knowledge I got at this time.

• Caution Since I am creating an environment on my personal computer while remembering those days to write articles, it is actually different from the environment and version I did in practice. Also, there is some code in this article, but it's all just a fake example I thought of to write the article.

Creating test code for unit tests

With a system that has been running for 20 years, no one knows what it means anymore, but there are a number of places where existing behavior should not be changed.

If you have no choice but to work on such a part, the following methods are effective.

First, write the test code for the existing code. And after confirming that all pass, we will expand the function little by little. This makes it possible to proceed with the function addition work while confirming that the new function addition does not break the existing function.

It's a so-called ** test first **, but in such a conservative environment with a long history, if you do what you said in XP, various distortions will occur. We'll talk about that at the end of this chapter.

Introducing xUnit

You can do your best with assert etc., but it is better to use xUnit test framework such as CUnit.

• This time, I have verified only CUnit, but it seems that there are the following unit test libraries for C language in recent products.

** Cutter ** (1.2.7 released on 2019-09-13 is the latest) https://cutter.osdn.jp/index.html.ja

** UNITY ** (v2.4.3 released on 2017/11. Some commit history is 2019/10) http://www.throwtheswitch.org/unity https://github.com/ThrowTheSwitch/Unity

You can also test from xUnit frameworks for C ++ such as CPPUnit, but this time I removed it.

Why xUnit test framework is needed

It is better to use a well-known test framework because there are many examples of realization in the world and it will be helpful in case of trouble. Also, using the well-known test framework makes it easier to work with Jenkins.

For example, Jenkins's xUnit Plugin has a function to aggregate XML output by famous xUnit, and it is easy to automatically test. It is possible to aggregate the results of.

CUnit CUnit is an xUnit framework for C language that can be used from gcc or visual c.

How to build with visual studio 2019

Since there are only solutions from the VS2008 and 2005 eras, it is old and warnings come out.

1. Open CUnit-2.1-3 / VC9 / CUnit.sln
2. Create CUnit.h based on CUnit-2.1-3 / CUnit / Headers / CUnit.h.in. The differences are as follows.
3. Build in the order of libcunit-> other projects.

• If you build all at once and get an error, review the project dependencies. Also, if you set the number of simultaneous builds of the project to 1, you will be able to build normally.

** How to check the number of simultaneous builds of a project in Visual Stduio 2019 ** Select "Tools"-> "Options" from the menu to open the options dialog, and select "Projects and Solutions"-> "Build / Run" to display the corresponding screen.

How to build with make command

The following is an example of building using the make command under the environment of Windows 10 + Ubuntu 18.04. You can basically build it in the same way as the following article, but the version is 2.1-3.

** Unit test C program with CUnit ** https://qiita.com/muniere/items/ff9a984ed7e51eee7112

wget https://jaist.dl.sourceforge.net/project/cunit/CUnit/2.1-3/CUnit-2.1-3.tar.bz2
tar xvf CUnit-2.1-3.tar.bz2
cd CUnit-2.1-3
aclocal
autoconf
automake
./configure
make
make install

troubleshoot

No automake or autoconf command

Execute the following to install.

sudo apt-get install autoconf              

When I run configure or something, I get the error "configure: error: cannot find install-sh, install.sh, or shtool in". "" ./.. "" ./../ .. "".

Execute the following command and start over from autoconf

autoreconf -i

"Error: Libtool library used but'LIBTOOL' is undefined" occurred during configure

Install libtool with the following command.

sudo apt-get install libtool

If "error while loading shared libraries: libcunit.so.1: cannot open shared object file: No such file or directory" appears at runtime

Set the environment variable LD_LIBRARY_PATH.

export LD_LIBRARY_PATH=/usr/local/lib

If you want to make it persistent, write it in .bashrc as well.

Execution example

When executing from the console, it will be as follows.

#include <CUnit/CUnit.h>

int max(int x, int y) {
	if (x>y) {
		return x;
	} else {
		return y;
	}
}
int min(int x, int y) {
	/** bug  **/
	if (x>y) {
		return x;
	} else {
		return y;
	}
}

void test_max_001(void) {
	CU_ASSERT_EQUAL(max(5, 4) , 5);
}
void test_max_002(void) {
	CU_ASSERT_EQUAL(max(4, 5) , 5);
}
void test_max_003(void) {
	CU_ASSERT_EQUAL(max(5, 5) , 5);
}
void test_min_001(void) {
	CU_ASSERT_EQUAL(min(5, 4) , 4);
}
void test_min_002(void) {
	CU_ASSERT_EQUAL(min(4, 5) , 4);
}
void test_min_003(void) {
	CU_ASSERT_EQUAL(min(5, 5) , 5);
}

int main() {
	CU_pSuite max_suite, min_suite;

	CU_initialize_registry();
	max_suite = CU_add_suite("max", NULL, NULL);
	CU_add_test(max_suite, "test_001", test_max_001);
	CU_add_test(max_suite, "test_002", test_max_002);
	CU_add_test(max_suite, "test_003", test_max_003);

	min_suite = CU_add_suite("min", NULL, NULL);
	CU_add_test(min_suite, "test_001", test_min_001);
	CU_add_test(min_suite, "test_002", test_min_002);
	CU_add_test(min_suite, "test_003", test_min_003);
	CU_console_run_tests();
	CU_cleanup_registry();

	return(0);
}

See below for other functions for ASSERT. http://cunit.sourceforge.net/doc/writing_tests.html

Please link to libcunit.a when compiling the source that runs CUnit.

gcc unit.c  -Wall -L/usr/local/lib -lcunit -o unit

When executed from the console, it will be as follows.

How to work with Jenkins

Learn how to get CUnit results into Jenkins.

Make the test code output XML

Modify the code to create an XML file to pass to Jenkins. When outputting to an XML file, use CU_automated_run_tests and CU_list_tests_to_file instead of CU_console_run_tests.

/**Abbreviation**/
	/**Console output
	CU_console_run_tests();
	*/
	/**XML output**/
	CU_set_output_filename("./unit");
	CU_automated_run_tests();
	CU_list_tests_to_file();
	CU_cleanup_registry();
/**Abbreviation**/

If you change the above file, compile and execute it, the following XML file will be created instead of outputting to the console.

• unit-Listing.xml
• unit-Results.xml

See below for details on how to run CUnit. http://cunit.sourceforge.net/doc/running_tests.html#overview

Make Jenkins parse the test result XML.

Install xUnit Plugin in Plugin Management.

In the job settings, add "Publish xUnit test result report" to the post-build process and select CUnit-2.1.

Specify unit-Results.xml for Pattern.

When you run the build, you will get the following result:

Creating test stubs

When automating tests, if you prepare only the xUnit test framework ** and try to automate it, the introduction will often fail. Either you can only test simple utility functions and they're obsolete, or you end up writing complex test code to make the dependent libraries consistent.

For example, suppose you have library 2 that depends on library 1 as shown below.

When testing library 2, create a stub for testing library 1 so that it returns convenient data for testing library 2. That way, even if library 1 depends on the network and database, you can put it in a convenient state for testing without actually using them.

Example of creating a simple stub

Consider how to write a test for library 2 that depends on library 1 with a simple example.

Library 1 sample

Library 1 is implemented as follows. Prepare a function called add that just takes two arguments and returns the result of adding them.

static1.h

#ifndef STATIC1_H
#define STATIC1_H
extern int add(int x, int y);
#endif

static1.c

#include "static1.h"

int add(int x, int y) {
	return x + y;
}

Library 2 sample

Library 2 has the following implementation. In library 2, the proc function uses the add function of library 1 for processing.

static2.h

#ifndef STATIC2_H
#define STATIC2_H

#include "static1.h"

extern int proc(int x, int y, int z);

#endif

static2.h

#include "static2.h"

int proc(int x, int y, int z) {
	int a = add(x,y);
	return add(a, z);
}

Library 1 stub

Create stubs instead of real code. The processing of this stub performs the following processing each time the add function is executed. ・ Record the number of calls -Execution of the callback function registered in advance in the test code

stub_static.h

#ifndef STUB_static1_H
#define STUB_static1_H
#include "static1.h"

typedef int (*pfunc_add_def) ( int x , int y );

typedef struct {

    int count_add;
    pfunc_add_def pfunc_add;
} stub_data_def_static1;
extern stub_data_def_static1 stub_data_static1;
#endif

stub_static.c

#include "stub_static1.h"
stub_data_def_static1 stub_data_static1;


int add ( int x , int y ) {
    ++stub_data_static1.count_add;
    
    return stub_data_static1.pfunc_add( x , y );
    
}

Test code example

The following is an example of executing the proc function by defining the callback function executed from the stub of the add function in the test code.

#include <CUnit/CUnit.h>
#include "static2.h"
#include "stub_static1.h"

int test001_stub_add(int x, int y) {
	/** add()Stub. X and y for testing with CUNIT**/
	if (stub_data_static1.count_add == 1) {
		/**First call**/
		CU_ASSERT_EQUAL(x , 10);
		CU_ASSERT_EQUAL(y , 5);
		return 15;
	}
	if (stub_data_static1.count_add == 2) {
		CU_ASSERT_EQUAL(x , 15);
		CU_ASSERT_EQUAL(y , 4);
		return 19;
	}
	CU_FAIL("Incorrect call count of add");
	return -1;
}

void test001() {
	memset(&stub_data_static1, 0, sizeof(stub_data_static1));
	stub_data_static1.pfunc_add = test001_stub_add;
	int ret = proc(10, 5,4);
	
	/**Check the number of calls**/
	CU_ASSERT_EQUAL(stub_data_static1.count_add , 2);
	
	CU_ASSERT_EQUAL(ret , 19);
}


int main() {
	CU_pSuite suite;

	CU_initialize_registry();
	suite = CU_add_suite("stubsample", NULL, NULL);
	CU_add_test(suite, "test001", test001);
	CU_console_run_tests();
	CU_cleanup_registry();

	return(0);
}

If the stub function is called multiple times, it checks the value of the argument that the stub will receive based on the number of calls and determines the value that the stub should return.

Automatic stub creation

In the previous section, we introduced an example of a simple test code using a stub. The next issue is the cost of creating stubs.

It is not impossible to create it by hand, but if there are a large number of functions in library 1 or if they are updated frequently, it can be difficult to modify them by hand. So let's think about how to automatically generate stubs.

It's basically a simple conversion. Therefore, any programming language can automatically generate stubs, but if you want to have fun, a programming language that can use the template library and C language parser is recommended.

Try to automatically generate stubs in Python

C language parser library

You can analyze C / C ++ source code by using pycparser. https://github.com/eliben/pycparser

Note that this library only works for code that has been processed by the preprocessor. Therefore, in fact, use the result of preprocessor execution as shown below for the source code to be analyzed.

gcc -E static1.h

This time, we will parse the following precompiled header files.

Header file to be analyzed

extern int add(int x, int y);
typedef struct {
 int x;
 int y;
} in_data_t;
typedef struct {
 int add;
 int minus;
} out_data_t;
extern void calc(in_data_t* i, out_data_t *o);
extern int max(int data[], int length);

Sample to parse header file

import sys
from pycparser import c_parser, c_ast, parse_file

# https://github.com/eliben/pycparser/blob/master/examples/cdecl.py
def _explain_type(decl):
    """ Recursively explains a type decl node
    """
    typ = type(decl)

    if typ == c_ast.TypeDecl:
        quals = ' '.join(decl.quals) + ' ' if decl.quals else ''
        return quals + _explain_type(decl.type)
    elif typ == c_ast.Typename or typ == c_ast.Decl:
        return _explain_type(decl.type)
    elif typ == c_ast.IdentifierType:
        return ' '.join(decl.names)
    elif typ == c_ast.PtrDecl:
        quals = ' '.join(decl.quals) + ' ' if decl.quals else ''
        return quals + _explain_type(decl.type) + '*'
    elif typ == c_ast.ArrayDecl:
        arr = 'array'
        if decl.dim: 
            arr = '[%s]' % decl.dim.value
        else:
            arr = '[]'

        return _explain_type(decl.type) + arr

    elif typ == c_ast.FuncDecl:
        if decl.args:
            params = [_explain_type(param) for param in decl.args.params]
            args = ', '.join(params)
        else:
            args = ''

        return ('function(%s) returning ' % (args) +
                _explain_type(decl.type))

    elif typ == c_ast.Struct:
        decls = [_explain_decl_node(mem_decl) for mem_decl in decl.decls]
        members = ', '.join(decls)

        return ('struct%s ' % (' ' + decl.name if decl.name else '') +
                ('containing {%s}' % members if members else ''))

def show_func_defs(filename):
    # Note that cpp is used. Provide a path to your own cpp or
    # make sure one exists in PATH.
    ast = parse_file(filename, use_cpp=False,
                     cpp_args=r'-Iutils/fake_libc_include')
    if not isinstance(ast, c_ast.FileAST):
        return

    for ext in ast.ext:
        if type(ext.type) == c_ast.FuncDecl:
            print(f"function name: {ext.name} -----------------------")
            args = ''
            print("parameters----------------------------------------")
            if ext.type.args:
                for arg in ext.type.args:
                    print(f"{_explain_type(arg)} {arg.name}")
            print("return----------------------------------------")
            print(_explain_type(ext.type.type))

if __name__ == "__main__":
    if len(sys.argv) > 1:
        filename  = sys.argv[1]
    else:
        exit(-1)

    show_func_defs(filename)

As a simple flow of the analysis process, FileAST can be obtained as the execution result of parse_file. By parsing the ext property there, you can get the information of the function defined in the file. The result of executing the above program and checking the function definition in the header file is output as follows.

C:\dev\python3\cparse>python test.py static1.h
function name: add -----------------------
parameters----------------------------------------
int x
int y
return----------------------------------------
int
function name: calc -----------------------
parameters----------------------------------------
in_data_t* i
out_data_t* o
return----------------------------------------
void
function name: max -----------------------
parameters----------------------------------------
int[] data
int length
return----------------------------------------
int

It was confirmed that C language source files can be easily analyzed by using pycparser in this way.

Template library

When creating header and source files for stubs, the template library makes it easier to create. Please refer to the following for the template library in Python.

** I want to output HTML in Python for the first time in a while, so check the template ** https://qiita.com/mima_ita/items/5405109b3b9e2db42332

This time I am using Jinja2.

Code for creating stubs

The following is an example of implementing a stub creation tool using a C language parser and template library.

create_stub.py

import sys
import os
from pycparser import c_parser, c_ast, parse_file
from jinja2 import Template, Environment, FileSystemLoader


stub_header_tpl = """
#ifndef STUB_{{name}}_H
#define STUB_{{name}}_H
#include "{{name}}.h"
{% for func in function_list %}
typedef {{func.return_type}} (*pfunc_{{func.name}}_def) ({% for arg in func.args %}{% if loop.index != 1 %},{% endif %} {{arg.typedef}} {{arg.name}}{{arg.array}} {% endfor %});
{% endfor %}

typedef struct {
{% for func in function_list %}
    int count_{{func.name}};
    pfunc_{{func.name}}_def pfunc_{{func.name}};
{% endfor %}
} stub_data_def_{{name}};
extern stub_data_def_{{name}} stub_data_{{name}};
#endif
"""

stub_source_tpl = """
#include "stub_{{name}}.h"
stub_data_def_{{name}} stub_data_{{name}};

{% for func in function_list %}
{{func.return_type}} {{func.name}} ({% for arg in func.args %}{% if loop.index != 1 %},{% endif %} {{arg.typedef}} {{arg.name}}{{arg.array}} {% endfor %}) {
    ++stub_data_{{name}}.count_{{func.name}};
    {% if func.return_type != "void" %}
    return stub_data_{{name}}.pfunc_{{func.name}}({% for arg in func.args %}{% if loop.index != 1 %},{% endif %} {{arg.name}} {% endfor %});
    {% else %}
    stub_data_{{name}}.pfunc_{{func.name}}({% for arg in func.args %}{% if loop.index != 1 %},{% endif %} {{arg.name}} {% endfor %});
    {% endif %}
}
{% endfor %}
"""


class ParameterData():
    def __init__(self, name, typedef):
        self.__name = name
        self.__typedef = typedef
        self.__array = ""
        if '[' in typedef:
            self.__array = "[" + typedef[typedef.index("[")+1:typedef.rindex("]")] + "]"
            self.__typedef = typedef[0: typedef.index("[")]

    @property
    def name(self):
        return self.__name

    @property
    def typedef(self):
        return self.__typedef

    @property
    def array(self):
        return self.__array

class FunctionData:
    def __init__(self, name, return_type):
        self.__name = name
        self.__return_type = return_type
        self.__args = []

    @property
    def name(self):
        return self.__name

    @property
    def return_type(self):
        return self.__return_type

    @property
    def args(self):
        return self.__args


# https://github.com/eliben/pycparser/blob/master/examples/cdecl.py
def _explain_type(decl):
    """ Recursively explains a type decl node
    """
    typ = type(decl)

    if typ == c_ast.TypeDecl:
        quals = ' '.join(decl.quals) + ' ' if decl.quals else ''
        return quals + _explain_type(decl.type)
    elif typ == c_ast.Typename or typ == c_ast.Decl:
        return _explain_type(decl.type)
    elif typ == c_ast.IdentifierType:
        return ' '.join(decl.names)
    elif typ == c_ast.PtrDecl:
        quals = ' '.join(decl.quals) + ' ' if decl.quals else ''
        return quals + _explain_type(decl.type) + '*'
    elif typ == c_ast.ArrayDecl:
        arr = 'array'
        if decl.dim: 
            arr = '[%s]' % decl.dim.value
        else:
            arr = '[]'

        return _explain_type(decl.type) + arr

    elif typ == c_ast.FuncDecl:
        if decl.args:
            params = [_explain_type(param) for param in decl.args.params]
            args = ', '.join(params)
        else:
            args = ''

        return ('function(%s) returning ' % (args) +
                _explain_type(decl.type))

    elif typ == c_ast.Struct:
        decls = [_explain_decl_node(mem_decl) for mem_decl in decl.decls]
        members = ', '.join(decls)

        return ('struct%s ' % (' ' + decl.name if decl.name else '') +
                ('containing {%s}' % members if members else ''))

def analyze_func(filename):
    ast = parse_file(filename, use_cpp=False,
                     cpp_args=r'-Iutils/fake_libc_include')
    if not isinstance(ast, c_ast.FileAST):
        return []
    function_list = []
    for ext in ast.ext:
        if type(ext.type) != c_ast.FuncDecl:
            continue
        func = FunctionData(ext.name,  _explain_type(ext.type.type))
        if ext.type.args:
            for arg in ext.type.args:
                param = ParameterData(arg.name, _explain_type(arg))
                func.args.append(param)
        function_list.append(func)
    return function_list


if __name__ == "__main__":
    if len(sys.argv) != 3:
        print("python create_stub.py preprocessed header output folder")
        exit(-1)
    filename  = sys.argv[1]
    dst_folder  = sys.argv[2]
    function_list = analyze_func(filename)
    if len(function_list) == 0:
        print("Could not find function")
        exit(-1)
    data = {
        'name' : os.path.splitext(filename)[0],
        'function_list' : function_list
    }
    #Creating a stub header file
    with open(f"{dst_folder}/stub_{data['name']}.h", mode='w', encoding='utf8') as f:
        f.write(Template(stub_header_tpl).render(data))

    #Creating a stub source file
    with open(f"{dst_folder}/stub_{data['name']}.c", mode='w', encoding='utf8') as f:
        f.write(Template(stub_source_tpl).render(data))

The stubs created using this tool are as follows.

stub_static1.h

#ifndef STUB_static1_H
#define STUB_static1_H
#include "static1.h"

typedef int (*pfunc_add_def) ( int x , int y );

typedef void (*pfunc_calc_def) ( in_data_t* i , out_data_t* o );

typedef int (*pfunc_max_def) ( int data[] , int length );


typedef struct {

    int count_add;
    pfunc_add_def pfunc_add;

    int count_calc;
    pfunc_calc_def pfunc_calc;

    int count_max;
    pfunc_max_def pfunc_max;

} stub_data_def_static1;
extern stub_data_def_static1 stub_data_static1;
#endif

stub_static.c

#include "stub_static1.h"
stub_data_def_static1 stub_data_static1;


int add ( int x , int y ) {
    ++stub_data_static1.count_add;
    
    return stub_data_static1.pfunc_add( x , y );
    
}

void calc ( in_data_t* i , out_data_t* o ) {
    ++stub_data_static1.count_calc;
    
    stub_data_static1.pfunc_calc( i , o );
    
}

int max ( int data[] , int length ) {
    ++stub_data_static1.count_max;
    
    return stub_data_static1.pfunc_max( data , length );
    
}

When the C language parser is not available

There are often situations where Python is not available or the C parser is not available. In that case, use doxygen.

Doxygen can parse the source file and output the function information to XML. It is also possible to create a file for stubs based on this XML. Many C and C ++ sites use doxygen, so it's probably a lower barrier to adoption than Python. *

• I used to make stubs with Doxyge + VBS.

How to create other stubs and mock

I didn't actually use it, but there are other methods as follows.

CMock https://github.com/ThrowTheSwitch/CMock It seems to create code for mock using Ruby. It may be possible if Ruby can be used in the environment.

** Ideas for using stub functions in C testing ** https://qiita.com/HAMADA_Hiroshi/items/e1dd3257573ea466d169 "How to perform unit tests by stubing functions in the same source without modifying the existing source", basic idea Uses macros to replace existing functions with aliases. Fortunately, I didn't have to write a test to call a function in the same source, so I didn't actually use it.

How to measure coverage rate

When you write test code, you can judge whether you are executing the test without exception by measuring the ratio covered by the test code. Fortunately, gcc comes with a tool called gcov to measure coverage.

10 gcov—a Test Coverage Program https://gcc.gnu.org/onlinedocs/gcc/Gcov.html

** How to use gcov ** https://mametter.hatenablog.com/entry/20090721/p1

Coverage measurement example

Let's measure the coverage with the example of the stub used earlier. This time, in order to intentionally create an unreachable route, modify static2.c of library 2 as follows.

static2.c

#include "static2.h"

int proc(int x, int y, int z) {
	int a = add(x,y);
	return add(a, z);
}

int proc2() {
	/**Not reached**/
	return 0;
}

Then compile with the -coverage option to measure coverage.

gcc -c -Wall -Wextra static2.c -coverage
ar r libstatic2.a static2.o 
gcc -c -Wall -Wextra stub_static1.c
ar r libstub_static1.a stub_static1.o
gcc test.c -coverage  -Wall -Wextra -L. -lstatic2 -lstub_static1   -L/usr/local/lib -lcunit -o test

Executing the above command will create the following files.

• static2.gcno
• test.gcno
• test

After that, running test will create the following files.

• static2.gcda
• test.gcda

To check the coverage of static2.c under test, execute the following command.

gcov static2.gcda 

When this command is executed, the following message will be displayed.

File 'static2.c'
Lines executed:60.00% of 5
Creating 'static2.c.gcov'

Here you can see that the coverage rate is 60%, and to find out where it is not covered, display static2.c.gcov. In this example, you can see that proc2 is not passing.

Other automated tests

About the effect

We judge that it is effective in the following points.

・ The risk of expanding the function without changing the behavior of the current function can be suppressed. -By writing a test code, you can clarify the functions that were in the dark until now. ・ If you write test code, you can detect defects that have been sleeping in existing code. → Whether or not to fix it is another matter. A balance between the frequency of occurrence, the degree of impact at the time of occurrence, and the correction cost.

However, there are some issues, so let's consider them in the next section.

What about SQL exams?

In a legacy environment, you may have to write SQL in the source code and actually connect it to a database and run it for testing. In this case, it is desirable to divide the DB area used by each developer and perform unit tests.

However, in a large organization, there are departments that specialize in DB, and it is not easy to tamper with the database. Also, if you put the database itself in the local environment, a poor PC becomes a bottleneck, or even if you break through it, if you leave the database management in your personal environment, some developers will continue to use the old environment. There is also a risk of doing so.

In such a case, you can handle it by a less conscious method of rolling back when the automatic test is over.

① Start automatic test ② Start a DB transaction ③ Store test data in DB ④ Perform an automatic test ⑤ Check how the DB has changed based on the results of the automatic test. ⑥ DB rollback

This method is not a perfect idea, but it is easy to implement.

Massive combination testing

If you write the input value and expected value in an array of structures, you can do a lot of tests. A common method is to write a list of inputs and outputs in Excel, convert it to an array of C language structures with a VBA macro, and write it in the test code.

Distortion doing test first in a conservative waterfall environment

If this method is used in a conservative waterfall environment, after implementation, you will create test specifications and checklists for unit tests and perform tests. However, if you do test-first things, the implementation will be completed = the unit test will be completed, so distortion will occur here.

Plans don't get on the traditional line

With the conventional method, the schedule line is described in three steps: implementation, creation of a checklist for unit tests, and implementation of unit tests. And with a highly controlled, complete and perfect organization, each process is timed exactly.

If you do test-first things, unit tests cannot be separated from the implementation, so individual work time cannot be measured, and you will report them separately.

Unit test specifications

You need to decide what to do with unit test specifications and checklists that were created in the traditional way. It would be nice if we could decide that the code review of the test code would be OK and that the product of the unit test would be the test code, but in a legacy environment, documents are important. In this case, you need to create test specifications and checklists from the test code.

The first countermeasure is to manually post the contents of the test code to the checklist. The problem is that the test code and test specifications get out of sync. Also, this method is a very bad response that eliminates the advantage of running tests automatically, but it is often adopted because anyone can do it if it takes time.

The second method is to generate a test specification from the comments in the test code. Describe the content of the test in the comment of the test code and summarize the results extracted by Doxygen etc. in the checklist. In this case, there is a risk that there will be a discrepancy between the comment and the actual processing.

The third method is to write Japanese that does not feel strange even if it is output to the checklist for the test name specified in the test code.

	CU_add_test(max_suite, "max(a,b)To run a=5,b=In case of 4, confirm that the value of a is obtained", test_max_001);

If you execute the test with the above description, the following will be output to the XML of the test result.

After that, output the XML checklist and you're done. The problem with this method is that the check contents performed by CU_ASSERT_EQUAL etc. are not output to XML, so the granularity of the checklist becomes large.

The fourth method is to create a function that wraps CU_ASSERT_EQUAL etc., and within that function, output the inspection contents to a file in a form that is easy to output to the checklist.

Regardless of which method you choose, be aware that sticking to the document can be a hassle. → In a culture that sticks to writing, let's make sure that such work is included in the estimate.

Handling of failure slips in unit tests

Since the unit tests are finished when the implementation is finished, it is impossible to raise the failure in the unit tests. However, a highly controlled, complete and perfect organization may focus on unit test failures. In this case, the problem that occurred during implementation will be appropriately made up as a bug like that.

The theory that should be supposed to be is to persuade and create new rules, but I think it is impossible because the culture is different.

The digits of the number of tests change, so conventional indicators become useless

Introducing automated testing makes it possible to mechanically process a large number of combinatorial tests. As a result, the number of tests performed automatically is larger than the number of tests performed by the conventional method. (That's an order of magnitude)

What used to be at most hundreds of checklist items is now usually a thousand or a million. I'm talking about "what's wrong with doing a lot of tests?", But if it is a highly controlled, complete and perfect organization, it will be treated as an abnormal value, so adjust the particle size appropriately and use it for reporting. It may be necessary to adjust the number to that level.

In addition, if you want to take proper measures, it would be a good idea to create a new index as a separate tabulation for automated testing.

Since the number of failures after the integration test is less than before, it is treated as an abnormal value.

If you do a comprehensive unit test, you will not have any problems with the integration test.

This is a problem with this. For example, let's say you've had dozens of failures in the past, but the number of defects is almost gone. If you're only looking at the numbers, you can't help wondering if the integration test perspective is wrong.

Many people don't want to write test code

As I wrote below, that's what it is. Let's give up.

** Automation of unit tests using unconscious Visual Studio ** https://qiita.com/mima_ita/items/05ce44c3eb1fd6e9dd46

It is possible to explain the effect of the automatic test by man-hours and quality, convert it into a monetary amount and raise a sense of crisis, and prepare educational materials, but honestly, it is a culture that can not be understood even if such a story is accumulated Does not work ** at all ** </ font>.

sudo apt-get install cppcheck

$ cppcheck --enable=all .
Checking cov.c ...
1/7 files checked 14% done
Checking main.c ...
2/7 files checked 28% done
Checking static1.c ...
3/7 files checked 42% done
Checking static2.c ...
4/7 files checked 57% done
Checking stub_static1.c ...
5/7 files checked 71% done
Checking test.c ...
6/7 files checked 85% done
Checking unit.c ...
7/7 files checked 100% done
[cov.c:7]: (style) The function 'bar' is never used.
[static1.c:7]: (style) The function 'calc' is never used.
[static2.c:8]: (style) The function 'proc2' is never used.
(information) Cppcheck cannot find all the include files (use --check-config for details)

cppcheck --enable=all --xml --xml-version=2 ~/libsample/. 2>cppcheck.xml