A general approach

Unit testing is a test activity which is part of software development. The development environment as well as the development activities are both important aspects when it comes to testing the code. This document explains how test activities are be integrated into development activities and shows how different techniques and tools can be exploited. The main course of developing software can be outlined as follows:

Figure 1.1: General approach to software development

Coding & static analysis

During the coding stage, the design model is implemented. To ensure that the code complies to programming standards, automated static analysis is applied using static analysis tools such as QAC/QACPP or splint. These are static analysis tools that find vialations agains predefined rules automatically. Static analysis means that the code under test, does not have to be executed to allow for finding defects. It helps to improve the correctness, stability, portability and maintainability of the source code.
Chapter 8 will address the use of QAC for doing static analysis in detail. Chapter 9 will address the use of splint in more detail.

Code review

When the code has been completed

walkthroughAn informal meeting in which a developer or author of a document explains his tasks and approach. Usually organized somewhere in the begin of a new task. The key idea of a code review is that it helps to assure consistency of the source code with the (intended) design. Also the dynamic behavior of the code can be examined more thoroughly than any static analysis tool can do. When, during static analysis, it has been decided to suppress warnings for special cases in the source code, these suppressions are also evaluated during a code review. When the issues of the code review have been solved, dynamic tests are developed.
The code review is elaborated upon in 3.

Test design

The most important challenge of dynamic testing is to design effective test cases. By definition, an effective test has a high probability of finding errors[1]. In general, more defects can be found when the amount of code that is exercised by the test increases[2]. Research at different development plants has demonstrated that the actual percentage of the code that is tested by developers is about 40% on average only. Also, it has been experienced that developers can easily increase this percentage by using coverage analysis tools. Developers were able to increase the effectiveness of there tests by using coverage analysis tools.. Coverage analysis makes it possible to see how much of the software (and, in more detail, which parts of the software) is actually exercised by the test. It helps the developer to suggest new test inputs that will bring about more coverage (i.e. tests the code more thoroughly). Without coverage analysis tool, it can be very difficult to design testinputs that achieve more than 40% coverage.

Figure 1.2: Relation between the number of detected defects and the amount of achieved code (branch) coverage with unit testing. This figure has been published by Malaiya[3] and Pasquini[4]

There doesn't exist a universal minimal percentage of coverage each test case must achieve. This is very much related to the risk that is involved particular piece of software and the costs that comes with testing that software.

Adding verification points

When sufficient coverage is achieved by the test design, this test design is used for various types of tests. Firstly, the outcome of the tests are examined to see whether the result or effect that is brought about by the test input is correct. Also, the code can be instrumented to pinpoint defects related to non-functional behavior of the code (i.e. memory leaks, misuse etc.). Frequently, dynamic analysis tool like insure   or purify   can be used for this purpose. Finally, the findings of the various tests are documented and archived. The test design activity is discussed further in 4.

Drivers and stubs

stub A stub is a small program routine that substitutes for a longer program, possibly to be loaded later or that is located remotely. For example, a program that uses Remote Procedure Calls (RPC) is compiled with stubs that substitute for the program that provides a requested procedure. The stub accepts the request and t hen forwards it (through another program) to the remote procedure. When that pr ocedure has completed its service, it returns the results or other status to the stub which passes it back to the program that made the request.

A problem which often discourages developers to employ unit test activities, is lack of testability of the software. In order to execute any of the software, other parts of the software (i.e. the software environment of that particular unit under test) are required. Because the software environment that is required to dynamically test a certain unit is developed simultaneously or is difficult to operate in practice, dynamic unit testing might become a difficult job to do.

Figure 1.3: A unit and its software environment

A solution to this is creating an artificial environment that simulates the intended software environment of the unit. The artificial environment is called stub. In order to call the functions defined by the unit, a so-called driver must be created. A driver is a piece of software which calls the functions in the unit under test. A driver is normally used to execute the dynamic unit tests. driverA driver is a piece of software which calls the functions in the unit under test. Normally it is used to execute the dynamic unit tests.

Figure 1.4: A unit and its stubs and driver

Creating drivers and stubs can be very time consuming. When every unit has to be tested in isolation, stubs and drivers need to be developed for every unit. Hence, the amount of software actually developed might grow to double the size of the software that is actually delivered to the customer. Of course, it is important to safe time and money by means of testing instead of wasting it. Therefore, development of testware must be very pragmatic. The next aspects must be judged when deciding to develop stubs and drivers.

The need for thorough testing

 
Is the risk of failure really that high?

The availability of the original software environment

 
Is the intended software environment of the unit available? Can it be used to test the unit?

The cost of stubs and driver development

 
How much effort would it take to create the stubs and drivers?

In practice, much software in the environment of the unit is not relevant to testing the unit in isolation. When functions in the environment only have a result (and no side effect), it will be easy to develop stubs for. When functions in the environment do have a side effect but the side effect is irrelevant to the unit under test (e.g. logging, tracing, billing) they can also easily be replaced by stubs. If the function would be difficult to be replaced, it may be useful to import the function (i.e. copying it in total form the original software). This way, the required functionality is isolated. Figure 1.5 depicts a decision flow diagram explaining this.

Figure 1.5: Skeleton for creating a stub

The design of a stub depends on many factors. In general it is useful to be able to specify the result of a stubbed function. This result can be specified by an extra interface of the stub that is called by the test driver. When, for instance, we would test a library for managing address information and this address library uses a (stubbed) database, the driver code could be as follows:

void test_StoreAddress( void )
{
    ...
    ...
    /* Calculate the number of entries currently in the database */
    db_size = DbGetSize( contacts );

    /* Store the new entry */
    address_result = StoreAddress( contacts, new_address1 );
    assert( address_result == ADDRESS_OK );

    /* Check that the size of the data base has increased by one */
    assert ( DbGetSize( contacts ) == (db_size+1U) );
    db_size = db_size + 1U;

    /* Let the database generate an error when trying to lock
       the database. This is an extra stub interface to enforce 
       a certain execution path for DbStore.
    */
    set_stubresult_DbLock( DB_ERR_LOCK_FAILED );
    address_result = StoreAddress( contacts, new_address2 );

    /* reset stub to default behavior */
    set_stubresult_DbLock( DB_OK );

    /* Check thaty StoreAddress returns an error */
    assert( address_result == ADDRESS_ERR_STORE );

    /* Check that the size of the data base has NOT increased */
    assert ( DbGetSize( contacts ) == (db_size) );
    ...
    ...
}

This example shows that the database used by the address library has been stubbed. The side effects were completely covered (i.e. storing an element and calculating the number of elements stored) and the results of the stubs could be specified by the driver. The stub for the database, in this example, allows to test the address library more thoroughly by means of the extra interface to the function result. An outline of a general stub skeleton to accomplish this is given by 1.6

Figure 1.6: Skeleton for creating a stub. It is recommended to maintain an array with function results as a global variable that can be changed from the test driver via the additional stub interface.

In practice, stubs and part of the intended software environment will be combined in a pragmatic way. This is explained by figure 1.7 that depicts the situation in which an address library is tested.

Figure 1.7: An example of testing a library that handles addresses.

It uses an LCD controller to display results. In our case, LCD functions have side effects that are irrelevant to the behavior of the address library. Therefore, they have been stubbed. A sprintf function is difficult to be stubbed (also there is no need to do so) and is used. Malloc is a function that is relevant to the library and its behavior is important as the software may run on a embedded platform. Therefore the malloc function has been stubbed.

Harness generation

test harnessAn environment in which a certain module can be tested. As explained in previous sections, creating drivers and stubs can be a time consuming job. There are commercial tools available that can automatically generate drivers and stubs for a specific module based upon an interface description. These tools are often referred to as test harness generation tools. In practice, these tools are great assets for unit testing. Examples of these are:

1. StubbyC
2. StubbyCPP
3. C Test Bed
4. LDRA testbed
5. Cantata
6. Parasoft C++ test