Test Design

Introduction

It is, of course, important to exercise the code by so-called dynamic testing (i.e. testing dynamic behavior of the code by executing it). The design of a dynamic test is the most important step for this part. The design is based upon two approaches: black-box and white-box test design. Both technique are discussed this chapter.

Input criteria

The source code has been reviewed before test design starts 15 Test design&The source code has been reviewed before test design starts&\ \hline

Process flow

Figure 4.1: Flow of designing dynamic test cases

Black-box test design

By definition, black box test cases are derived from (implicit) requirements by using formal test design techniques such as boundary value analysis, cause effect graphing or equivalence class partitioning[5]. For example, when the user can input the lifetime of a certain hash value between 0 and 1000 ms, this boils down to:

1. Try to find out an element for input representing a type of input class (-10, 10, 2000 ).
2. Try to find the inputs that focus on the boundaries of the requirement ( 0, 1000, 1001).

Figure 4.2: Using equivalence-class-partitioning and boundary value analysis

The black-box test cases are implemented using unit test platoforms like CppUnit, cunit, or check.

White-box test design

Test cases are derived from the software itself using coverage analysis. This means that it is analyzed how many (and which) of the code is actually exercised by the test. Using the results of this coverage analysis, the inputs are adapted an the code is executed with the new inputs. This iterative process results in a set of inputs that will cause enough code to be executed by the test. After generating test inputs this way, checks are inserted in the code that verify the supposed effect or result of that specific input.

Figure 4.3: Design of a unit test using a white-box approach.

There are several types of code coverage:

Routine coverage

:
Every function defined in the code is called at least once.

Statement coverage

:
Every statement in the code is executed at least once.

Branch coverage

:
Every decision is evaluated at least in two different manners (i.e. true or false

Multi-condition coverage

:
Every decision is evaluated at least in all different manners. Example

if (a || b )

To reach 100% multi-condition coverage we need to test the next situations:

 a  , !b
!a  ,  a
!a  , !b
 a  ,  b

Path coverage

:
All possible execution paths are tested.

Loop coverage

:
Every loop (e.g. for, while) is evaluated at least in three different manners (i.e. once or more than once and never.

Operator coverage

:
Every operator (e.g. for, while) is evaluated with all possible boundary values. Example

 if ( a < b )

To reach 100% operator coverage we need to test the next situations:

a < b
a == b
a > b

gcov is used to analyze the achieved coverage after executing the code. See chapter 11 for more details. gcov supports statement coverage and branch coverage. Page also provides an example of using gcov . a > b

Dealing with infeasible coverage

It can be very time consuming to achieve a 100 % statement- or branch coverage. If certain parts of the code are difficult to test because, for instance, a specific exception is hard to enforce during testing, this part of the code must be labelled as infeasible coverage as follows:

.....
catch( ... )
{
	// <INFEASIBLE_COVERAGE>
	// <REASON> We don't know the type of exception to expect here </REASON>

	 ...... (code that cannot be tested)

	// </INFEASIBLE_COVERAGE>
}

When the code of a class contains infeasible coverage parts, these must documenten as shown by the previous example and discussed with another designer/implementer.

Exit criteria

• A minimum of 100% statement coverage must be achieved by testing. 15 Test design&A minimum of 100% statement coverage must be achieved by testing.&\ \hline

• For safety-involved software 100% branch coverage must be achieved. 15 Test design&For safety-involved software 100% branch coverage must be achieved.&\ \hline

• After test design, gcov must not issue any coverage warnings 15 Test design&After test design, gcov must not issue any coverage warnings&\ \hline

• The test that were designed must be able to run automatically. (no gui) 15 Test design&The test that were designed must be able to run automatically. (no gui)&\ \hline