Next: Static analysis
Up: Introduction to unit testing
Previous: Introduction to unit testing
Unit testing is a test activity which is part of software
development. The development environment as well as the development
activities are both important aspects when it comes to testing the
code. This document explains how test activities are be integrated
into development activities and shows how different techniques and
tools can be exploited.
The main course of developing software can be outlined as follows:
General approach to software development
During the coding stage, the design model is implemented. To ensure that the code complies to programming standards, automated static analysis is applied using static analysis tools such as QAC/QACPP or splint. These are static analysis tools that find vialations agains predefined rules automatically. Static analysis means that the code under test, does not have to be executed to allow for finding defects. It helps to improve the correctness, stability, portability and maintainability of the source code.
Coding & static analysis
Chapter 8 will address the use of QAC for doing static analysis in detail. Chapter 9 will address the use of splint in more detail.
When the code has been completed
walkthroughAn informal meeting in which a developer or
author of a document explains his tasks and approach. Usually organized
somewhere in the begin of a new task. The key idea of a code review
is that it helps to assure consistency of the source code with the
(intended) design. Also the dynamic behavior of the code can be examined
more thoroughly than any static analysis tool can do. When, during
static analysis, it has been decided to suppress warnings for special
cases in the source code, these suppressions are also evaluated during
a code review. When the issues of the code review have been solved,
dynamic tests are developed.
The code review is elaborated upon in 3.
The most important challenge of dynamic testing is to design
effective test cases. By definition, an effective test has a high
probability of finding errors. In general, more defects
can be found when the amount of code that is exercised by the test
increases. Research at different development plants has
demonstrated that the actual percentage of the code that is tested by
developers is about 40% on average only. Also, it has been experienced
that developers can easily increase this percentage by using coverage
analysis tools. Developers were able to increase the effectiveness
of there tests by using coverage analysis tools.. Coverage analysis makes it possible to see how much of the
software (and, in more detail, which parts of the software) is actually
exercised by the test. It helps the developer to suggest new test
inputs that will bring about more coverage (i.e. tests the code more
thoroughly). Without coverage analysis tool, it can be very difficult
to design testinputs that achieve more than 40% coverage.
There doesn't exist a universal minimal percentage of coverage each
test case must achieve. This is very much related to the risk that is
involved particular piece of software and the costs that comes with
testing that software.
Relation between the number of detected defects and the amount of achieved code (branch) coverage with unit testing. This figure has been published by Malaiya and Pasquini
When sufficient coverage is achieved by the test design, this test
design is used for various types of tests. Firstly, the outcome of the
tests are examined to see whether the result or effect that is brought
about by the test input is correct. Also, the code can be instrumented
to pinpoint defects related to non-functional behavior of the code
(i.e. memory leaks, misuse etc.). Frequently, dynamic analysis tool
like insure or purify can be used for
this purpose. Finally, the findings of the various tests are documented
and archived. The test design activity is discussed further in 4.
A stub is a small program routine that substitutes for a longer program,
possibly to be loaded later or that is located remotely. For example,
a program that uses Remote Procedure Calls (RPC) is compiled with stubs
that substitute for the program that provides a requested procedure.
The stub accepts the request and t hen forwards it (through another
program) to the remote procedure. When that pr ocedure has completed
its service, it returns the results or other status to the
stub which passes it back to the program that made the request.
A problem which often discourages developers to employ unit test
activities, is lack of testability of the software. In order to execute
any of the software, other parts of the software (i.e. the software
environment of that particular unit under test) are required. Because
the software environment that is required to dynamically test a certain
unit is developed simultaneously or is difficult to operate in practice,
dynamic unit testing might become a difficult job to do.
A solution to this is creating an artificial environment that simulates
the intended software environment of the unit. The artificial environment
is called stub. In order to call the functions defined by
the unit, a so-called driver must be created. A driver is a piece of
software which calls the functions in the unit under test. A driver is
normally used to execute the dynamic unit tests. driverA
driver is a piece of software which calls the functions in the unit
under test. Normally it is used to execute the dynamic unit tests.
A unit and its software environment
Creating drivers and stubs can be very time consuming. When every
unit has to be tested in isolation, stubs and drivers need to be developed
for every unit. Hence, the amount of software actually developed might
grow to double the size of the software that is actually delivered to
the customer. Of course, it is important to safe time and money by means
of testing instead of wasting it. Therefore, development of testware
must be very pragmatic. The next aspects must be judged when deciding
to develop stubs and drivers.
A unit and its stubs and driver
In practice, much software in the environment of the unit is not relevant
to testing the unit in isolation. When functions in the environment only
have a result (and no side effect), it will be easy to develop stubs
for. When functions in the environment do have a side effect but the
side effect is irrelevant to the unit under test (e.g. logging, tracing,
billing) they can also easily be replaced by stubs. If the function would
be difficult to be replaced, it may be useful to import the function
(i.e. copying it in total form the original software). This way, the
required functionality is isolated. Figure 1.5 depicts a
decision flow diagram explaining this.
- The need for thorough testing
Is the risk of failure really that high?
- The availability of the original software environment
Is the intended
software environment of the unit available? Can it be used to test the unit?
- The cost of stubs and driver development
How much effort would it take
to create the stubs and drivers?
Skeleton for creating a stub
The design of a stub depends on many factors. In general it is useful
to be able to specify the result of a stubbed function. This result can
be specified by an extra interface of the stub that is called by the
test driver. When, for instance, we would test a library for managing
address information and this address library uses a (stubbed) database,
the driver code could be as follows:
void test_StoreAddress( void )
/* Calculate the number of entries currently in the database */
db_size = DbGetSize( contacts );
/* Store the new entry */
address_result = StoreAddress( contacts, new_address1 );
assert( address_result == ADDRESS_OK );
/* Check that the size of the data base has increased by one */
assert ( DbGetSize( contacts ) == (db_size+1U) );
db_size = db_size + 1U;
/* Let the database generate an error when trying to lock
the database. This is an extra stub interface to enforce
a certain execution path for DbStore.
set_stubresult_DbLock( DB_ERR_LOCK_FAILED );
address_result = StoreAddress( contacts, new_address2 );
/* reset stub to default behavior */
set_stubresult_DbLock( DB_OK );
/* Check thaty StoreAddress returns an error */
assert( address_result == ADDRESS_ERR_STORE );
/* Check that the size of the data base has NOT increased */
assert ( DbGetSize( contacts ) == (db_size) );
This example shows that the database used by the address library has
been stubbed. The side effects were completely covered (i.e. storing an
element and calculating the number of elements stored) and the results of
the stubs could be specified by the driver. The stub for the database,
in this example, allows to test the address library more thoroughly by
means of the extra interface to the function result. An outline of a
general stub skeleton to accomplish this is given by 1.6
Skeleton for creating a stub. It is recommended to maintain an array
with function results as a global variable that can be changed from the test
driver via the additional stub interface.
In practice, stubs and part of the intended software environment will be
combined in a pragmatic way. This is explained by figure 1.7
that depicts the situation in which an address library is tested.
It uses an LCD controller to display results. In our case, LCD
functions have side effects that are irrelevant to the behavior of
the address library. Therefore, they have been stubbed. A sprintf
function is difficult to be stubbed (also there is no need to do so)
and is used. Malloc is a function that is relevant to the library
and its behavior is important as the software may run on a embedded
platform. Therefore the malloc function has been stubbed.
An example of testing a library that handles addresses.
test harnessAn environment in which a certain module can be
As explained in previous sections, creating drivers and stubs can be
a time consuming job. There are commercial tools available that can
automatically generate drivers and stubs for a specific module based
upon an interface description. These tools are often referred to as test
harness generation tools. In practice, these tools are great assets for
unit testing. Examples of these are:
- C Test Bed
- LDRA testbed
- Parasoft C++ test
Next: Static analysis
Up: Introduction to unit testing
Previous: Introduction to unit testing