Component testing

D. Adams
05nov01 0920

Here we present some requirements for component testing. We follow the ideas of Lakos.

Status and history
Definitions
Use cases
Requirements
Design issues
References

Definitions

Header file

A header file contains C++ source code intended to be included in other files via the #include mechanism. In ATLAS, these have the suffix ".h".

Source file

We use the abbreviation "source file" for a C++ source implementation file which is intended to be compiled directly and not to be included in any other C++ sources. In ATLAS these have the suffix ".cxx".

Component

A component is a single C++ source file, header and associated component test source. It will typically define one class and its associated free functions. We assume that all symbols introduced in the component header are defined in the header or source of that component.

Package

A package is a collection of components. Typically there is a one-to-one mapping between components and libraries and CVS modules. A package has a manager whose is responsible for its maintenance.

Component test

A component test is an executable used to test a single component.

Dependency

One component explicitly uses a second component if it includes code from the second or if its compiled code references symbols from the second. We say that the first component depends on the second. The dependency still exists if the useage is not explicit: if A depends on B and B depends on C, then A depends on C.

One package depends on a second if any of the components in the first package depend on any of those in the second.

Use cases

Code development

A developer checks out a single package and makes some modifications. The developer then modifies the component tests, and builds and runs them to verify that the package is working correctly before committing the changes.

Building a release

When building a release, the release manager wishes to verify each component performs as intended. The manager compiles all the components, creates libraries and then compiles, links and runs all of the component tests. Failed tests are reported to the appropriate package manager.

Patching a release

After a developer submits a patch for a release, the release manager wishes to verify the repaired package and the release. The component tests for that package are used to verify the package. The component tests for all packages which depend on this package are also rebuilt and rerun to verify the release. There should be no need to run tests for packages which do not depend on the modified package.

Requirements

1. Code is organized into components and packages as described above.

2. The dependencies between packages are known and are acyclic (if A depends on B, then B does not depend on A).

3. Each package provides a separate component test source for each component.

4. The build system must provide mechanisms for building and running these component tests and rebuilding and rerunning any test whose dependencies have changed.

5. Component test output (for success and failure) should be available for inspection. This includes the indication of success or failure and any output files. This is true for local and release builds.

6. The build system must provide the run-time enviroment required by the component tests. This includes mechanisms for defining enviromental variables, providing input files and a location for output files.

7. In the event of test failure, the test executable and its run environment should be available or easily reproduced so the developer can debug.

8. Abstract classes or classes with absent or complicated constructors should provide a mechanism to generate one or more concrete instances for use in higher level packages. This is called the test helper.

Design issues

We make some specific choices for developers following the specification of the CTEST interface:

Package interface

1. The suffix for component tests is _t.cxx, i.e. package Xyz has header Xyz.h, source Xyz.cxx and test Xyz_t.cxx.
2. Each component test is a main program which returns 0 to indicate success and nonzero to indicate failure.
3. The developer may provide an optional component test script with suffix ".sh" (e.g. Xyz.sh). If present, it is run in place of the component test.
4. The test helper is a header to be included in other test sources but not in any library sources. Suggested name is XyzTest.h.

Build environment

1. Component test sources are discovered automatically using the above naming convention.
2. Each component test is run in a separate directory. In the case of failure it is possible to save the executable and the contents of this directory for debugging.
3. If present, component test scripts are run in place of the test executable. The script is run with at least three arguments:
```
   Xyz.sh exe pkgdir reldir
```
where exe is the component test executable, pkgdir is the top-level directory of the package (for copying files from the package) and reldir is the top-level directory of the release (for copying files from other packages). Additional arguments spcify additional release directories if present.

User interface

It should be possible to trigger component testing with a special ctest target (gmake ctest) or as part of standard testing (gmake test?). The tests should be run once and then not run again unless their dependencies change.

References

J. Lakos, "Large Scale C++ Software Design", Addison Wesley (1996)

D. Adams, "CTEST 2.00: an interface for building C++ packages" (1999)

dladams@bnl.gov