June 26, 2020

Creating Testable Software

  —What does it mean to have testable code in McCall's Software Quality Model?

In McCall’s Software Quality Model I take a look at McCall’s Software Quality Framework. Here, I explore the concept of Testabilty in this framework as a means of improving software quality. McCall defines Testability as

Effort required to test a program to ensure it performs its intended function.

Effort is measured in terms of time, cost (dollars), or people and is the subject of Managing Cost in McCall’s Quality Model. How to ensure a program fulfills its intended function is another matter entirely.

Intuitively, good testing provides confidence in the correct operation of a program.

Testability is a measure of the degree a program supports testing. (See Software Testability for another definition.) Ensuring a program fulfills its intendend function is a different factor.

McCall’s definition of Testability includes criterion to ensure ease of testing.

• simplicity: avoid practices that increase complexity.
• modularity: introduce practices that promote highly independent modules (focus on high cohesion and low coupling).
• instrumentation: measure usage and error identification.
• self-descriptiveness: provide explanations of the functions performed by the software.

These criterion don’t ensure intended function because they don’t explicitly address it. The intuitive notion of correct operation isn’t part of Testability. It’s part of Correctness.

The Correctness quality factor is defined as

Extent to which a program satisfies its specifications and fulfills the user’s mission objectives.

Fortunately, Testability is improved by focusing on factors that positively affect it. Graphically, this looks like:

This graphic uses the rows and columns for Correctness and Testability in Tables 4.2-2 and 4.2-3 of Factors in Software Quality. The criterion are from Table 4.1-1 in the same document.

All factors depicted here positively affect Testability and Correctness. Several criteria positively affect Testability. The intuitive notion of whether a program fulfills its intended function lies in the traceability, completeness and consistency criterion. Only the traceability and consistency criterion positively affect Testability.

In fact, traceability is defined as

Those attributes of the software that provide a thread from the requirements to the implementation with respect to the specific development and operational environment.

and consistency as

Those attributes of the software that provide uniform design and implementation techniques and notation.

Testability is enabled by the factors linked to it but its definition is fulfilled by traceability between requirements and implementation. Traceabilty creates a closed-loop that provides confidence the tests test the program’s intended function.

Traceability relies on cross referencing related modules to requirements (through the design and implementation phases at the system level). The model suggests a traceabiltiy matrix be used for this.

However, traceability is insufficient to for Testability. It is insufficient because traceability doesn’t ensure all requirements are covered in both design and system test.

Traceability is defined as the total number of requirements traced divided by the total number of requirements. This is achieved by

1. using design to identify the requirements statisfied by a module.
2. using implementation to identify the requirements statisfied by a module’s implementation.

Ensuring that the total number of requirements traced is fulfilled by the completeness criteria. Completeness ensures, among other things, that

• the design agrees with the requirements and
• the code agrees with design.

Completeness is defined as

Those attributes of the software that provide full implementation of the functions required.

Completeness create a closed-loop between requirements and implementation by leveraging traceability. Is consistency needed? It’s defined as

Those attributes of the software that provide uniform design and implementation techniques and notation.

Consistency relies upon explicitly employing conventions that aid completeness and traceability. The interplay between consistency, completeness and testability ensure a program fulfills its intended function.

If I want to employ Testability metrics I need to deploy mechanisms that measure the criterion supporting it. I also need to employ metrics for Correctness and, very likely, factors defined by quality criteria identified as positively affecting Testability.

In all, Testability relies on eight direct or indirect criterion and six factors. If you want testable code you can’t focus on just Testability. This fact emphasizes one of the great points of this quality model: quality is a lifecycle activity.