March 7, 2016

Successful Tests Find Bugs

  —Worth remembering: successful tests find bugs.

The software team I work with has been challenging my notion of testing. Fundamentally, I approach testing from the perspective of the constantly dissatisfied. I am dissatisfied whenever a test doesn't produce a failure. Comments have been made that I should be satisfied with a successful test result. I beg to differ.

In my view, the purpose of testing is to find bugs. Thus a successful test is one that identifies a product deficiency relative to its stated requirements. If a test produces a successful result, great. It may be a valuable component of our regression tests but it hardly means we've been successful.

If we run a series of tests and have a series of successful results we are no longer getting value from that testing. We should re-evaluate the requirement and make a decision on whether we have exceeded the requirement or need to alter our testing.

The contrary position is also important: tests which exceed requirements are not valuable either. Except perhaps to delineate a boundary on the product's operation.