What Does a Good Test Suite Look Like?
February 29, 2012
Arlo Belshee posed a question by email last week: "What are the characteristics of a good test suite?" Interesting question. He plans to collate the answers on his blog sometime soon. Meanwhile, here's my answer:
A good automated test suite runs fast, provides a safety net for refactoring, documents the intent of the code, and alerts the team to regressions. It doesn't prevent bugs on its own, but if bugs are escaping the team, that's a sign that there's a flaw in the team's development approach--which includes the team's approach to design, technical debt, requirements, and overall defect prevention as well as its approach to automated tests.
Run Fast
The test suite needs to run fast because a slow test suite leads to broken builds. When the test suite is too slow, people don't run all the tests before integrating, and integration failures aren't always fixed right away because the person or pair who made the mistake isn't always available to fix it. This often compounds so that one build failure hides a second, and you end up with the team letting the build remain broken for hours or even days. This violates a fundamental principle of continuous integration, which is that the code in the repository always builds and passes its tests.
Further reading:
Provide a Safety Net
The test suite needs to provide a safety net for refactoring so that the team is able to improve their design and pay down technical debt. Without refactoring, technical debt steadily increases, leading to degraded design quality, which leads to defects.
Further Reading:
Document Intent
The test suite also acts as living documentation. This is necessary because comments and requirements documents go out of date. Tests can document both programmer intent (at the class and method level) and business intent (at the business rules level). Although people often write end-to-end tests to document business intent, this isn't necessary; it's actually more effective to write "customer unit tests" that focus on specific business rules.
Further Reading:
Detect Regressions
The test suite needs to be comprehensive enough to alert the team to regressions. This is important because manual regression testing is too slow and expensive to do every iteration, and this is a burden that steadily increases as the application gets bigger. I typically want to see unit tests for every class and method, focused integration tests for every interaction with outside processes, files, and systems, and a small number of end-to-end tests for smoke testing. (This assumes your design doesn't require end-to-end tests to ensure everything hangs together, which is also necessary to make your build run fast.)
Further Reading:
Prevent Defects
Finally, an automated test suite is just part of an overall strategy of defect prevention. It's an important part, but automated tests on their own aren't enough to prevent defects. You also need to focus on refactoring and technical debt control, close interaction with business experts and other requirements donors, and finding and fixing gaps in the team's thought process.
Further Reading: