• Next: Version Control
• Previous: "Done Done"
• Up: Chapter 7: Releasing

in 99 words

Rather than fixing bugs, agile methods strive to prevent them.

Test-driven development structures work into easily-verifiable steps. Pair programming provides instant peer review, enhances brainpower, and maintains self-discipline. Energized work reduces silly mistakes. Coding standards and a "done done" checklist catch common errors.

On-site customers clarify requirements and discover misunderstandings. Customer tests communicate complicated domain rules. Iteration demos allow stakeholders to correct the team's course.

Simple design, refactoring, slack, collective code ownership, and fixing bugs early eliminates bug breeding grounds. Exploratory testing discovers teams' blind spots, and root-cause analysis allows teams to eliminate them.

Extras

• Singed Egos looks at how we responded to criticism of this section.
• Alternative to Acceptance Testing provides another perspective on the same material. It's a bit more polished and concise, but not quite as thorough. It's good to read as a refresher.
• TDD Research Mischaracterized corrects our [Janzen & Saiedian] citation and adds more context.

Full Text

The following text is excerpted from The Art of Agile Development by James Shore and Shane Warden, published by O'Reilly. Copyright © 2008 the authors. All rights reserved.

No Bugs

Audience

Whole Team

We confidently release without a dedicated testing phase.

Let's cook up a bug pie. First, start with a nice, challenging language. How about C? We'll season it with a dash of assembly.

Next, add extra bugs by mixing in concurrent programming. Our old friends Safety and Liveness are happy to fight each other over who provides the most bugs. They supplied the Java multithreading library with bugs for years!

Now we need a really difficult problem domain. How about real-time embedded systems?

To top off this disaster recipe, we need the right kind of programmers. Let's see... experts... no... senior developers... no... aha! Novices! Just what we need.

Take your ingredients—C, real-time embedded systems, multitasking—and don't forget the novices—add a little assembly for seasoning, shake well, and bake for three years. (I do love me a bug pie.)

Here's how it turns out:

The GMS team delivered this product after three years of development [60,638 lines of code], having encountered a total of 51 defects during that time. The open bug list never had more than two items at a time. Productivity was measured at almost 3 times the level for comparable embedded software teams. The first field test units were delivered after approximately six months into development. After that point, the software team supported the other engineering disciplines while continuing to do software enhancements.¹

¹"Embedded Agile Project By the Numbers with Newbies" [Van Schooenderwoert]

These folks had everything stacked against them—except their coach and her approach to software development. If they can do it, so can you.

How Is This Possible?

If you're on a team with a bug-count in the hundreds, the idea of "no bugs" probably sounds ridiculous. I'll admit: "no bugs" is an ideal to strive for, not something your team will necessarily achieve.

However, XP teams can achieve dramatically lower bug rates. [Van Schooenderwoert]'s team averaged one and a half bugs per month in a very difficult domain. In an independent analysis of a company practicing a variant of XP, QSM Associates reported an average reduction from 2,270 defects to 381 defects [Mah].

You might think that improvements like this are terribly expensive. They're not. In fact, agile teams tend to have above-average productivity.²

²See, for example, [Van Schooenderwoert], [Mah], and [Anderson 2006].

Evidence for these results is as yet anecdotal. Rigorous scientific studies of complete software development methodologies are rare due to the large number of variables involved. While there are organizations that draw performance conclusions by aggregating hundreds of projects, none that I am aware of have enough data to draw conclusions on agile projects in general, let alone XP specifically. (QSM Associates is a well-regarded example of such an organization; as of June 2006, they only had data from a few agile projects.³)

³Personal communication with Michael Mah of QSM Associates.

In the absence of conclusive proof, how can you know if your team will achieve these results? There's only one way to know for sure: try it and see. It doesn't take superpowers. Teams of novices coached by an experienced developer have done this. All you need is commitment to follow the XP practices rigorously and support from your organization to do so.

How to Achieve Nearly Zero Bugs

Many approaches to improving software quality revolve around finding and removing more defects⁴ through traditional testing, inspection, and automated analysis.

⁴I use "defect" synonymously with "bug".

The agile approach is to generate fewer defects. This isn't a matter of finding defects earlier; it's a question of not generating them at all.

For example, [Van Schooenderwoert] delivered 21 bugs to customers. Working from Capers Jones' data, Van Schooenderwoert says that a "best in class" team building their software would have generated 460 defects, found 95 percent of them, and delivered 23 to their customer.⁵ In contrast, Van Schooenderwoert's team generated 51 defects, found 59 percent of them, and delivered 21 to their customer. At 0.22 defects per function point, this result far exceeds Capers Jones' "best in class" expectation of two defects per function point.

⁵An "average" team would have generated 1,035, found 80%, and delivered 207.

To achieve these results, XP uses a potent cocktail of techniques:

1. Write fewer bugs by using a wide variety of technical and organizational practices.

2. Eliminate bug breeding grounds by refactoring poorly-designed code.

3. Fix bugs quickly to reduce their impact, writing tests to prevent them from reoccurring, then fix the associated design flaws that are likely to breed more bugs.

4. Test your process by using exploratory testing to expose systemic problems and hidden assumptions.

5. Fix your process by uncovering categories of mistakes and making those mistakes impossible.

This may seem like a lot to do, but most of it comes naturally as part of the XP process. Most of these activities improve productivity by increasing code quality or removing obstacles. If you do them as part of XP, you don't have to do many of the other more expensive activities that other teams perform, such as an exhaustive upfront requirements gathering phase, disruptive code inspections, or a separate bug-fixing phase.

Ingredient #1: Write Fewer Bugs

Don't worry—I'm not going to wave my hands and say, "Too many bugs? No problem! Just write fewer bugs!" To stop writing bugs, you have to take a rigorous, thoughtful approach to software development.

Allies

Test-Driven Development

Energized Work

Pair Programming

Sit Together

Customer Tests

Exploratory Testing

Iteration Demo

Coding Standards

"Done Done"

Start with test-driven development (TDD), which is a proven technique for reducing the number of defects you generate [Janzen & Saiedian]. It leads to a comprehensive suite of unit and integration tests, and perhaps more importantly, structures your work into small, easily-verifiable steps. Teams using TDD report that they rarely need to use a debugger.

To enhance the benefits of test-driven development, work sensible hours and program all production code in pairs. This improves your brainpower, which helps you make fewer mistakes and allows you to see mistakes more quickly. Pair programming also provides positive peer pressure, which helps you maintain the self-discipline you need to follow defect-reduction practices.

Test-driven development helps you eliminate coding defects, but code isn't your only source of defects. You can also produce good code that does the wrong thing. To prevent these requirements-oriented defects, work closely with your stakeholders. Enlist on-site customers to sit with your team. Use customer tests to help communicate complicated domain rules. Have testers work with customers to find and fix gaps in their approach to requirements. Demonstrate your software to stakeholders every week, and act on their feedback.

Supplement these practices with good coding standards and a "done done" checklist. These will help you remember and avoid common mistakes.

Ingredient #2: Eliminate Bug Breeding Grounds

Writing fewer bugs is an important first step to reducing the number of defects your team generates. If you accomplish that much, you're well ahead of most teams. Don't stop now, though. You can generate even fewer defects.

Even with test-driven development, your software will accumulate technical debt over time. Most of it will be in your design, making your code defect-prone and difficult to change, and it will tend to congregate in specific parts of the system. According to [Boehm], about 20% of the modules in a program are typically responsible for about 80% of the errors.

These design flaws are unavoidable. Sometimes a design that looks good when you first create it won't hold up over time. Sometimes a shortcut that seemed like an acceptable compromise will come back and bite you. Sometimes your requirements will change and your design will need to change as well.

Technical debt breeds bugs.

Whatever its cause, technical debt leads to complicated, confusing code that's hard to get right. It breeds bugs. To generate fewer defects, pay down your debt.

Allies

Simple Design

Refactoring

Slack

Although you could dedicate a week or two to fixing these problems, the best way to pay off your debt is to make small improvements every week. Keep new code clean by creating simple designs and refactoring as you go. Use the slack in each iteration to pay down debt in old code.

Ingredient #3: Fix Bugs Now

Programmers have long known that the longer you wait to fix a bug, the more it costs to fix [McConnell] (p.75). In addition, unfixed bugs probably indicate further problems. Each bug is the result of a flaw in your system that's likely to breed more mistakes. Fix it now and you'll improve both quality and productivity.

To fix the bug, start by writing an automated test that demonstrates the bug. It could be a unit test, integration test, or customer test, depending on what kind of defect you've found. Once you have a failing test, fix the bug. Get a green bar.

Don't congratulate yourself yet—you've fixed the problem, but you haven't solved the underlying cause. Why did that bug occur? Discuss the code with your pairing partner. Is there a design flaw that made this bug possible? Can you change an API to make such bugs more obvious? Is there some way to refactor the code that would make this kind of bug less likely? Improve your design. If you've identified a systemic problem, discuss it with the rest of your team in your next stand-up meeting or iteration retrospective. Tell people what went wrong so they can avoid that mistake in the future.

Allies

Collective Code Ownership

Sit Together

Fixing bugs quickly requires the whole team to participate. Programmers, use collective code ownership so any pair can fix a buggy module. Customers and testers, personally bring new bugs to the attention of a programmer and help him reproduce it. These actions are easiest when the whole team sits together.

In practice, it's not possible to fix every bug right away. You may be in the middle of working on something else when you find a bug. When this happens to me, I ask my navigator to write the problem on our to-do list. We come back to it 10 or 20 minutes later, when we come to a good stopping point.

Ally

Slack

Some bugs are too big to fix while you're in the middle of another task. For these, I write the bug on a story card and announce it to the whole team. (Some teams use red story cards for this purpose.) We collectively decide if there's enough slack in the iteration to fix the bug and still meet our other commitments. If there is, we create tasks for the newly-created story and pairs volunteer for them as normal. Sometimes your only task will be "fix the bug." I use the story card as its own task card when this happens.

If there isn't enough slack to fix the bug, estimate the cost to fix it and ask your product manager to decide whether to fix it in this release. If it's important enough to fix, schedule it into the very next iteration.

Although you may wish to fix every bug right away, you need to consider its cost and value before doing so. Some bugs are expensive to fix but have limited impact; these are often not worth fixing. However, because bugs generally become more expensive to fix over time, you should typically only choose between "fix" (as soon as possible) or "don't fix" (until a later release).

Don't Play the Bug Blame Game

FEATURE

—License plate seen on the back of a Volkswagon Beetle

If you tell a programmer that there's a bug in his software, you're likely to get a prickly response. "That's not a bug, it's a feature!" or, "What did you do wrong?"

I suspect that this is because "bug" usually means "you screwed up".⁶ Arguing about whether something is a bug or not often seems to be a fancy approach to finger-pointing. Some folks have taken this to a high art, making elaborate distinctions between bugs, defects, faults, errors, issues, anomalies, and of course, unintentional features. I prefer to avoid formal categorizations. What really matters is whether you will do or not do something, regardless of whether that thing is a fixing a bug or implementing a feature.

Mistakes do occur, of course, and we want to prevent those mistakes. Pointing fingers is counter-productive. The whole team—on-site customers, testers, and programmers—is responsible for delivering valuable software. Regardless of who made the mistake, the whole team is responsible for preventing it from reaching stakeholders. To that end, when I think about bugs, I focus on what the team delivers, not where the bug came from.

A bug or defect is any behavior of your software that will unpleasantly surprise important stakeholders.

Before you yell "bug!", however, there are a few things to know.

• If the team finds and fixes a problem as part of their normal work—in other words, before a story is "done done"—it's not a bug.

• If the on-site customers have intentionally decided that the software should behave this way, it's not a bug. (Some of your software's behavior will be unpleasant to stakeholders—you can't please all people all the time—but hopefully it won't be a surprise.)

Remembering these things won't help you settle contract disputes, so don't use them for formal bug categorization. Instead, change the way you think about bugs. Stop worrying about who made what mistake, and start focusing on what you can do, as a team, to increase value by delivering fewer bugs.

⁶Thanks to Ron Jeffries for this insight.

Ingredient #4: Test Your Process

These practices will dramatically cut down on the number of bugs in your system. However, they only prevent bugs you expect. Before you can write a test to prevent a problem, you have to realize the problem can occur.

Ally

Exploratory Testing

Exploratory testing, in the hands of a good—some might say diabolical—tester, is an invaluable way to broaden your understanding of the types of problems that can occur. An exploratory tester uses her intuition and experience to tell her what kinds of problems programmers and customers have the most trouble considering. For example, she might unplug her network cable in the middle of an operation or perform a SQL injection attack on your database.

If your team had typical adversarial relationships between programmers, customers, and testers, these sorts of unfair tests might elicit bitter griping from programmers. Remember, though—you're all in this together. The testers have exposed a hole in your thought process and, by doing so, saved you from uncomfortable apologies to stakeholders or from dramatic failures in production. Congratulate your testers on their ingenuity—and as you write tests, remember the problems they have exposed.

Exploratory testing is a very effective way of finding unexpected bugs. It's so effective that the rest of the team might start to get a little lazy.

Think of exploratory testing as testing your process, not your software.

Don't rely on exploratory testing to find bugs in your software. (Really!) Your primary defense against bugs is test-driven development and all of the other good practices I've mentioned. Instead, use exploratory testing to test your process. When an exploratory test finds a bug, it's a sign that your work habits—your process—have a hole in them. Fix the bug, then fix your process.

Testers, only conduct exploratory testing on stories that the team agrees are "done done." Programmers and customers, if your testers find any problems, think of them as bugs. Take steps to prevent them from occurring, just as you would for any other bug. Aim for a defect rate of under one or two bugs per month including ones found in exploratory testing.

A good exploratory tester will find more bugs than you expect. To make the bug rate go down, fix your process.

Ingredient #5: Fix Your Process

Some bugs occur because we're human. (D'oh!) More often, bugs indicate an unknown flaw in our approach. When the number of bugs you generate gets low enough, you can do something usually associated with NASA's Space Shuttle software: root-cause analysis and process improvement on every bug.

Even if your defect count isn't low enough to allow you to do root cause analysis of every bug, consider randomly picking a bug each iteration for further analysis.

When you fix a bug, start by writing an automated test and improving your design to make the bug less likely. This is the beginning of root-cause analysis, but you can go even further.

Ally

Root-Cause Analysis

As you write the test and fix the design, ask questions. Why was there no test preventing this bug? Why does the design need fixing? Use the "five whys" technique to consider the root cause. Then, as a team, discuss possible root causes and decide how best to change your work habits to make that kind of bug more difficult.

Making Mistakes Impossible

The best way to fix your process is to make mistakes impossible. For example, if you have a problem with UI field lengths being inconsistent with database field lengths, you could change your design to base one value off of the other.

The next best solution is to find mistakes automatically. For example, you could write a test to check all UI and database fields for consistency and run that test as part of every build.

The least effective approach (but better than nothing!) is to add a manual step to your process. For example, you could add "check modified database/UI fields for length consistency" to your "done done" checklist.

Next, ask yourselves if the root cause of this bug could also have led to other bugs that you haven't yet found. Testers, ask yourselves if this type of mistake reveals a blind spot in the team's thinking. Perform additional testing to explore these questions.

Invert Your Expectations

If you follow these practices, bugs should be a rarity. Your next step is to treat them that way. Rather than shrugging your shoulders when a bug occurs—"Oh yeah, another bug, that's what happens in software"—be shocked and dismayed. "That's our fourth bug this month! Why do we have so many bugs?"

I'm not so naïve as to suggest that the power of belief will eliminate bugs, but if you're already close, the shift in attitude will help you make the leap from reducing bugs to nearly eliminating bugs. You'll spend more energy on discovering and fixing root causes.

For this reason, I recommend that new XP teams not install a bug database. If you're only generating a bug or two per month, you don't need a database to keep track of your bugs; you can just process them as they come in. Explicitly not providing a database helps create the attitude that bugs are abnormal. It also removes the temptation to use the database as a substitute for direct, personal communication.

Depending on your industry, you may need a formal way to track defects, so a database may be appropriate. However, I never assume that a database will be necessary until the requirements of the situation prove it. Even then, I look for ways to use our existing process to meet the regulatory or organizational requirements. Although some shudder at the informality, archiving red bug cards in a drawer may suffice.

Even if you are able to eliminate your bug database, you still need to be able to reproduce bugs. You may need screenshots, core dumps, and so forth. If you fix bugs as soon as they come in, you may be able to work directly out of your email inbox. Otherwise, you can keep this information in the same place that you keep other requirements details. (See Incremental Requirements in Chapter 9.)

Questions

If novices can do this, what's stopping management from firing everybody and hiring novices?

All else remaining equal, experienced developers will always produce better results and more quickly than novices. If you have the option to bring high-quality people into your team, do it.

The point is that these practices are within the grasp of average teams—even teams of novices, as long as they have an experienced coach. They won't achieve zero bugs, but they are likely to achieve better results than they would otherwise.

How do we prevent security defects and other challenging bugs?

This approach only prevents bugs that you think to prevent. Security, concurrency, and other difficult problem domains may introduce defects that you never considered.

In this situation, using exploratory testing to test and fix your process is even more important. You may need to hire a specialist tester, such as a security expert, to probe for problems and teach the team how to prevent such problems in the future.

How long should we spend working on a bug before we convert it into a story?

Ally

Slack

It depends on how much slack you have left in the iteration. Early in the iteration, when there's still a lot of slack, I might spend as much as four pair-hours on a defect. Later, when there's less slack, I might only spend ten minutes on it.

Bugs are usually harder to find than to fix, so enlist the help of the testers. The fix often takes mere minutes once you've isolated the problem.

If our guideline is to fix bugs as soon as we find them, won't we have the unconscious temptation to overlook bugs?

Perhaps. This is one reason we pair: pairing helps us maintain our team discipline. If you find yourself succumbing to the temptation to ignore a bug, write it on a story card rather than letting it slide by. Let the rest of the team know about the bug and ask somebody to volunteer to fix it.

If we don't find any bugs, how do we know that our testers are doing exploratory testing correctly?

It's a bit of conundrum: the team is supposed to prevent bugs from occurring in "done done" stories, so exploratory testing shouldn't find anything. Yet if exploratory testing doesn't find anything, you could be testing the wrong things.

If bugs are particularly devastating for your project, ask an independent testing group to test a few of your iteration releases. If they don't find anything surprising, then you can have confidence in your exploratory testing approach.

This is probably overkill for most teams. In practice, if you're following the practices and your testers haven't found anything, you can comfortably release your software. Re-evaluate your approach if your stakeholders or customers find a significant bug.

We have a large amount of legacy code. How can we adopt this policy without going mad?

Most legacy code doesn't have any tests and is chock-full of bugs. You can dig your way out of this hole, but it will take a lot of time and effort. See "Applying XP to an Existing Project" in Chapter 4 for details.

Results

When you produce nearly zero bugs, you are confident in the quality of your software. You're comfortable releasing your software to production without further testing at the end of any iteration. Stakeholders, customers, and users rarely encounter unpleasant surprises, and you spend your time producing great software instead of fighting fires.

Contraindications

"No Bugs" depends on the support and structure of all of XP. To achieve these results, you need to practice nearly all of the XP practices rigorously:

• All of the "Thinking" practices are necessary (Pair Programing, Energized Work, Informative Workspace, Root-Cause Analysis, and Retrospectives); they help you improve your process, and they help programmers notice mistakes as they code.

• All of the "Collaborating" practices except "Reporting" are necessary (Trust, Sit Together, Real Customer Involvement, Ubiquitous Language, Stand-up Meetings, Coding Standards, and Iteration Demo); most help prevent requirements defects, and the remainder help programmers coordinate with each other.

• All of the "Releasing" practices except "Documentation" are necessary ("Done Done", No Bugs, Version Control, Ten-Minute Build, Continuous Integration, and Collective Code Ownership); most help keep the code organized and clean. "Done Done" helps prevent inadvertent omissions.

• All of the "Planning" practices except "Risk Management" are necessary (Vision, Release Planning, The Planning Game, Iteration Planning, Slack, Stories, and Estimating); they provide structure and support for the other practices.

• All of the "Developing" practices except "Spike Solutions" are necessary (Test-Driven Development, Refactoring, Simple Design, Incremental Design and Architecture, Performance Optimization, Customer Reviews, Customer Testing, Exploratory Testing); they improve design quality, reduce requirements defects, and provide a way for testers to be involved in defect prevention as well as defect detection.

If you aren't using all these practices, don't expect dramatic reductions in defects. Conversely, if you have a project that's in XP's sweet spot (see Is XP Right For Us? in Chapter 4) and you're using all of the XP practices, more than a few bugs per month may indicate a problem with your approach to XP. You need time to learn the practices, of course, but if you don't see improvements in your bug rates within a few months, consider asking for help (see "Find a Mentor" in Chapter 2).

Alternatives

You can also reduce bugs by using more and higher quality testing (including inspection or automated analysis) to find and fix a higher percentage of bugs. However, testers will need some time to review and test your code, which will prevent you from being "done done" and ready to ship at the end of each iteration.

Further Reading

"Embedded Agile Project By the Numbers with Newbies" [Van Schooenderwoert] expands on the embedded C project described in the introduction.

• Next: Version Control
• Previous: "Done Done"
• Up: Chapter 7: Releasing