What's the value of functional testing?

At the Agile Alliance's Functional Testing Tool workshop in October, I argued that functional tests are a way of telling us if our software works, if it's done, and if it's right. But they are also expensive and wasteful. I'd like to find a more agile way of accomplishing the goal. In this lightning talk, I present one way of doing so.

Elisabeth Hendrickson gave a rebuttal to this talk titled "Why Automated Acceptance Tests Are Crucial".

(Thanks to Elisabeth for recording and posting these videos! You can find more videos from the conference on Google Video.)

Transcript

(I've edited the transcript in a few places to clarify my wording.)

It's easy to focus on tools, especially at a workshop like this where we're talking about functional testing tools. But I've come to question the value of these tools--or, more specifically, I've come to question the value of functional testing in general.

As it's commonly practiced--as commonly practiced--functional tests cover the software end-to-end. They start at the UI (or just below it), they go down to the database, they say, "do this, do this, do this, check that." And, as commonly practiced, I've found this to be expensive, high maintenance, brittle, and slow.

(By "slow" I mean that each test takes a couple of seconds at least. I don't like slow because that leads to long builds, which leads to poor feedback, which leads to broken builds, which leads to preventable bugs because people aren't running the tests and seeing the regression failures.)

So I want to step back from functional testing and say, "Why are we doing this? This is a pain--why do we bother? What are we trying to accomplish here?" And that brings us to the title of my talk:

Does It Work? Are We Done? Is It Right? (Keep It Light!)

I'm not a traditional QA guy; I have a development background. But it seems to me that we want to know three things.

1. We want to know if programmers built what they intended to. ("Does it work?")
2. We want to know if the programmers intentions are correct. The programmers may have built what they intended to, but they may have intended to do the wrong things. (So, "is it right?")
3. And did we do everything that we were supposed to do. ("Are we done?")
4. And, because I'm an agile guy, I want to minimize waste during this process.

Actually, I'm an XP guy to be specific, and for me, if it doesn't contribute to delivering product--that is, software and associated artifacts like manuals--it's waste. So, for me, I feel that functional tests are wasteful. They're expensive, they're slow, they're brittle, and they don't contribute directly to delivering product.

In comparison, something like test-driven development is not wasteful because, even though TDD produces a comprehensive regression test suite, I actually go faster with TDD than I do without it... so it's not wasteful; it's the opposite of wasteful.

So I want to answer these quality questions, "does it work, are we done, is it right," without incurring waste... and ideally, I'd like to do it in a way that makes me faster, not slower.

I think that TDD is actually part of the solution. It helps us understand if the software works, because TDD excels at confirming programmer intent. (It's not just unit testing; it's anything that requires confirmation.) And, if you're not totally confident in your TDD, you can use exploratory testing to help you identify the gaps in your thought process and thus improve your use of TDD, and thus your confidence in your work.

"Are We Done?" Well, there's a really easy way to know that, and that's ask a customer! We can show them, we can involve them throughout the process. Everyday, we can say, "This is what we're doing; is this what you expected, are we going the right direction?" Onsite customer--it isn't just for Fridays anymore!

(I'm glossing over who the customer is--there's many customers. You can interaction designers who talk about the UI; you can have product managers who talk about priorities and business expertise; you can have domain experts who talk about domain rules.)

Does it work, are we done... "Is It Right?" Well, this is a little more complicated, but again, if we involve the customer, we might be able to answer that question. Is the UI right? Well, let's show our interaction designer! Let's show the people who are determining what the UI looks like and ask them: "Is this right?" And you know what, we have these great GUI development tools that allow us to co-develop with our interaction designer. We can sit down together at the same keyboard and say, "Let's move this around and see what it looks like."

Domain rules. Are they right? Well, again, we can show them to our customers. Now, that works really well for simple domain rules, but I have to admit that it does break down with more complex domain rules because there's just too many things to look at. And in fact those rules may not even be fully known or understood by our customers... but, Brian Marick was up here a few minutes ago and he has the perfect solution for this, I really like it: examples. Examples are a great tool, they're a great boundary object for showing what goes on with those domain rules. As Brian said on the mailing list, "they provide a lot of opportunity for surprise as tacit knowledge is made explicit, and then gets questioned."

I've found examples to be incredibly powerful, especially when they're declarative--in Fit terms, when you use column fixtures rather than action fixtures--because they focus on the domain rather than the application. The power of these examples, however, comes from the conversation. It's not about the tool, it's about getting up in front of a whiteboard with your experts and saying, "let's talk about some examples of your domain. Let's look at what goes on." Whenever I've done this with customers, there's been this great "ah ha" of "oh my goodness, we didn't realize that we misunderstood each other so much." And this always happens before we start talking about Fit. This happens while we're actually at the whiteboard talking about examples. Fit is actually secondary to this process.

(Nowadays I describe Fit as providing a way of automating your examples. But you know what, you don't need Fit to do that. You get the power without using Fit. But if you use Fit without examples, it won't be that useful.)

So, to answer these questions: does it work, are we done, is it right, and minimize waste, I think we can use TDD, perhaps with exploratory testing to tell us if it works; involved customers to tell us if it's done; concrete examples to tell us if its right, and we might automate those examples.

Thus it seems to me that there's no need for [functional] testing; we can eliminate waste and keep it light.