It's odd to talk about the decline and fall of the agile movement, especially now that it's so popular, but I actually think the agile movement has been in decline for several years now.

The State of the Art

I've seen a shift in my business over the last few years. In the beginning, people would call me to help them introduce Agile, and I would sell them a complete package that included agile planning, cross-functional teams, and agile engineering practices.

Now many people who call me already have Agile in place (they say), but they're struggling. They're having trouble meeting their iteration commitments, they're experiencing a lot of technical debt, and testing takes too long. So they hire me to help them with one of these things. When I go visit, I see a team that is nominally agile, but is suffering huge numbers of problems and is anything but the joyful, settled, smooth-running workplace I expect from an agile organization.

Other consultants I've talked to report the same experience. "Rescuing Scrum teams keeps me in business," joked one colleague. It's funny because it's true.

The Role of Scrum

Scrum is undeniably the winner of the agile method wars. Thanks to the Scrum Alliance's vast (and lucrative) network of Certified Scrum Trainers and Certified ScrumMaster courses, when people say "Agile," they usually mean Scrum. So when "Agile" fails, it's generally Scrum that's failing. And Scrum is incomplete, purposefully so. "Scrum is like your mother-in-law," Ken Schwaber said at the recent Agile Vancouver conference. "It's constantly pointing out your shortcomings." The trick is that you're supposed to learn from that feedback and fix your problems.

But because Scrum works in short cycles and doesn't include any engineering practices, it's very easy for teams using Scrum to throw out design. Up-front design doesn't work when you're using short cycles, and Scrum doesn't provide a replacement. Without continuous, incremental design, Scrum teams quickly dig themselves a gigantic hole of technical debt. Two or three years later, I get a call--or one of my colleagues does. "Changes take too long and cost too much!" I hear. "Teach us about test-driven development, or pairing, or acceptance testing!" By that time, fixing the real problems requires paying back a lot of technical debt, and could take years.

What frustrates me the most is that this situation is entirely avoidable. In a green-field environment, the solid agile engineering practices included in Extreme Programming pay for themselves within the first few months. Without XP's agile engineering practices, code quality and productivity asymptotically decreases over time. With them, productivity starts lower, but then it asymptotically increases. I can't prove it, but my sense is that the two curves cross at about the eight-week mark. Maybe sooner. Agile engineering practices aren't only important--they pay for themselves! Doing anything else is pure negligence... if you understand your options. Scrum is silent on the matter.

Scrum, Misapplied

It's not entirely Scrum's fault. Agile development isn't just about good engineering practices. It's also about acknowledging the importance of people in software development, forming cross-functional teams, obtaining high-bandwidth communication, constantly reflecting and improving, delivering value, and changing plans to take advantage of opportunities.

Scrum includes these other points. It says that the team should be cross-functional and recommends collocating the team in a shared workspace. It says the team should deliver a valuable, shippable product at the end of every Sprint, and that the team should self-organize, discover impediments, and remove them.

Oh, and it also has a few mechanical things about a monthly Sprint and daily Scrum. Trivial stuff compared to the rest. But guess which part people adopt? That's right--Sprints and Scrums. Rapid cycles, but none of the good stuff that makes rapid cycles sustainable.

You're Doing It Wrong

There are a lot of teams right now failing with Agile. These teams are working in short cycles. The increased planning frequency has given them more control over their work and they're discovering and fixing some problems. They feel good, and they really are seeing more success than they were before.

But they aren't working in shared workspaces or emphasizing high-bandwidth communication. They don't have on-site customers or work in cross-functional teams. They don't even finish all of their stories by the end of each Sprint, let alone deliver releasable software, and they certainly don't use good engineering practices.

These teams say they're Agile, but they're just planning (and replanning) frequently. Short cycles and the ability to re-plan are the benefit that Agile gives you. It's the reward, not the method. These psuedo-Agile teams are having dessert every night and skipping their vegetables. By leaving out all the other stuff--the stuff that's really Agile--they're setting themselves up for rotten teeth, an oversized waistline, and ultimate failure. They feel good now, but it won't last.

Failures in the Midst

It's human nature to only do the stuff that's familiar and fun, and that's what has happened with Agile. People look at agile methods as a chinese menu of practices, choose the few that look cool, and ditch the rest. Unfortunately, the parts they leave out are the parts that make Agile work. Scrum makes it worse by ignoring important (but hard) agile engineering practices, and the Scrum Alliance makes it worse still with their armies of trainers--some good, some not--issuing dubious "ScrumMaster" certificates to people who demonstrated competence in connecting butt to chair for two days.

So, unfortunately, a lot of self-described Agile projects are going to fail. They're failing right now. And eventually Agile will take the blame, and it will pass, as all fads eventually do.

It's too bad. The good Agile--the real Agile--it really works. I've seen it. My colleagues have seen it. It's been repeated hundreds of times, and some of those projects have succeeded for years. But those hundreds of successes will be drowned out by the thousands of failures.

Can we prevent this trend? I don't know. Perhaps we should start by giving up on the "Agile" brand name. It's never been clearly defined, which has allowed a lot of dysfunctional projects to call themselves "Agile."

Or maybe we need to stop selling Agile. Maybe we need to say, "Agile is hard, and you can't master it by sitting through a two-day course." Maybe we need to be firm and say, "Sorry, if you don't use agile engineering practices, if you don't have high-bandwidth communication, and if you don't include a strong customer voice, you're not going to succeed. Try something else instead." Scrum is popular because it's easy--and that's part of the problem.

Whatever we do, we need to do it soon. Agile is failing all around us, and I'd hate for the failure of the fad to take down the truly useful ideas that we started with.