When I speak out against certifications, one common response is a question: "Well, we need some way to hire good developers. How can we filter through the piles of candidates we get without using certifications?"

It's a good question. Filtering is necessary. The problem with certifications is they're not a good filter--they don't prove anything about candidates' on-the-job ability. At best, they prove the candidate's ability to cram for the certification requirement, which can be as low as "connect butt to chair (retries allowed)." My experience is that programmers who promote their certifications are generally mediocre at best. Even George Dinwiddie agrees, and he was arguing against that sort of value judgment:

A number of [the smartest people I know] have made public statements that they discriminate against people who hold certifications. That really saddens me.

It's true that I've found a pretty strong inverse correlation between collecting certifications and ability to do the work. I'm not sure I ever found a job applicant with Certified Java Programmer on their resume who passed muster when I asked them to write some code. I am sure that I gave everyone an equal chance.

George Dinwiddie, "Some of the smartest people I know..." (highlighting added)

Actually, I doubt George really gives everyone an equal chance, because that would mean he interviews every candidate. When hiring, you usually have too many candidates to interview every one. You need some way to filter out the dross without sacrificing your chance to get a great candidate. That's why people look at certifications. Unfortunately, if you do that, you're probably filtering for "mediocre" rather than "great."

So, what can you do instead? The following is an approach I used recently. There are more innovative approaches out there--this approach was designed for a company with a very traditional view of HR and limited patience for team-directed hiring--but it makes a good starting point and illustrates how to hire well without depending on certifications.

The Approach

This approach consists of five stages. The earlier stages take the least effort and filter out the greatest number of candidates. The later stages take more effort and delve into candidates' specific qualifications. Throughout the process, the team, not the hiring manager, is in charge. The teams knows most about the work to be done and the type of person they need to hire. (I'm assuming an Agile-like team-based approach to the work.)

• Stage 1 creates a barrier to entry that filters out people who aren't willing to make an effort.
• Stage 2 gives the team a brief look at the candidate's capabilities--but more than the resume--allowing them to filter out people who aren't worth interviewing.
• Stage 3 allows the team to learn more about the candidate's personality and experience.
• Stage 4 provides a detailed look at the candidate's real-world abilities and behavior.
• Stage 5 is the job offer, which is when the hiring manager gets involved.

Stage 1: Candidate Self-Filtering

Start by including about five essay questions with each job posting. The team creates the questions--or rather, a pair does, and the team revises--and they're conversations rather than tests. About half are about personality ("Describe a time when you had a disagreement with someone and later had new insight about the disagreement. What was your insight and what did you do about it?") and about half are about skills ("Describe a time when you added tests to legacy code. How did you approach the problem and what was the result?").

(I stole this idea from Ternary Software, who now appear to be defunct, perhaps because two of the three founders moved on to other interests).

This stage eliminates the majority of non-serious applicants without any effort on your part. The questions take real effort to answer, which eliminates resume spammers entirely, and puts off anyone who doesn't really want the job. The remaining non-serious applicants are eliminated in the next stage.

Any self-filtering approach will do. The idea is to make it unlikely for unqualified candidates to apply in the first place. In the early days of Extreme Programming, I replaced this step and the following one by posting job openings in my email signature on the XP mailing list. I only got three candidates, but they were good ones.

Stage 2: Pre-Interview Filtering

The team reviews candidate submissions as soon as they come in. The essay questions make it very easy to filter out unqualified applicants. I've seen prima donna and lone-wolf personalities that I've been happy to reject, and I've seen even more candidates that didn't have a clue and were bull-shitting their way through the answers. The nice thing is that it doesn't take long to review the answers. There's only about a page worth. I didn't even look at the resume at this point, although other team members did.

We take a thumb vote once everyone's reviewed the essay questions. Any down vote means the candidate is rejected. If the candidate survived the stage, we schedule an interview immediately.

In the early days, we would wait until we had collected all of the candidates, in an attempt to rank and interview the "best" ones. Then we lost a promising candidate to another company because we moved too slow. Now I don't try to hire the best candidate; I try to hire the first qualified candidate, and I set the bar for "qualified" high enough that any qualified candidate will be a great hire. By looking for the first qualified candidate rather than the most qualified, we can schedule an interview within a day of receiving the application, giving us first shot at the best candidates.

Stage 3: Behavioral Interview (Morning)

The in-person interview takes most of a day. In the morning, we conduct a group interview that includes everyone on the team, and sometimes the hiring manager, too. We use a behavioral interviewing approach, which means that we ask the candidate to describe specific experiences from her past, what happened, and what she would do differently. This approach focuses the interview on the candidate's actual experiences, not whitewashed theories.

One of the mistakes we made early on was to include far too many questions. The interview took too long and we always felt rushed. We improved the process by cutting down on questions and moving technical topics to the pairing sessions.

After the behavioral interview, another thumb vote determines if the candidate should be rejected. We added this step after wasting time with a few bad candidates--everyone knew they weren't qualified, but no one was willing to be rude and say so in front of the candidate.

Stage 4: Pairing Sessions (Afternoon)

After lunch, the candidate pairs with several programmers on whatever tasks we have scheduled for this iteration. (We have the candidate sign a NDA when he comes in.) Pairing sheds light on the candidate's actual capabilities--one candidate we interviewed got this far, but then wouldn't touch the keyboard. We realized later that he had been using subtle tricks throughout the interviews, such as open-ended answers and strategically-placed pauses, to parrot our own answers back to us. Those tricks didn't work in a real-life coding session, though, thank goodness.

After the pairing interview, we hold a thumb vote to quickly reject any obviously poor candidates. The following morning, after we've had a chance to sleep on it, we debrief the interview using the ORID format. This format first looks at Observations, then emotional Reactions, than Interpretation of those observations and reactions, and finally Decisions.

(Thanks to Diana Larsen for introducing me to this technique.)

The final decision in the debrief is a hire/no-hire recommendation to the hiring manager. The hiring manager has been otherwise uninvolved, other than the group interview; the team is the most qualified to make this decision, and he trusts the team to do its job.

Stage 5: Offer

We hire the first qualified candidate. Again, we set the bar for "qualified" high enough that any qualified candidate will be a great hire. The hiring manager and HR take care of required paperwork, background checks, salary negotiation, and the actual offer.

No Certificates Need Apply

Certifications don't prove ability. No experienced developer I've met thinks they do. In fact, the consensus is that certifications are negatively correlated with developer ability. Not always, of course, but generally. At best, they're a way of motivating the mediocre--a marketing technique.

Yet you need to hire great programmers. The technique I describe above is one way to do so. Start by creating an environment that makes the least-qualified developers unlikely to even apply. Continue by using essay questions to give you a sense of who's worth talking to in person. For qualified candidates, use behavior-based interviews and live pair-programming sessions to probe into details. Finish with a team-based debrief and recommendation.

Throughout this process, it's best for the team, not the manager, to design the protocol and review the candidates. They're the most qualified. Finally, I've seen the best results from hiring the first candidate that qualifies--and setting the "qualified" bar high enough that any qualified candidate will be a great candidate.

The most common excuse I've seen for certifications--other than their utility as a marketing tool--is that they make filtering applicants easier. They do, but I don't think they filter out the right people. I hope this essay helps you find another way.

Acknowledgements and Further Reading

Thanks to the Salem team--you know who you are--for their further development and refinement of this technique.

If you're looking for more detail, I've heard good things about Johanna Rothman's book, Hiring the Best Knowledge Workers, Techies, and Nerds: The Secrets and Science of Hiring Technical People. I haven't yet read it myself, but it comes highly recommended.