The "well-calibrated interviewer" curse

Once, there was a place which did a whole bunch of technical interviewing. They were something of an anomaly in their industry, and they got a reputation as being seriously tough. Everyone "knew" that it took all sorts of skills and more than a little luck to even get your foot in the door, never mind getting through all of it to get a job.

What's not as nearly well-known is how that process actually worked on the inside. There are tales about things like founders who insisted on personally vetting every candidate, but I'm talking about things which haven't gotten much press. These are the day to day operations which fed the pipeline which ultimately fed that founder's personal review stack.

The usual procedure is that a candidate would be identified by a sourcer. That sourcer would then get this candidate into the system and would route them toward a target job or jobs. This would get a recruiter involved, and they'd do an introductory phone call. They were given technical questions and their answers and were supposed to score the candidate's response. If they were happy with them, the candidate would go on to the "technical phone screen".

This is where I entered the picture. As one of the technical types at this company, my name would be pulled from a hat once in a while, and then a phone screen or onsite interview would pop onto my schedule. I had to do some number of them per unit of time or it would have a negative impact on my quarterly review. This is how things went for a while, but then something changed.

Apparently the company was having a problem with "disaster onsite interviews". They'd fly a candidate out to the offices to spend a day being grilled, and the candidate would wash out halfway through. I heard tales of interviewers calling up the recruiters to kill the rest of a candidate's sessions so as to spare the other employees from also wasting their time.

I will try to convey about how much of my time a given interview would consume. First, there's the schedule wrangling when it first drops onto my plate. I'd have to acknowledge it in my calendar if it would work, or try to get the recruiter to wiggle it around so it didn't step on some other obligation I had. (The recruiters were particularly bad about booking these things right up against other stuff with no padding.) Figure that's 15 minutes of schedule wrangling and e-mailing.

In the 30 or so minutes before it was scheduled, I'd flip through their resume and do a little lightweight stalking of any URLs they happened to list. I would sometimes learn some interesting things this way, and occasionally found something we both had in common to use as an ice breaker. Then I'd either head over to the interview room (for onsites) or find a nice phone room somewhere and wait for the time to arrive.

The actual interview would take 45 minutes, and I'd take notes the whole time. Then I'd have to go back and just dump from my notes back into the "applicant tracking system" web site which asked a bunch of questions about what I thought about their technical abilities, culture fit, and that sort of thing. That could take another 45 minutes easily. I tried to do it immediately following an interview since I was much more likely to remember the context around some random scribble on my notebook immediately afterward. A couple of hours later, it might not be the same.

So figure right there I'd spend around 2 hours on a typical candidate, and I apparently managed to get a pretty good read on what they were like and what sort of technical chops they had. This level of thoroughness earned me membership in the so-called "well-calibrated interviewer list" when the hiring people tried to do something about those "disaster onsites" I mentioned earlier. They wanted to make it so these candidates would be filtered out sooner in the process.

Their plan was to feed these candidates to those of us on this new list as soon as possible in the process. We'd now be the first technical phone screen they'd encounter, and we could (in theory) knock them out of the running with that one call. Nobody else would encounter them.

What this meant in practice is that I wound up getting a steady load of these interviews as soon as they switched this thing on. That typically meant two phone screens per week.

Think about this now: that's 4 hours of my week (2 candidates at 2 hours each), or about 10%, and it happened week after week after week... after week. They thought it was that important.

At some point, they published numbers on who was doing what in terms of handling the load. I was shocked to see how high up the list I was. This was a large company, and I figured I'd be somewhere in the middle because I'm nobody special, but I was in the top 10 by quantity - globally.

That's when I went and looked up my teammates. I think one or two of them had done a single interview that quarter, and the rest had none.

These interviews were no walk in the park, either. More than a couple of these candidates were misogynistic bastards. One call went like this:

"Hi, this is Rachel calling from (company). May I speak to (candidate), please?"

"Oh, this is (candidate), but let me save us some time. I thought this was going to be a technical interview."

That guy bombed the interview right then and there. I didn't tell him that, though. I let him dig the deepest hole he could imagine by doing my best ditzy/derpy voice and just saying things like "they want me to ask about the load... average...?". He'd make up some garbage, and I'd log all of it. This way I could bury him both for being a sexist bastard and for being a lying piece of trash at the same time.

Let's review. It was a nontrivial amount of work, it kept happening at a steady pace with no relief in sight, it included maddening circumstances at times, and my "teammates" weren't pulling their weight. Most of the time, they didn't do them at all.

I wonder how many other people were completely burned out by this and simply refused all interviews from then on. At some point, I stopped caring about what it "might do to my quarterly review" since I figured that value was only used to punish people who didn't toe the line or to bribe people into being quiet about things they might have "discovered".

In all, I did about 100 phone screens or onsite interviews. Only 2 of them lead to people being hired, and one of them quit exactly a year later. So much for growing the company!

Well-calibrated interviewers, indeed.

January 5, 2013: This post has an update