You Can’t Measure Software Engineering Productivity, so Measure Job Satisfaction Instead

It’s the best approximation of an unmeasurable quantity.

Dan Fabulich is a Principal Engineer at Redfin. (We’re hiring!)

It’s infamously impossible to measure the productivity of a software engineer (or a software engineering team), because there’s no way to measure the output of a software engineer.

Over the years, people have tried to measure output in a variety of ways: lines of code, bugs fixed, stories closed, passing tests. None of these measures are good, and none of them are remotely suitable for managing performance, because clever engineers can easily manipulate the results. There’s a quote about this commonly attributed to Bill Gates:

“Measuring software productivity by lines of code is like measuring progress on an airplane by how much it weighs.”

Without a measure of output, we can’t know whether you’re a more or less productive engineer than you were last year, or whether your team is more productive than it was last year. We can’t decide whether I’m more productive than you are, or whether your team is more productive than my team. We can’t decide whether to change programming languages, whether to adopt “agile” methodology or reject it, or whether to change the way we interview candidates.

But, intuitively, such variations in productivity must exist. For example, as engineers, we often need to change our code and then re-run it to see the results. If we made your compile-and-refresh loop take ten times as long — or a hundred times as long — surely that would impact your productivity.

And yet, we can’t measure how much productivity would be impacted. If it takes us ten times as long to compile/refresh, maybe that’s a critical “hair on fire” bug, or maybe it’s just mildly annoying but harmless to productivity, or maybe it would paradoxically even improve productivity, forcing you to take a break to refresh your mind and spirits.

It turns out that that if you ask engineers what they like/dislike about their jobs, the things they’ll complain about seem to have something to do with their productivity. They’ll tell you about long compile times, having too many meetings, or a noisy work environment. They’ll ask for another big monitor, clearer specifications, or more time to pay down technical debt.

Across all industries, satisfaction at work comes down to feelings of autonomy (directing our work), mastery (accomplishment and self-improvement), and purpose (working on something more important than yourself).¹

Unproductivity is the opposite of mastery. It attacks job satisfaction directly. But, unlike productivity, we can measure job satisfaction.

Use Net Promoter Score to measure employee satisfaction

Redfin is both a real-estate brokerage and a software company. (It’s weird, but it’s a good weird. We’ve even adopted a slogan: “Keep Redfin Weird.” It works for us.)

We run our real-estate business using a standard metric for customer satisfaction, the Net Promoter Score (NPS). NPS was designed to measure customer satisfaction, but you can use it to measure employee satisfaction as well.

If you haven’t heard of NPS, it works like this. We survey each of our customers with this question: on a scale from 0 to 10, 10 being highest, how likely are you to recommend Redfin to a friend? A 9 or a 10 rating on that scale is called a “promoter”; 7 or 8 is called “passive,” and 6 or below is called a “detractor.” When computing NPS, you take the percentage of promoters and subtract the percentage of detractors. Thus, NPS ranges from +100 (everyone’s a promoter) to –100 (everyone’s a detractor).

We use NPS to manage the performance of individual real-estate agents and their managers; NPS is the single largest factor in our agents’ compensation. Traditional real-estate agents are paid solely on commission—a percentage of the sale price—which means that the agent and the buyer’s interests are conflicted. The more you pay, the more your agent gets. Redfin’s agents are salaried employees who receive a bonus for high NPS scores. The data shows that our approach delivers better results than traditional real-estate agents.

We’ve had fantastic results measuring customer satisfaction with NPS, so we naturally decided to measure the satisfaction of our employees the same way, with the same basic question: “How likely are you to recommend working at Redfin?”

We measure NPS in every team in the company, but our engineering management, starting from our CTO, Bridget Frey, focuses on the NPS of our engineering team. We do that partly because we want to recruit and retain the best employees (which is a huge challenge in today’s job market), but also because we believe that the NPS of our engineers is the closest metric we’ll ever have to a measurement of their productivity.

NPS is the best measure of engineering productivity

If you’re an engineer reading this, you’re probably skeptical, and I say that because so many of the best engineers I’ve worked with maintain a healthy sense of skepticism at all times.

I readily admit that we can’t prove that NPS is the best measure of engineering productivity, or even a good measure of engineering productivity, because that would require us to directly measure engineering productivity, which is impossible.

But let me toss a few brief arguments at you in favor of measuring and improving employee NPS, in the hope that one of them sticks to your face.

I predict that if you survey your engineering organization about what makes them unhappy at work, the vast majority of the issues they’ll report are factors of unproductivity. Don’t take my word for it; just try it. I also predict that you’ll find, as we find, that if you focus on your engineers’ top complaints—things that they say are driving down their productivity—then you’ll find that the satisfaction of your engineers rises. Measuring employee NPS doesn’t mean we have to stop recording other metrics, such as bug counts, user retention, or conversions. That stuff still matters. But when asking, “How do we measure whether our engineers are more productive?” we’re looking for the least confounded metric. A zillion factors can confound the NPS measurement of productivity, but other metrics are even more confounded. By “best measure” I mean it’s the “least bad measure.” If we grant for the sake of argument that unproductivity doesn’t cause dissatisfaction, I argue that satisfaction causes productivity, and that dissatisfaction causes unproductivity. I may not be able to measure my productivity, but I know that I feel a lot more productive when I come into the office in a good mood, excited about the job. This creates a virtuous cycle, where improving productivity improves satisfaction, which improves productivity, which improves satisfaction.

In the worst case, you might “waste” a bunch of time/money improving the satisfaction of your engineers for no productivity gain at all. (Aw, darn!) But employee NPS is a good leading indicator for employee attrition, which is a huge and quite measurable cost to any engineering organization. It’s also a good indicator for how many candidates your employees will actually refer to you, reducing the costs of recruiting.

Finally, if you’re an engineering manager, I hope you believe, as we do at Redfin, that the purpose of management is to nurture employees, not to rule them. In that case, NPS is a direct measure of the productivity of engineering managers. Judge your own mastery of your work by the satisfaction of your team. One way to improve team morale, perhaps the easiest way, is to fix the factors of unproductivity that they identify to you. Listen to their feelings of unproductivity and address them. It’s worth it.

Discuss on Hacker News

Discuss on Reddit

P.S. Redfin is hiring.

¹ ^ For more information about the role of mastery, autonomy, and purpose, you could read Dan Pink’s book, Drive, or just skip to the back of the book and read the citations, especially Deci and Ryan’s work on Self-Determination Theory.