In 2011, ESPN’s Stats & Information Group released Total QBR as a way to measure quarterback efficiency in all facets of the game. Total QBR began as the brainchild of Dean Oliver, Alok Pattani, Albert Larcada, Jeff Bennett and Hank Gargiulo, and they leaned heavily on the work done by Brian Burke, formerly of AdvancedNFLStats.com, and others in its development.

Burke is now a senior analytics specialist for ESPN Stats & Info, and he spent the summer revamping QBR’s underlying model. In the interest of transparency, he has agreed to discuss some of the back-end details and misconceptions relating to Total QBR.

Q: Brian, to many people, Total QBR is a “black box” because there is no easy formula or even a simple regression available publically. Can you try to open the black box and give a high-level overview of how it works?

A: The basis of QBR is Expected Points, a measure of how each play is likely to affect the score of the game based on down, distance and yard line. For example, a big conversion on third down might be worth about plus-2.1 Expected Points Added (EPA), or an interception might be worth about minus-4.2 EPA, depending on the situation. QBR calculates EPA for every play on which the quarterback has a meaningful role, including passes, sacks, runs, scrambles, turnovers and even some types of penalties. We call these plays “QB Action Plays.”

The next step of the process is division of credit (or blame). QBR takes the EPA on each Action Play and gives a share of it to the QB. The size of that share depends on how challenging the play was. A bubble screen on first down will give relatively little credit to the QB for the play outcome, but a deep outside pass on third and long will give the QB a relatively large share of the credit for the outcome. Plays in which the QB is under duress by the pass rush give him extra credit for the play, whereas plays in which he is not under any pressure result in a slightly smaller degree of credit.

Next, QBR devalues “trash time.” Normally, each play receives equal weight, but as the game becomes out of reach, QBR discounts play as low as 25 percent. The purpose of the weighting is to account for how defenses tend to play with large leads, allowing offenses to move the ball while burning clock. High-leverage or “clutch” situations are not overweighted.

Lastly, QBR calculates each QB’s EPA per weighted Action Play. This makes it a rate stat and not a measure of total cumulative production. This, in turn, is scaled to be 0 to 100, with 50 pegged as an average performance. During the course of a season, a QBR of 25 is roughly replacement-level play, while a QBR of more than 75 would be what many fans would consider “elite.” You can think of a player’s QBR as roughly the percentage chance his team would win a game, all else being equal. So a QBR of 75 would suggest a team would win 75 percent of the time, or 12 out of 16 games in a full season, given that level of QB performance.

Q: What do you think the biggest misconception is relating to Total QBR?

A: There is a notion that QBR has a subjective component, but it doesn’t. I can understand where some fans might get that impression, because QBR relies on video analysis of each play. But this analysis is not critiquing QB performance, but rather it is cataloging things such as the depth of passes or whether a run play was a designed rush or scramble. Overthrows, underthrows and drops are also catalogued to help assess how much credit or blame the QB deserves for an interception or incompletion. The one aspect that could be accused of subjectivity is whether the QB was under duress, but this is not assessing the QB’s performance. It assesses only how the play was affected by the pass rush.

Q: You spent this summer updating Total QBR. What were the biggest changes you and the team made to the underlying algorithm?

A: We tried to simplify things wherever possible, while preserving the overall approach. The Expected Points model has been updated and features some significant improvements. For example, it now accounts for the bias in the data due to the fact that better offenses and worse defenses tend to see the most snaps. Previously, quarterbacks received small shares of credit for a variety of penalties on teammates or the opponent, and we have eliminated those that don’t have a direct connection with typical QB play.

There are a number of other changes, such as how QBR scores turnovers by the QB and how it credits QB for performance under duress in the pocket.

Previously, QBR counted both a turnover and the subsequent return against the QB, but what happens on the return is almost completely out of the QB’s control. So QBR now assesses the expected return against the QB, given the type of turnover, the game situation and where the turnover occurred with respect to the line of scrimmage.

Additionally, QBR credits QBs more for plays in which he was under duress by a pass rush. We have updated the amount of added credit a QB receives when he is pressured in the pocket.

Q: What are some of the biggest changes you saw after the update, and what are the reasons for those changes?

A: QBR now puts more of an emphasis on third-down conversions, so the QBs who make their money in those do-or-die situations climb a bit in the rankings, and those who don’t might fall a few spots. This effect includes both passing and running/scrambling. It turns out the greater emphasis on third downs is due to the new Expected Points model, which sees more leverage on third and fourth downs.

Q: What’s next when it comes to Total QBR?

A: QBR will soon feature an opponent adjustment. This will allow QBR to reflect the quality of opposing defenses, a particularly important consideration early in the season or for QBs who haven’t faced a full season of opponents. It goes a long way to putting each QB on a level playing field.

Q: Is there anything else you would like people to know about the process of revamping QBR or the metric itself?

A: QBR is far from perfect. Like with any analytic system, there are a number of modeling decisions involved where reasonable minds could disagree. But it is a quantum leap better than what it replaces. The traditional passer rating is a real disaster. Its weights are totally arbitrary. It double-counts completion percentage. It ignores fumbles, runs and scrambles. It confuses rate statistics and total statistics. QBR addresses all those problems and more. If we’re going to compare QBs using statistics, we might as well use the best ones available.