Getty Images

In 2011, when ESPN introduced its new statistic Total Quarterback Rating, it was declared the “one stat that measures the totality of a quarterback’s performance.”

In 2015, that one stat says Ryan Fitzpatrick is better than Tom Brady: ESPN’s QBR rankings for this season have Fitzpatrick third in the NFL with a Total QBR of 76.7. Brady is fifth, with a Total QBR of 73.9.

Those rankings obviously look screwy, but taken alone they might seem more like a fluke than an indictment of Total QBR as a stat. The problem is that you can go through the Total QBR rankings and find all kinds of other rankings that just don’t pass the smell test: Brian Hoyer is a top 10 quarterback, ranking ahead of Ben Roethlisberger, Derek Carr, Matt Ryan, Philip Rivers and Russell Wilson. Ryan Mallett, who was benched and cut, ranks ahead of Cam Newton, who’s getting some MVP buzz. Ryan Tannehill, who’s had his ups and downs, ranks dead last among the 32 ranked quarterbacks — behind Colin Kaepernick, who’s been nothing but down.

The biggest problem with QBR is that ESPN isn’t transparent about how it’s calculated. The NFL’s traditional passer rating stat is flawed, but at least there’s no secret about how it’s calculated: Just plug the quarterback’s attempts, completions, yards, touchdowns and interceptions into the formula, and you get his passer rating. With QBR, we’re just supposed to take ESPN’s word for it that they’ve got it all figured out.

QBR incorporates many more elements than passer rating, coming not only from the box score but also from watching tape of the quarterbacks. In theory that should improve the statistic — ESPN says it tracks overthrows, underthrows, dropped passes, defended passes and yards after the catch — but in practice it adds a layer of subjectivity to the process. Does the person analyzing Jets tape have a different criteria for what constitutes a drop than the person analyzing Patriots tape? Baseball statistics have run into problems with different scorers having different standards for when a bobbled ball or a bad throw is an error, and ESPN’s Total QBR may have those same problems.

ESPN also says its QBR calculations are “clutch-weighted.” Again, we’re just supposed to take ESPN’s word for it that the “clutch weighting” is done properly, but the basic idea is that a quarterback gets extra credit if he plays well late in close games. ESPN wants to reward a quarterback who plays his best football while marching his team down the field playing from behind in the fourth quarter.

And that gets to a fundamental problem with the way QBR assesses Brady: The Patriots are rarely losing late in the fourth quarter, so Brady rarely gets an opportunity to prove he can perform in the clutch. A quarterback whose team is often losing in the fourth quarter, like Philip Rivers, gets many opportunities to improve his QBR “clutch-weighting.” And QBR seems to reward a quarterback for his performance in the clutch even if that clutch performance doesn’t result in his team winning. In Week Eight, Rivers had a Total QBR of 92.3, a higher QBR than Brady has had in any game this season. Rivers’s QBR in that game was presumably boosted by leading the Chargers on an eight-play, 49-yard drive to score a game-tying field goal with 2:27 left — even though the Ravens used that remaining 2:27 to march into position for a game-winning field goal with no time left.

In the past, Brady has, of course, had some famous performances playing from behind in the fourth quarter. In last season’s Super Bowl, when Brady rallied the Patriots with two fourth-quarter touchdown passes in a 28-24 win, his QBR was 81.1. That’s a good number, but not as good as the losing quarterback, Russell Wilson, whose Total QBR in the Super Bowl was 90.8.

Brady’s worst QBR this season was a 24.1 — well below average — in the Patriots’ 30-6 win at Dallas. In that game, Brady completed 20 of 27 passes for 275 yards, with two touchdowns and no interceptions. Brady also ran for a touchdown, and QBR also incorporates a quarterback’s rushing ability. So how could QBR rank Brady below average? ESPN doesn’t say, but Brady apparently doesn’t get any clutch credit because the Patriots had at least a two-touchdown lead throughout the second half.

But why shouldn’t QBR reward Brady for his ability to protect leads? Brady is probably the best quarterback in the league at protecting leads by picking up first downs through the air, while other teams protect leads on the ground. Shouldn’t that bolster Brady’s QBR?

There are some good things about Total QBR. The simplest one is that it uses a scale of 0-100, with 50 as average. That makes it easy to understand at a glance, unlike the traditional NFL passer rating, which is on a scale of 0-158.3 and has seen the league average rating rise from 57.8 in 1977 to 89.1 this year, as passing offenses have evolved.

QBR also benefits from using aspects of quarterback play like rushing and avoiding sacks, which are ignored by traditional passer rating. Fitzpatrick is good at making plays with his feet (he has picked up 10 first downs rushing this season) and good at getting rid of the ball when he’s under pressure (he has been sacked only six times, fewest of any quarterback with at least 200 pass attempts), and it’s good that QBR recognizes those attributes of Fitzpatrick’s play. Fitzpatrick really has played better than his traditional passing stats suggest, and there’s room for advanced stats to try to quantify the things the old stats miss.

But ESPN, in proclaiming QBR the “one stat that measures the totality of a quarterback’s performance,” has tried to turn QBR into something it isn’t. A stat that puts the totality of Fitzpatrick’s performance ahead of the totality of Brady’s performance is a stat that needs work.