In what seems fairly counterintuitive, we can’t answer these questions by examining specific pass catchers — instead, we have to examine quarterbacks, and specifically quarterbacks similar to Roethlisberger and those more similar to Brown and Bell’s new QBs. Ultimately, if we want to generalize whatever conclusions we draw to more than just the three aforementioned players, we convert our initial questions into this generalized analytic question: What is the isolated impact of replacing a non-HOF QB with a HOF QB? By correctly interrogating this question, we can

Determine the more important driver of offensive production (QB vs Pass Catcher) Quantify the value of a Hall of Fame QB Create an expectation for pass catchers that either join or depart from a team quarterbacked by a Hall of Famer

To answer the above question, I once again turned to the nflscrapR package to gather play by play level data from 2009–2018 (you can find all the code for this analysis here).

Once I had my data cleaned and prepared, I began my analysis. While our initial question seems rather simple to answer— choose my KPI of interest, and examine its levels for those HOF QBs versus those non-HOF QBs — this would be the wrong approach. By measuring production in this way, we wouldn’t be actually measuring the QB, but rather the QB’s team. This underpins one of the fundamental issues with all sports analysis, and arguably none more than in football: sports are dynamic systems, and therefore suffer from entanglement. In other words, measuring the QB’s performance is just another way of measuring the wide receiver’s performance and the offensive line’s performance and the opposing defenses performance, and vice versa.

This is obviously a problem when it comes to quantifying things. It may not matter to Patriots fans if the Patriots make Tom Brady good, or if Tom Brady makes the Patriots good (the Patriots are good either way!) but it matters when we’re trying to determine the value of each piece of the pie, so that we can make better decisions about how we pay players, whether or not to trade players, and how we should talk about the game itself. So how can we disentangle the entanglements? How can we know if Antonio Brown or Ben Roethlisberger was the driver of the Steelers offense? And how can we know what to expect when we replace a Hall of Fame QB with someone…normal?

The With or Without Analysis

Well, the short answer is that we can’t, at least not really. The game is too complex, there are too many moving parts, and we can’t have free rein to run an actual experiment within the confines of the league. But we can approximately decompose passer from receiver using a relatively simple analytical method: the with or without (WoWo) analysis.

So what is WoWo? The name is fairly descriptive — at the most basic level, we’re going to examine every play in which Ben Roethlisberger (and our other HOF QBs) threw to Antonio Brown (or Le’Veon Bell, or any other WR / RB) versus every time someone else threw to AB or Bell or any other WR that Roethlisberger also threw to (i.e. with Roethlisberger vs without Roethlisberger). We’re going to repeat this process for every WR-QB combo, paying particular attention to our HOF QBs, and from there, we can start to generalize the impact of a Hall of Famer on receiver performance, because we’re theoretically controlling for all other variables (this last part isn’t very accurate — in fact, we’re making a lot of assumptions, some of which may be flawed and that I’ll point out as we go along).

My first step was to find the set of players I felt reasonably reflected a “Hall of Fame” class of players. With that in mind, I isolated all passing plays from 2009–2018 for the following QBs (feel free to quibble with my selections, but this is what we’re going with):

Ben Roethlisberger

Tom Brady

Peyton Manning

Eli Manning

Aaron Rodgers

Philip Rivers

Matt Ryan

Drew Brees

Tony Romo

Russel Wilson

At least seven of the selected are sure-fire Hall of Famers, while Romo, Ryan, and Wilson either have decent shots, or were at some point considered elite.

These selections prove out when we examine our KPI of choice: Expected Points Added, or EPA (see the link for a detailed explanation — I used the Ron Yurko’s version of the EPA model in this analysis). The Hall of Famers I selected averaged 188% more EPA per attempt than the non-HOF QBs over the same time frame (for context, the overall mean was 0.14 EPA).

So we’ve established that the Hall of Famers produce more EPA than the non-HOF group, but that doesn’t actually tell us about the quality of the QB versus the quality of the team. For example, the level in EPA for Hall of Fame QBs may even be explained by a larger trend in overall EPA. So let’s check that.