A strong passing attack is crucial to NFL success. Seven of the top 10 teams this season in passing EPA per play are competing in the divisional round this weekend. That makes stopping the pass equally critical, of course. But quantifying pass defense in football has proven difficult. Our current metrics for measuring defensive performance just aren’t very good.

As has been noted before, much of the trouble comes from the fact that it’s hard to count the absence of a thing. For a cornerback or a safety in pass coverage, no news is good news: If they’ve done their job, the opposing QB often throws the ball to a more inviting target, throws the ball away or doesn’t throw at all. Pass breakups and interceptions are helpful and certainly signs of skill, but they’re rare. Starting defensive backs typically play more than 500 coverage snaps a year, yet in 2019, only 13 DBs had 10 or more pass breakups. Interceptions are even more infrequent, averaging 0.80 per game in 2019 across all positions. New England cornerback Stephon Gilmore tied for the league lead with just six INTs.

With the advent of player-tracking data, however, things are beginning to change. Michael Chiang has shown us where teams aren’t targeting opposing offenses, and now the NFL’s NextGen Stats team has released a metric that attempts to better quantify a defender’s performance when he is targeted.

Named (somewhat confusingly) Opponent Receptions Plus/Minus, what the metric actually does is estimate the number of incompletions over expected that a team creates; then, it assigns credit for the incompletion to the nearest defender on a play. The metric builds on completion probability and allows us to gather a larger sample of performance data on each defender. Couple this metric with the pass coverage classification system from ESPN analyst Brian Burke, and we can get a good sense of how well it does at identifying defender performance.

Stephon Gilmore led the league in forced incompletions Top 10 and bottom 10 NFL defenders in man coverage by Opponent Receptions Plus/Minus, which measures incompletions forced over expected, 2019 regular season Rk player team Pos G Targets Opp. Recs. Plus/Minus 1 Stephon Gilmore New England CB 16 77 6.16 2 Malcolm Jenkins Philadelphia S 16 19 5.99 3 Bradley McDougald Seattle S 15 20 4.90 4 J.C. Jackson New England CB 16 48 4.53 5 Eric Kendricks Minnesota LB 15 24 3.72 6 Jimmy Smith Baltimore CB 9 37 3.34 7 Darqueze Dennard Cincinnati CB 9 25 3.28 8 Marshon Lattimore New Orleans CB 14 45 3.11 9 Carlton Davis Tampa Bay CB 14 66 2.83 10 Marcus Peters Baltimore CB 16 56 2.70 … … 272 Terrell Edmunds Pittsburgh S 16 29 -4.68 273 Fabian Moreau Washington CB 12 22 -4.75 274 James Bradberry Carolina CB 15 34 -4.77 275 A.J. Bouye Jacksonville CB 14 47 -4.77 276 Pierre Desir Indianapolis CB 12 45 -5.12 277 Rashaan Melvin Detroit CB 13 61 -5.25 278 Kevin King Green Bay CB 15 51 -5.43 279 Adoree’ Jackson Tennessee CB 11 28 -5.76 280 Vernon Hargreaves III Houston CB 15 58 -7.74 281 Xavier Rhodes Minnesota CB 15 45 -10.52 Teams in bold made the playoffs. Sources: NFL Next Gen Stats, ESPN Stats & Information Group

For man coverage, the “face validity” of Opponent Receptions Plus/Minus is high. Gilmore is generally considered to have been one of the best corners in the league this season. He was named a first-team All-Pro and is a candidate for Defensive Player of the Year. According to the metric, he was the top man-coverage defender in the league, preventing approximately six receptions over expected on 77 targets. Moreover, Gilmore’s Plus/Minus game logs this season track fairly well with Pro Football Focus’s coverage grade, giving us further confidence that it’s capturing the right things.

The bottom of the defender list seems to make sense as well. I think most would agree that Xavier Rhodes has been about as useless as snake mittens this year, and PFF gave him a 46.4 grade for the season.

Perhaps more promising for Plus/Minus is that eight of the top 10 defenders identified are from teams who made the playoffs. Obviously, successful teams can and do overcome poor individual performances, as we see with Rhodes and the Vikings. But as a descriptive metric for man coverage, Opponent Receptions Plus/Minus appears solid.

But what about zone? Since Plus/Minus measures a defender’s proximity to a receiver on targeted passes, we might expect it to do better with man coverage versus zone. And that’s just what we find.

The zone coverage leaderboard is less convincing Top 10 and bottom 10 NFL defenders in zone coverage by Opponent Receptions Plus/Minus, which measures incompletions forced over expected, 2019 regular season Rk player team Pos G Targets Opp. Recs. Plus/Minus 1 Denzel Ward Cleveland CB 12 30 5.81 2 Ross Cockrell Carolina CB 14 46 5.17 3 Eddie Jackson Chicago S 16 19 4.97 4 James Bradberry Carolina CB 15 55 3.84 5 Shaquill Griffin Seattle CB 14 30 3.57 6 Charvarius Ward Kansas City CB 16 21 3.19 7 Mike Hilton Pittsburgh CB 16 32 3.07 8 Sean Murphy-Bunting Tampa Bay CB 16 25 2.96 9 Jamal Adams N.Y. Jets S 14 18 2.95 10 Troy Hill L.A. Rams CB 14 18 2.94 … … 272 Kyle Fuller Chicago CB 16 45 -4.70 273 Jordan Hicks Arizona LB 16 31 -4.75 274 Prince Amukamara Chicago CB 15 31 -5.23 275 Tahir Whitehead Oakland LB 16 28 -5.42 276 Bobby Wagner Seattle LB 16 56 -5.59 277 Desmond King L.A. Chargers CB 15 37 -5.97 278 Trae Waynes Minnesota CB 14 39 -5.97 279 Rock Ya-Sin Indianapolis CB 15 29 -5.98 280 De’Vondre Campbell Atlanta LB 16 40 -7.08 281 Byron Murphy Arizona CB 16 39 -8.59 Teams in bold made the playoffs. Sources: NFL Next Gen Stats, ESPN Stats & Information Group

The face validity of Opponent Receptions Plus/Minus is murkier in zone coverage. Denzel Ward is a former first round pick but has struggled at times this season, and Ross Cockrell has earned a grade of 61.3 in coverage from PFF this season. More worrisome, few of the players identified as positive contributors are on playoff teams. Zone accounted for roughly half of NFL coverage plays in 2019, so we would expect teams with elite zone defenders to have success.

The holy grail for a defensive metric is stability. Opponent Receptions Plus/Minus shows promise as a descriptive man coverage metric, but are players who are forcing incompletions now likely to continue their performance in the future?

Unfortunately for the metric — and for football analytics in general — the answer is no. How a player does in one season in Opponent Receptions Plus/Minus holds no predictive value for how he will do in the following season. This is also true for split seasons; how a player does in Plus/Minus for the first eight games of the year has no relation to his performance in the final eight games.

Gilmore is an interesting example of the overall instability at DB. In 2016 — his last year in Buffalo — his Opponent Receptions Plus/Minus for all coverages was -0.6. In his first year with New England, his Plus/Minus dipped to -2.76, but he took a leap in 2018 to 2.42. In 2019 his Plus/Minus was 7.97 — second-best in the league behind Malcolm Jenkins. But even in perhaps the best season of his career, he still helped Devante Parker to eight receptions and 137 yards in a loss that cost the Patriots a playoff bye.

Whether defensive coverage is inherently unstable and highly variable, or it’s just that we’re still in the infancy of football analytics, accurately predicting player performance in pass defense continues to elude us.

Check out our latest NFL predictions.