Guest Column by Jim Armstrong

We are entering the second decade of professional football statistical analysis on the web, which most prominently started with Doug Drinen in the early 2000s and, of course, Football Outsiders in 2003. These days, there are quite a lot of football stats sites on the Internet. Many of us host our own stats-heavy football sites and blogs, and some even write for mainstream media outlets and appear on national television. We publish research in academic journals and attend professional conferences. And, naturally, we don’t always agree with each other. We don’t always agree on how best to project the success of quarterbacks, regardless of whether they are recently-drafted rookies or aging veterans. We don’t always agree on the risks of overusing running backs, or injury prevention in general. We don’t agree on which teams are primed for success and which teams need to be blown up and rebuilt. And although we’re starting to tackle in-game strategies -- when to blitz, when to call time outs, which formations to use, what’s the optimal run/pass ratio -- for the most part, we have reached nothing remotely close to a consensus.

But there is one aspect of the game that virtually all statistical analysts agree on: NFL coaches should be going for it on fourth down much more often.

The statistically-supported idea that NFL coaches are too timid on fourth downs long predates the web. It started with a rather unlikely author: NFL quarterback Virgil Carter, a statistics major who taught college courses during the offseason. Carter teamed with systems engineering professor Robert Machol to publish a pair of academic papers in 1971 and 1978 which examined field position and fourth-down decisions. And in their 1988 seminal book on football statistics, The Hidden Game of Football, Bob Carroll, John Thorn, and Pete Palmer devoted a chapter to fourth-down decisions.

But the modern movement towards analyzing fourth-down decisions in the Internet age can be traced to Berkeley professor David Romer. In 2002, an early version his working paper on fourth-down decisions first appeared on the web and was updated and republished a few years later. This 43-page treatise included several pages of charts describing the recommended choices on fourth downs in various situations. This study remains today the most well-known and widely-cited academic paper on professional football among NFL analysts.

Since then, Romer’s paper has served as a launching pad for other writers to dissect, review, and reanalyze Romer’s findings that NFL coaches should be more aggressive on fourth downs.

William Krasker launched FootballCommentary.com in 2003. He used his dynamic programming model to analyze the fourth-down decisons encountered in NFL games.

On the pro-football-reference.com blog in 2006, Doug Drinen thoroughly examined the methodology behind Romer’s paper and its conclusions in a series of articles.

The authors of the ZEUS football tool, Frank Frigo and Chuck Bower, analyzed fourth-down decisions in NFL games from 2005-2007 and even provided an estimate of how many wins coaches cost their teams as a result of poor decisions.

Most recently, Brian Burke of advancednflstats.com has provided his own Expected Points analysis and has spent much of the past couple years second-guessing coaches’ calls on fourth downs.

Here at Football Outsiders, we created the Aggressiveness Index (originally appearing in Pro Football Prospectus 2006) to rank coaches based on how often they go for it on fourth downs. Although no NFL coach is as aggressive as the data suggests he should be, we discovered there is quite a wide range of fourth-down tendencies among coaches. To compute AI, we analyzed fourth-down decisions when the offense was in the opponent’s territory, where a coach’s tendencies were most distinguished from his peers. We also excluded obvious catch-up situations: Third quarter, trailing by 15 or more points; Fourth quarter, trailing by 9 or more points; Last five minutes of the game, trailing by any amount. AI measures how often a coach attempted a fourth-down conversion compared to the league averages in similar situations, based on the field position and the distance needed for a first down.

Below are the updated AI ratings for all 2011 coaches as well as the career rankings for 1992-2011. Not only does this show us which coaches are the gutsiest, it also identifies those with the most to gain from a more aggressive approach. Note that "opportunities" here lists the number of opportunities which qualified for measurement in AI, not total fourth-down go-for-it opportunities.

Aggressiveness Index, 2011 Rank Coach AI Att. Opportunities Rate x Rank Coach AI Att. Opportunities Rate

1 M.Tucker 1.396 2 11

18.2% x 19 K.Whisenhunt 0.868 4 35 11.4%

2 S.Spagnuolo 1.238 8

42 19.0% 20 M.Lewis 0.867 5 46 10.9%

3 Jim Harbaugh 1.184 11 74 14.9% 21 H.Jackson 0.866 7 45

15.6%

4 M.Smith 1.102

7 38 18.4% 22 L.Smith 0.863 5 44 11.4%

5 S.Payton 1.100 6 44

13.6% 23 P.Shurmur 0.860 3 44

6.8%

6 B.Belichick 1.076

8 44 18.2% 24 J.Caldwell 0.817 2 32 6.3%

7 N.Turner 1.070 4 42 9.5% 25 J.Garrett 0.816 3 46

6.5%

8 T.Coughlin 1.046

8 39 20.5% 26 M.Tomlin 0.801 5 42 11.9%

9 C.Gailey 1.001 6 37

16.2% 27 J.Schwartz 0.791 6 43

14.0%

10 P.Carroll 0.998

6 50 12.0% 28 T.Sparano 0.790 5 45 11.1%

11 T.Haley 0.989 3 27 11.1% 29 John Harbaugh 0.753 6 46 13.0%

12 A.Reid 0.983 9 46 19.6% 30 R.Crennel 0.743 1 9 11.1%

13 G.Kubiak 0.962 7 52 13.5% 31 Rex Ryan 0.732 4 41

9.8%

14 M.McCarthy 0.961

6 42 14.3% 32 L.Frazier 0.729 2 35 5.7%

15 M.Munchak 0.959 5 53 9.4% 33 T.Bowles 0.705 1 13 7.7%

16 M.Shanahan 0.933 6 50 12.0% 34 J.Fox 0.698 3 40 7.5%

17 R.Morris 0.917 8 38 21.1% 35 R.Rivera 0.661 2 36 5.6%

18 J.Del Rio 0.869 2 36

5.6%

Most Aggressive Coaches, 1992-2011 (min. 100 opportunities)

Rank Coach AI Att. Opportunities Rate

1 R.Kotite 1.88 37 211 17.5%

2 B.Belichick 1.86 173 810 21.4%

3 M.Smith 1.72 44 210 21.0%

4 G.Seifert 1.67 82 366 22.4%

5 S.Payton 1.61 52 252 20.6%

6 B.Parcells 1.58 142 608 23.4%

7 M.Ditka 1.47 29 166 17.5%

8 T.Haley 1.46 23 121 19.0%

9 J.Haslett 1.44 50 307 16.3%

10 T.Cable 1.43 19 116 16.4%

Least Aggressive Coaches, 1992-2011 (min. 100 opportunities)

Rank Coach AI Att. Opportunities Rate

79 J.Schwartz 0.711 16 122 13.1%

80 J.Fox 0.693 46 431 10.7%

81 M.Riley 0.685 9 140 6.4%

82 J.Jones 0.682 14 143 9.8%

83 M.Levy 0.675 42 277 15.2%

84 M.Holmgren 0.669 103 838 12.3%

85 Rex Ryan 0.641 18 164 11.0%

86 N.Saban 0.639 6 104 5.8%

87 M.Tomlin 0.614 25 231 10.8%

88 A.Reid 0.594 67 634 10.6%

Coverage of coaches’ timid fourth-down tendencies hasn’t escaped the mainstream media, either. The December 18, 2006 issue of ESPN The Magazine featured an article by Moneyball author Michael Lewis accusing coaches of being gutless, complete with an illustration of the cowardly lion from The Wizard of Oz wearing coaches’ gear and a headset.

The New York Times Fifth Down blog has also introduced contemporary analysis of fourth-down decisions to its wide audience.

Steven Levitt of Freakonomics fame was one of the editors of Romer’s paper prior to its publication in the Journal of Political Economy. Levitt has also written about fourth down decisions on his own site.

A year ago, Levitt wrote an article speculating that teams are indeed becoming more aggressive on fourth downs. Citing Falcons coach Mike Smith’s record of going for it frequently in short-yardage situations, Levitt argued that no coach would have done so ten years ago. Well, Levitt is wrong because Bill Parcells was that guy in the 1990s.

Earlier this month, in a piece for the Slate/Deadspin roundtable, Brian Burke lauded Sean Payton and Mike Smith for their aggressive fourth-down decisions in the playoffs. He also wondered whether these coaches are spearheading a slow evolution in fourth-down strategy.

Perhaps that’s all just wishful thinking. Have NFL coaches really learned anything from all the analysts? When we asked this question a year ago, the answer was "maybe," but with another year’s worth of data, the answer is more clearly "no." In fact, NFL coaches were as conservative on fourth downs in 2011 as they’ve ever been in the 20 years that we have play-by-play data, continuing a sharp two-year trend in timidity.

In particular, the decrease in fourth-down go-for-it attempts seems to be coming at the expense of increased field goal attempts rather than punts. As field goal accuracy improves, coaches are apparently becoming much more willing to take what they view as the "sure" three points rather than risk a turnover on downs.

You’ll notice that even long-time AI champ Bill Belichick has hit a relative conservative streak, dropping to No. 2 in the career rankings. What happened? Although Belichick continues to go for it frequently on fourth-and-1, his signature tendency had always been fourth-and-2, where he typically goes for it 33 percent of the time in AI-qualifying situations. That is more than twice the NFL average. However, in 2011, Belichick kicked the field goal on all seven qualifying fourth-and-2 opportunities. That was enough to drop him from the top slot, although he is still much more aggressive than most coaches. It's still fair to wonder whether we’re seeing the effects of the fallout from his memorable fourth-and-2 failure against Indianapolis in 2009, suggesting that even Belichick isn’t immune to the pressures of increased media scrutiny.

You’d think that a quick perusal of the career AI rankings showing five Super Bowl-winning coaches among the top seven would motivate more coaches to keep their punters and kickers on the sidelines. Apparently not. Analysts have also put forth many logical arguments in support of the data. If the coaches are listening, it hasn’t changed the way they approach their decision-making on game day.

When pressed about why they don’t go for it more often, coaches usually cite circumstances that make their situations different from the league averages on which the historical analyses are based: injuries, weather conditions, specific matchups, momentum, etc. It’s always something. And of course, there aren’t many conclusive statistics that take these factors into account because the sample sizes are too small, so it’s often impossible to refute the coaches on specific plays. Yet when we look at fourth-down tendencies collectively, NFL teams remain far more conservative than the statistics dictate. It’s almost like a reverse Lake Wobegon effect: coaches all believe that their teams’ abilities to convert fourth downs are below average.

So is there any hope for the coaches? Or should statistical analysts simply declare fourth-down decisions a dead subject and move on to more tractable problems? A couple of years ago there appeared to be promise, as 2007 and 2009 were historically aggressive seasons. But now, those years look like nothing more than random fluctuations. After all these years, we seem to be stuck back at square one. Somewhere, Virgil Carter isn’t too impressed.