In his response to last week’s news that the Nationals had fired manager Matt Williams, The Washington Post’s Adam Kilgore described a scene from the fourth and final game of the 2014 National League Division Series, when Williams went to rookie reliever Aaron Barrett with one out and two on in a seventh-inning tie. Citing the proverbial “person familiar with the situation,” Kilgore wrote that “Giants Manager Bruce Bochy turned giddily incredulous that Williams had not instead chosen all-star Tyler Clippard, whom Bochy feared because of his experience and ability to face both left-handed and right-handed hitters. As Barrett warmed up, the person said, Bochy expressed to one of his coaches that Williams’s decision had just given the Giants the series.” We’ll never know what would have happened had Williams signaled for someone else, but Bochy wasn’t wrong: Barrett lost the game, walking Hunter Pence to load the bases and throwing a wild pitch to Pablo Sandoval that scored the winning run. The Nats slipped into the offseason without turning to Clippard, Drew Storen, or Stephen Strasburg (who was available out of the pen) for salvation in the decisive game.

More 2015 MLB Playoffs World Series Weekend: Five Qs

Citi Field Psyops: Step by Step

World Series Wrap: Cueto's Gem

World Series Wrap: Game 1 Thrills

The Biggest Plays in Baseball History

Your World Series Rooting Guide

Keys to Mets-Royals World Series

The National League Has a DH

Our Picks for Cy Young, MVP, More

More 2015 MLB Playoffs Coverage! We’ve all been Bochy, either giddy or aghast about a major league manager’s decision to commit suicide by bullpen. We all know what it’s like for a soul to be scarred by the memory of a game that might have ended differently had a manager not summoned the last guy in the world everyone watching wanted to see. And we know that these emotions are particularly acute in October, when the importance of every managerial move is magnified. Some managers are known for making mind-blowing bullpen mistakes, but their bad bullpen reps are supported less by numbers than by an amorphous anxiety among fans as the inning gets later and the starter’s pitch count climbs.

There’s no well-known statistic we can use to quantify bad bullpenning. But as the Grinder would say, “What if there was?” After the 2009 season, former Baseball Prospectus author and quantitative consultant Tim Kniker developed a managerial reliever-use rating system, which he dubbed Bullpen Management Above Random. BMAR’s purpose is simple: It’s a way of assessing how closely each manager lined up his best relievers with his team’s highest-leverage relief opportunities. Essentially, it’s a stat that tells us how often a manager’s bullpen moves made fans lie awake and wonder, “Why him?”

Some of the managers who ranked most favorably in the BMAR standings in 2009 (Joe Girardi, Mike Scioscia, and, yes, Bruce Bochy) are still with us, while others (Ron Gardenhire, Ken Macha, Tony La Russa) are no longer active. Several seasons have passed since the original research, so I asked Kniker to bring BMAR up to date by generating ratings for 2012 to 2015.

BMAR rates managers based on their relievers’ Weighted On-Base Average allowed to righties and lefties over the previous 12 calendar months. BMAR’s baseline — the standard by which all managers are judged — is a team’s theoretical distribution of bullpen outings had the manager assigned relievers using eeny, meeny, miny, moe. To produce the ratings, Tim sums each bullpen’s actual totals of left- and right-handed batters faced and calculates how much more effectively the manager could have distributed the same workload, given the relievers on his roster at the time. You can think of each manager’s BMAR as an answer to this question: In light of the bullpen he had, how much better (in wOBA points allowed) were the relievers he did choose than the relievers he could’ve chosen at random?

As an example of a high-leverage matchup that BMAR deems misguided, consider the eighth inning of an August 18 game between the Dodgers and A’s. With the Dodgers up 4-1, right-handed Dodgers reliever Pedro Baez replaced Clayton Kershaw to begin the bottom of the eighth and allowed three consecutive hits, trimming Los Angeles’s lead to one run. After Baez got Billy Butler to ground out, Dodgers manager Don Mattingly made a sensible choice, calling on lefty reliever J.P Howell to face Oakland left-handed hitter Josh Reddick. But Reddick managed an infield single and then Mattingly made a mistake: He kept in the lefty to face right-handed hitter Marcus Semien, who lined a single to left, scoring one run and tying the game; Oakland would go on to win 5-4 in the 10th inning. It was the first time the Dodgers had blown a three-run lead in the eighth inning or later since 2011.

Commenters in the game thread at Dodgers Digest split their anger between the bullpen and the bullpen management. “Our pen is a random mess,” wrote one commenter, who noted that “JP is horrible with righties” and that leaving him in was “[expletive] dumb.” As Chad Moriyama wrote in his recap at the same site, “Kenley Jansen threw zero pitches, by the way.” Howell’s wOBA allowed to right-handed hitters over the previous 12 calendar months was .356. Compare that to the wOBAs allowed to right-handed hitters over the same span by the other options in the Dodgers’ pen: Jansen (.200), Yimi Garcia (.279), Luis Avilan (.288), Chris Hatcher (.315), and Jim Johnson (.322), who relieved Howell to secure the last out of the inning. From BMAR’s perspective, the optimal choice was Jansen, and the “random,” baseline choice was .291 — the weighted-average wOBA allowed to righties of the bullpen as a whole. BMAR holds the difference between the baseline and Howell against Mattingly, whose error was compounded by the situation’s high stakes. The more make-or-break the spot, the more BMAR makes it matter, and Howell’s ill-fated faceoff with Semien had a Leverage Index of 4.9 (where the average is 1.0).

A couple of caveats: First, while we know which pitchers were on the active roster on a given day, we don’t know which pitchers were actually unavailable (or less effective than usual) because of overuse, illness, nagging injury, or a stubborn latch on the bullpen-bathroom door. And we’re assessing only one aspect of bullpen management: which reliever the manager chose once he decided to make a move. If some managers are better than others at judging when to take out their starters, or at keeping relievers fresh by limiting how often they warm up or enter games — and some of them undoubtedly are — then this method will miss that extra value. And since we’re simply reassigning each team’s actual matchups against left-handed and right-handed batters to the most logically leveraged scenarios based on ability, we’re also overlooking the ability of some managers to gain the platoon advantage more often. But we can judge managers independent of bullpen quality: To adjust for the fact that some managers have better late-game relievers to work with, and can therefore make more of a difference by choosing their best available reliever at a crucial moment instead of their mop-up man, we’ll compare each manager to a personalized standard of how well he could have done with his own array of relievers.

The table below shows 2015 BMAR ratings for every manager, excluding Dave Roberts, who guided the Padres for one game between Bud Black and Pat Murphy. The second column, “Reliever BF,” tells us how many batters the bullpen faced during each manager’s tenure. The third, “wOBA Points Saved,” shows how much better the arms each manager chose were expected to be, relative to random selections. And the fourth, “% of Optimal,” tells us how close each manager came to the maximum difference between the baseline and the best option available on his team — in other words, how close he was to making the perfect move at every opportunity.

Manager Reliever BF wOBA Points Saved % of Optimal Bud Black 446 .021 38.7 Joe Girardi 2222 .022 37.2 Mike Scioscia 2013 .015 36.5 Ned Yost 2160 .017 35.0 Kevin Cash 2214 .012 34.0 Pat Murphy 1187 .012 32.6 Pete Mackanin 1287 .014 26.8 Robin Ventura 1848 .010 26.5 Clint Hurdle 2102 .010 26.0 Bryan Price 2321 .010 25.8 Buck Showalter 2137 .012 24.7 Dan Jennings 1637 .013 23.7 A.J. Hinch 1830 .007 23.4 Ryne Sandberg 1067 .010 22.6 Paul Molitor 2130 .009 21.1 Torey Lovullo 304 .011 19.5 Bob Melvin 2141 .009 19.0 AVERAGE — .008 18.2 Joe Maddon 2123 .008 17.9 John Farrell 1835 .008 17.1 Jeff Banister 2124 .007 16.9 Don Mattingly 1947 .008 16.8 Lloyd McClendon 2200 .009 16.1 Matt Williams 1939 .008 16.1 Brad Ausmus 2161 .008 15.2 Mike Redmond 516 .008 14.2 Bruce Bochy 2031 .006 13.4 Terry Collins 1884 .005 12.7 Fredi Gonzalez 2083 .005 9.8 Walt Weiss 2460 .004 8.1 Terry Francona 1875 .002 6.4 Craig Counsell 1850 .003 6.2 Ron Roenicke 316 .002 4.5 Chip Hale 2385 .001 2.3 John Gibbons 1907 .000 1.4 Mike Matheny 2011 -.005 -18.3

Among all managers, there isn’t much of a year-to-year correlation in BMAR, which suggests that a single season’s ranking isn’t all skill. Still, certain names tend to rise to the top or sink to the bottom year after year. The following table shows 2012 to 2015 weighted averages for every manager who was active in at least three of those four years. The first thing to notice is that even the best managers didn’t come close to what BMAR considers optimal bullpen usage, which would include unheard-of ideas like bringing in the closer in non-save situations, even — brace yourself for some serious heresy — tie games on the road. This might explain why fans of almost every team seem so convinced that their manager has no idea what he’s doing. The second thing to notice is that 2009 standouts Girardi and Scioscia are still at the top. The third thing to notice is Mike Matheny, whom BMAR says was actually worse at running his bullpen than a random number generator wearing a warm-up jacket. You might remember Matheny from his odd last act of the 2014 season, which was using struggling starter Michael Wacha, for the first time in weeks, in the ninth inning of a tied elimination game, instead of a rested closer whom he ruled out because of bullpen orthodoxy. If we believe BMAR, the Cardinals would have had better bullpen matchups this season had Matheny made his moves by closing his eyes, spinning around 10 times, and gesturing in the pen’s general direction. (Maybe Matheny’s new iPad will help.) All snark aside, though, these rankings are another reminder that the manager’s impact pales in comparison to the quality of his players. Matheny and the Blue Jays’ John Gibbons may have been among the worst bullpen managers this season, but they didn’t prevent their teams from taking division titles.

Manager wOBA Points Saved % of Optimal Joe Girardi .014 28.8 Mike Redmond .010 26.0 Bud Black .013 25.7 Bruce Bochy .012 25.5 Ned Yost .011 24.1 Mike Scioscia .011 24.0 Buck Showalter .010 23.2 Clint Hurdle .008 22.4 Bob Melvin .010 22.3 Fredi Gonzalez .011 21.9 Joe Maddon .009 21.3 Don Mattingly .009 19.6 AVERAGE .009 18.9 Robin Ventura .008 18.4 Ron Roenicke .008 18.0 Ryne Sandberg .007 17.7 John Farrell .009 16.7 John Gibbons .006 14.8 Terry Francona .005 13.1 Walt Weiss .005 11.3 Terry Collins .005 11.1 Mike Matheny .003 7.5

Matheny — who told USA Today’s Ted Berg in April that he uses three-year statistical stats as a “tiebreaker” when making bullpen moves but believes that managers “first and foremost have to trust our gut” — ranks at the bottom over a multiyear sample as well as in 2015. Mets manager Terry Collins and Toronto’s Gibbons, who made the perplexing decision to use David Price in low-leverage relief in Monday’s Game 4, aren’t far behind. But Girardi is BMAR’s golden boy.

Girardi is already regarded as a bullpen whisperer; BMAR just clinches the case. In April 2014, Fox Sports reporter Ken Rosenthal passed on the praise of a “rival AL exec” who’d told him that Girardi was “the best [he’d] seen at managing a bullpen.” The BBWAA is onboard: Girardi (who won the NL Manager of the Year Award in 2006) hasn’t finished lower than sixth in AL MotY voting since 2008, even though his high-payroll teams have missed the playoffs twice. After eight seasons in the country’s largest media market, Girardi’s bullpen use still gets a seal of approval from both bloggers and tabloid beat writers, two tough-to-please constituencies. It’s even more impressive that Girardi leads in BMAR given his reluctance to use relievers three days in a row, which limits his options in some high-leverage situations: This season, the Yankees ranked 29th in reliever appearances on zero days’ rest, but they also ranked first in multi-inning outings. For what it’s worth, the Yankees have exceeded their Pythagorean win-loss record by an average of two games per season during Girardi’s years at the helm, as have the Angels during Scioscia’s 16 years in Anaheim. Bochy’s teams have exceeded their Pythagorean records by an average of one win during his 21 seasons.

With Girardi’s Yankees eliminated from the postseason, the best remaining bullpen manager by BMAR belongs to the Royals. (So much for the notion that stats have nothing nice to say about Ned Yost.) Black, a former pitcher and bullpen coach, appears to have been something of a bullpen savant, and Mike Redmond, another manager who lost his job in 2015, also seemed to excel in this area. Buck Showalter, who distinguished himself by being unusually flexible with his closer and setup men in the 2014 ALDS, rounds out the top seven on the multiyear list.

According to Kniker’s wOBA-to-wins conversion, the difference between the best and worst manager by BMAR in a given season is on the order of four wins: not enough to turn a good team into a bad one (or vice versa), but more than enough to make the difference in a tight race (or a must-win game), even though there are elements of bullpen management that this method still can’t capture. So if you believe that Girardi deserves partial credit for leading the Yankees to winning records with negative run differentials from 2013 to 2014, or for winning a wild-card spot this season with Andrew Miller and Dellin Betances, BMAR has your back. And if you’re a Cardinals fan feeling a little extra Matheny-related agita about today’s must-win Game 4 against the Cubs, BMAR won’t bring you back from the ledge.

Thanks to Tim Kniker, and R.J. Anderson and Rob McQuown of Baseball Prospectus for research assistance.