I've written several posts discussing the 2015 Hall of Fame ballot, and one of the things that struck me was the extreme difficulty in evaluating relievers. Several factors require serious consideration:

Treatment of role players -- A typical closer pitches 60-70 innings a year out of around 1,500. Do they deserve the same status as starters who pitch far more innings or the position player who plays daily? With only four relievers in the Hall so far, it appears to be an open question.

Separating players -- Mariano Rivera set the standard for relievers, not just for his generation but for all of baseball history. Subsequent voters will have to work hard to not say to themselves "Well, Pitcher X, he was no Mariano Rivera." That's not a fair standard -- it's like saying Ken Griffey Jr. was no Willie Mays.

Proper comparisons -- My pet peeve has always been comparisons between apples and oranges, so in my mind, the proper comparison is between relievers and relievers, to see how they compare to each other and to not lump them with other pitchers or position players.

Use of the closer -- In today's game, the closer only appears in save situations, absent extenuating circumstances. This means his team is ahead and he enters the game with the bases empty. I'm not suggesting this is easy, since there's plenty of pitchers who have shown just how difficult this is, but does the number of saves denote achievement, let alone greatness?

This is from a Tableau data viz (view it yourself here):

The Bill James Hall of Fame Monitor and Career Standards depict how players should be rated, but they don't work for relievers since James never created criteria specific to their role. Earlier this year, Beyond the Box Score Associate Editor Ryan Morrison wrote a post that proposed using RE24 as a measure of reliever effectiveness. This is the definition from the FanGraphs glossary page:

RE24 (runs above average by the 24 base/out states): RE24 is the difference in run expectancy (RE) between the start of the play and the end of the play. That difference is then credited/debited to the batter and the pitcher. Over the course of the season, each players’ RE24 for individual plays is added up to get his season total RE24.

For closers, every at-bat is important, since in modern usage they're only in the game if it's a high leverage save situation. If RE24 can work for a given year, it can certainly be used for career evaluation purposes as well. This table shows the top relievers in RE24 since around 1940:

Row Labels From To G W L rIP BF SV BS SD MD RE24 Mariano Rivera 1995 2013 1105 79 57 1233.7 4870 652 73 584 121 367.2 Billy Wagner 1995 2010 853 47 40 903.0 3600 422 66 389 100 201.2 Hoyt Wilhelm 1952 1972 982 112 95 1780.7 7258 216 33 364 207 199.3 Rich Gossage 1972 1994 965 115 85 1556.0 6408 310 97 434 195 184.9 Trevor Hoffman 1993 2010 1034 61 75 1088.7 4388 601 73 518 138 181.6 Lee Smith 1980 1997 1015 71 87 1251.7 5230 478 99 478 188 172.6 Joe Nathan 1999 2014 747 50 29 754.3 3019 376 45 346 69 169.2 Rollie Fingers 1968 1985 907 107 101 1505.7 6104 341 80 424 195 167.6 Keith Foulke 1997 2008 611 40 33 748.7 3028 191 35 248 75 163.2 Jesse Orosco 1979 2003 1248 87 78 1276.7 5374 144 64 356 220 154.4 Tom Henke 1982 1995 642 41 42 789.7 3194 311 52 289 98 144.2 Tug McGraw 1965 1984 785 89 69 1301.3 5391 180 37 298 149 142.2 Kent Tekulve 1974 1989 1050 94 90 1436.7 6001 184 67 362 226 141.5 Francisco Rodriguez 2002 2014 799 46 41 835.3 3442 348 63 383 95 139.5 Jonathan Papelbon 2005 2014 590 35 29 611.3 2455 325 44 300 62 134.9 Arthur Rhodes 1995 2011 840 69 48 865.0 3566 33 45 311 127 133.8 Dennis Eckersley 1975 1998 710 48 41 807.3 3204 390 62 323 107 128.0 John Hiller 1965 1980 502 72 57 963.0 4054 125 43 202 118 125.6 Steve Reed 1992 2005 833 49 44 869.7 3661 18 41 245 132 125.6 Mike Jackson 1986 2004 998 61 63 1154.7 4810 142 58 337 168 125.3

rIP = relief innings pitched BF = batters faced BS=blown saves SD = shutdowns MD= meltdowns.More data can be seen in this Google Docs spreadsheet

Games shown are relief appearances only, so for example, Mariano Rivera's data doesn't include the ten games he started at the beginning of his career. There are some surprises (Keith Foulke?), but generally it helps show the distinction between effective relief and just saves, since even in the brief history of the save, closer utilization has changed.

This will be the 13th year Lee Smith is on the Hall of Fame ballot, and this is the HOF voting pattern:

He had been building momentum, but ran into a brick wall in 2014. Viewing him using RE24 places him right up there with every other top reliever in baseball history not named Mariano Rivera, and look at the other players around him -- those players will give HOF voters fits when their time for consideration comes. I tend against enshrinement for Smith, since his was more a lengthy career than one marked with a sustained run of being considered among the best closers in the game.

We need better ways to evaluate relievers than saves. RE24, as Ryan suggested in his post, shouldn't be used by itself (no stat should), but it does an excellent job of cutting across time and showing who the effective relievers were.

All data from Baseball-Reference and FanGraphs

Scott Lindholm lives in Davenport, IA. Follow him on Twitter @ScottLindholm.

