Seven years ago, our good friend Scott McKinney compiled a great study on the success rate of prospects, using a sample of prospects ranked by Baseball America from 1990 to 2003. That article has been cited countless times, not just here, but on FanGraphs, Fivethirtyeight, and other sabermetric sites. While perhaps it’s not as important as DIPS theory in the realm of sabermetrics, it meant a lot to understanding prospects.

In his piece, which you can read in its entirety here, Scott found that about 70% of Baseball America top 100 prospects fail and that position player prospects succeed much more often than pitching prospects. He also found that prospect success rates have not improved much over time.

This somewhat confirmed what we may have thought - prospects are far from a sure thing - but I was surprised by the rate being a bit higher than I imagined. While Scott’s piece was great, I wondered if maybe there was a different way to look at it?

Scott just used two wins as the benchmark for success or failure, a fair value, but rather rigid and binary. That isn’t Scott’s fault as its (a) impossible to assign a single overall value for success and (b) even if 1.9 counted as a failure, he had to draw the line somewhere. While I think Scott did the best he could (and a great job by any measure), I think the benchmark for success shouldn’t necessarily be stagnant.

So when thinking about a different way to look at success rates of prospects, I wanted to change the way we define success. Instead of a stagnant number, what about a moving number? Let’s try to define success in the context of that individual player.

So what I came up with was what I’ll call relative success, as determined by a prospect succeeding/failing to meet the average value of prospects who were ranked similarly. This is similar to how you would benchmark the success of investments. An extremely risky investment shouldn’t return the same as an extremely conservative one, so we need to define relative benchmarks for both.

This makes sense with prospects, right? You wouldn’t necessarily grade the #1 overall prospect the same as the #100 overall prospect in terms of success? The expected return on the #1 prospect is much higher than the expected return on the #100 prospect. If you were trading for both, you’d pay more for the #1 prospect than the #100 prospect, thus you’d expect a better performance (higher return).

So what I’ve done is the following:

Separated every top 100 prospect (per Baseball America) from 1993-2013 into buckets

Sorted each player by their highest ranking in any given year (so a prospect who was ranked #30 in 2005 and then #1 in 2006 would only be counted as #1)

ranking in any given year (so a prospect who was ranked #30 in 2005 and then #1 in 2006 would only be counted as #1) Found the average fWAR for that bucket

Found how many prospects beat/failed the average fWAR of that bucket

Now 2013 is running a bit up against the wall from a standpoint of time for success but I felt that was enough time (six years) for them to at least debut and have some success. Scott wrote his piece in 2011 (using 2010 as the most recent season) and started with 2003 (an eight year span). There aren’t likely any players who are on the 2013 list that are still rookie eligible and likely to have an impact (sorry Mike Olt), and if there are, Baseball America probably ranked them too high (sorry Jurickson Profar - even though you were injured).

I’ll note that I didn’t use the bucket average as a strict cutoff, but gave a -0.5 win buffer. Maybe there should be more, but like Scott did, there has to be a cutoff somewhere.

Success Rates

Top prospects (as defined in the 1-10 bucket) succeeded the most, but 60% of them failed to reach the average performance of their peer group. Prospects in the 11-20 and 21-30 range failed (or succeeded, depending on how you look at a glass of water) at a similar rate, similar to prospects in the 41-50 bucket. Prospects in the 31-40 bucket failed at a higher rate than their close bucket (probably not that meaningful of a difference or value).

Regardless if you are looking at the buckets of ten or buckets or twenty on the right, higher rated prospects succeed at a higher rate than their lower bucket peers.

Compared to Scott’s original rates:

Prospects in the 1-10 and 11-20 buckets failed at a 15% higher rate using relative success than using Scott’s benchmark. On the other end, prospects in the 61-70, 71-80, and 81-90 bucket succeeded at a higher rate under my benchmark than Scott’s. This makes sense as lower rated prospects are more likely to clear the 2-win hurdle compared to a 5-7 win one.

Hitter and Pitcher Rates

Using the twenty buckets, top rated hitters (1-20 bucket) succeeded at a 7% higher rate than their pitching counterparts, whereas pitchers in the 21-40 bucket had a slightly higher upperhand. The same dynamic tradeoff can be found in the 41-60 and 61-80 buckets. Back rated prospects (81-100) succeed at a similar rate regardless of position type.

Hitters and pitchers succeeded at roughly the same rate when you look at the average WAR for the type. First basemen and second baseman succeeded at a higher rate than their hitting peers whereas third basemen and left handed pitchers failed at a higher rate than their type peers.

A theory: 1B and 2B are the two smallest groups to make top 100 lists (it’s really hard to make a top 100 list as a 1B typically). While their average WAR isn’t that low (1B is actually among the highest), there is some selection bias because only good 1B make a top 100 list. Compared to say, RHP, many RHP who make a top 100 list are 50/50 chance of being a reliever or starter. If they end up as a reliever, they have a very high bar to clear to reach eight fWAR than their right handed starting peers.

On a positional and bucket standpoint (comparing the average fWAR for that position and bucket - ie: 1B in the 21-40 bucket):

*A note about 61-80 bucket 1B: the average fWAR for that group was effectively replacement level, so even being slightly better than replacement here counted as a win.

Overall

(link to full size image)

As you’d imagine, the #1 overall prospect has accounted for the highest total value of a top 100 list, bringing in 5% of the total value with the #2 and #3 ranking being worth 3.8% and 2.77% respectively. It’s probably not worth reading into that the #7 and #10 overall prospect has been worth more combined than the #3 prospect (thanks to Pedro Martinez for #10).

#1 to #10 prospects account for ~35% of the total top 100 list WAR, with the next three buckets being worth ~11% each, and the final five being worth ~5% each, meaning the 1-10 bucket has been worth more than the 51-100 buckets combined.

From what I’ve found top 100 prospects have accounted for ~55% of the league's WAR from 1991-2017. This somewhat jives with what Jeff Sullivan of FanGraphs found recently that ~41% of the best players in the league weren’t top 100 prospects.

Putting a dollar value to it

One of my favorite research pieces comes from The Point of Pittsburgh, where they annually update the surplus value of prospects based on past research they have done. In their exercise, they look at surplus value, or value above what they are being paid ($1.6M for the first three years and then a discounted present value WAR due to arbitration). I’m going to do it a little differently.

Assume the cost of a win in 2018 is worth ~$8.5M (and that might be low)

Add 5% inflation per year

A player is under control really for seven years (we are assuming that top prospects are going to be kept down to gain an extra year of control at the cost of possibly paying them as a super two - a tradeoff teams are happy with for star players)

Spread their total expected WAR (the average bucket value) over the following years:

2018: 5%

2019: 10%

2020: 15%

2021: 15%

2022: 15%

2023: 20%

2024: 20%

This assumes that players start off just okay to begin with and then get increasingly better before turning into their best controlled years in the final two (increasing their cost and value).

Discount the dollar value by the success rate

Point of Pittsburgh broke theirs out a bit differently, but some of the values are the same. The biggest discrepancy is in the 21-40 bucket (they grouped by 11-25 and 26-50), while mine were a few million shy by the average value of those two buckets. I prefer theirs to mine here, as intuitively it doesn’t make sense that 21-40 prospects are worth less than 41-60 prospects. Their value are likely close, but not $12M difference.

Organizational Rates

Like Scott, I wanted to see which organizations were most successful at developing prospects over this time, although with the same understanding of the limitations of using this as a way of assessing clubs. For one, the sample sizes are too small - teams only put a handful of prospects on these lists, even over a decade. Second, some players were drafted by one team and developed by another making it difficult to evaluate.

So using Scott’s 2-win benchmark, below are the team success rates for prospects. The Nationals have been the best at getting success from their top 100 prospects (though they have the lowest number of prospects appearing given that they have only been around since 2005). The Phillies have been the best at getting production from their prospects, as each top 100 prospect has averaged ~14 wins.

*Note - the Nationals do not include the Expos, who are absent from the above list but were included in everything else. I decided to exclude the Expos from the Nationals for several reasons:

Scott’s study went from 1993-2003, and the Expos were defunct in 2004. The only missing players from Scott’s list and mine would be Clint Everts and Mike Hinckley who made the 2004 Baseball America top 100 list. Everts never made the major leagues and Hinckley threw 23.1 replacement level innings from 2008-2009.

The two teams had different ownership and management. The Nationals don’t really recognize the Expos even much themselves, Tim Raines isn’t wearing a Nationals hat in the Hall of Fame, and Montreal baseball fans don’t really like the Nationals that much.

The fail rate of prospects has not changed much since the original study. Prospects still fail more than they succeed no matter how you look at them, from a binary or relative basis. While with Scott’s study, about 55% of top end prospects succeeded, only about 40% did in my study. I think Scott’s benchmark works well on the lower end, but it is too kind on the upper-end of prospects who we should expect more from than two wins. I don’t expect my findings to break the internet or change the way we think about prospects, but I think this shows a different a good way to think about what constitutes a “successful” prospect.