<< Back to main

?John Dewan makes this startling claim:

One of the public perceptions has been that a player needs three full seasons before his defensive metrics provide a true indication of his defensive abilities. That has been my own personal rule of thumb, though I’ve known there is some reliability to sample sizes smaller than three years. Based on the new research, BIS has found that Defensive Runs Saved based on as small a sample size as 350 innings in the field (about a quarter of the season) produces reliable results. This is a very significant finding.

And by reliable, it looks like he’s after r=.50 or greater, which is reasonable enough (more signal than noise), and so, closer to 0% regression toward the mean than 100%. But, I’m not buying it, not for a second.

I did a quick check by grabbing the data from Fangraphs, 2013-2014 data. I have the DRS, DRS/9Innings, and Innings. John makes the claim that he gets r=.55 after using only 350 Innings. Not a minimum of 350 innings, but exactly 350 innings.

If I use a MINIMUM of 350 innings over each of 2013 and 2014, at a particular position, I get 210 fielders. The AVERAGE innings in 2013 is 933 and in 2014 is 938, which is the equivalent of 104 full games. My correlation, if I use DRS/9innings is r=.48. If I use just DRS, it’s r=.46.

If r=.48 when I use an average of 104 full games, then I’d get to r=.50 with an average of around 113 games (a regression amount of roughly 1000 innings).

When I used to do this with UZR several years back, I’d use a rule of thumb of r=.50 at games=100 (though obviously less for SS and more for 1B). The results I see here from DRS data is in the same ballpark, and so, I see no reason to change my opinion on that.

I have no idea what John did, since he says the details are in his forthcoming book. Maybe he meant a MINIMUM of 350 innings, and he (probably) separated by position (meaning he ran one correlation per position as opposed to what I did here for a sniff test), and he did use more years. So, all of that would match what I’ve done. So, if he meant a minimum of 350 innings (implying an average of around 900-1000 innings), that’s fine, and it’s consistent with what we’ve always known.

But if you had EXACTLY 350 innings (or taking the first 350 innings or a random set of 350 innings), the correlation would come in at around r=.26.