No matter your league’s settings for starting pitchers, you’re forced to make decisions about whether to start or bench pitchers each time their turn in the rotation comes up. If you’re in a league with an innings or games started cap, you have to try to maximize the limited opportunities you have to start a pitcher. If you play in a league with no such limitations, the natural friction between the counting categories and the ratio categories forces you to make similar decisions. Sure, you can start Jake Odorizzi and take advantage of the tenth best strikeout rate among qualified starters. But his 4.23 ERA and 1.31 WHIP count, too.

The number one factor I consider when making such a decision is who the pitcher is facing that day. The strength of the other team’s offense against pitchers of my starter’s handedness usually determines whether I start or bench a pitcher I’m on the fence about. Matchup is probably the biggest factor for most fantasy owners when it’s not a must-start pitcher. I asked Twitter where the cutoff is for must-start guys. I got answers ranging from only Felix and Kershaw all the way to the top 25-30 starters.

Top 30 was sort of the number I had in my head. I recently wrote a fantasy football piece in which I examined how matchups affected fantasy production for the top 25 wide receivers last year. I just tested the correlation between a player’s weekly production and the strength of the opposing defense measured by pass defense DVOA from Football Outsiders. Some interesting results came from that exercise, but one thing that wasn’t a surprise was the lack of correlation between production and matchup for the top receivers. To amass enough points to finish the season among the best, a receiver has to accumulate points each week regardless of the opponent. Likewise, I thought the top starting pitchers would be matchup proof with the correlation growing stronger as we moved away from the elite guys.

Turns out that’s not the case. Well, the second part isn’t. The top pitchers didn’t have much of a correlation at all as expected, but neither did many of the pitchers on down the list. I was so expecting my hypothesis to be correct that I’m questioning whether the methodology for testing this was correct. What I did was take each pitcher’s individual game scores and compared them to how far above or below league average the opponent was against pitchers of the same handedness. For example, here’s a graph showing the relationship between John Lackey’s game scores and the strength of his opponents.

Lackey is an interesting example for two reasons. The first reason is anecdotal. I own Lackey in my big money league, and I’ve had him active for 135 of his 169.1 innings. In those 135 innings he’s had a 4.33 ERA and 1.33 WHIP. For the year he has a 3.77 ERA and 1.25 WHIP. I can assure you decisions to start or sit him were based purely on matchup. That clearly hasn’t worked out for me.

But that’s just one guy, right? Well, no. That brings me to the second reason. I tested the correlation between matchup and production for the top 50 starters on ESPN’s player rater. Again, I was expecting the relationship between the two to get stronger after the top 30 or so starters. But Lackey ranks 47th on the player rater. And as you can see below, Lackey is far from an outlier among pitchers outside the top 30.

PR Rank Player r-squared 1 Felix Hernandez -0.0072 2 Clayton Kershaw -0.0022 3 Johnny Cueto -0.0166 4 David Price 0.0732 5 Corey Kluber 0.0828 6 Chris Sale 0.0123 7 Adam Wainwright -0.0025 8 Garrett Richards 0.0273 9 Jon Lester 0.0046 10 Max Scherzer -0.0035 11 Masahiro Tanaka 0.1358 12 Julio Teheran 0.0742 13 Hisashi Iwakuma -0.0201 14 Madison Bumgarner 0.0011 15 Zack Greinke 0.0683 16 Tyson Ross 0.0778 17 Tanner Roark -0.0042 18 Scott Kazmir 0.0005 19 Rick Porcello -0.0145 20 Jake Arrieta 0.1818 21 Sonny Gray 0.0001 22 Doug Fister 0.1924 23 Lance Lynn 0.0549 24 Cole Hamels 0.0051 25 Jordan Zimmermann 0.0755 26 Stephen Strasburg 0.0192 27 Phil Hughes 0.0023 28 Yu Darvish 0.0165 29 Danny Duffy 0.0971 30 Hyun-Jin Ryu 0.1653 31 Jeff Samardzija 0.0053 32 Alfredo Simon 0.128 33 Chris Young -0.0108 34 Kyle Lohse 0.0178 35 Dallas Keuchel 0.1184 36 Henderson Alvarez -0.0213 37 Bartolo Colon 0.0071 38 James Shields 0.0153 39 Jered Weaver 0.0389 40 Ervin Santana 0.1597 41 Alex Cobb 0.2246 42 Tim Hudson 0.0146 43 Chris Tillman -0.0001 44 Alex Wood -0.0953 45 Matt Shoemaker -0.009 46 Wily Peralta 0.0013 47 John Lackey 0.0005 48 Collin McHugh 0.0125 49 Mike Leake 0.0239 50 Chris Archer 0.0611

According to that, my hypothesis could not have been more wrong. I guess I could have gone past the top 50 to see if the relationship strengthened at some point, but given that both the Twitter responses and my own practices considered top 30 to be about the cutoff for must-start pitchers, I didn’t think I needed to go further to prove that matchup may not be as decisive as I thought it was. That, and I doubt the relationship would have gotten any stronger.

Again, because it seems so intuitive that lesser pitchers will perform better against lesser opponents and vice-versa, I’m questioning my methodology. Is game score the right measure of production? I guess I could have used game-by-game point totals from a daily fantasy scoring system, but given that game score is just a points system that rewards pitchers similar to a daily scoring system, I doubt that would have changed the results all that much. Or maybe there is something else I’m completely overlooking.

But assuming I haven’t missed something here, matchup may not be as important as I/we think it is. Maybe it’s a better practice to just ask ourselves if the pitcher is any good. The definition of good is a movable target based on replacement level in your league, but that may be the only question you need to ask. In my big money league there is no innings or start cap. The 12 teams in the league carry about 10-11 starters on average. Fear of getting destroyed in the ratio categories is the only reason to leave a guy on the bench becuase the races for strikeouts and wins are so fierce. So in a league with with about 130 starting pitchers owned, why am I ever sitting Lackey, even if he’s facing the best offense in the league? He’s so far above replacement level that matchup should go right out the window.