While I was on my daily ambulation through the equivalent of the basement archives at Baseball Reference (the manager pages), I came across a strange fact: Kevin Cash, rookie manager of the Tampa Bay Rays, has challenged nine times this season and has not won a single time. That seems very strange, given the fact that managers generally have a good idea of when they’re going to win challenges nowadays: it’s why they stand at the top of the dugout steps while someone looks at video before they actually challenge.

We’ve had just over a year of the challenge system in major league baseball, and already we have a good idea of the types of challenges managers are most likely to win, the ones they’re likely to lose, and the ones that still seem to go either way for reasons that haven’t been and might never be fully established. That understanding is engrained in managers to varying degrees, and we now have enough data on the subject to identify the ones that seem to get it and the ones that don’t.

So, how and why exactly has Cash gone 0-9 so far this year? Does it matter that he seems bad at winning challenges? It also got me thinking: how have other managers fared, and are we thinking about this in the wrong way?

Needless to say, challenges are a tiny part of the massive equation of manager evaluation, but they’re certainly an interesting one. There was some great work done By Jesse Wolfersberger at The Hardball Times on when managers should and should not challenge that can inform our discussion today. This is the main part of the article that interests us, on the frequency of challengable plays (though you should certainly read the whole thing):

“So, should a manager challenge a one-on, one-out play in the first inning? According to my analysis, the likely answer is ‘yes.’ I found that the expected number of any over-turnable plays in a given game is less than one. Further, the expected number of over-turnable plays that go against your team in a medium-to-high leverage situation is far less than one. Bottom line for managers: If there is a call that you have even a 50/50 chance of reversing, you should probably challenge, no matter the inning or the game situation. The odds are, there won’t be a better time to use that challenge later in the game.”

That final point is the kicker — when faced with half a shot at winning a challenge at any point in the game, do it. First, let’s take a look at every manager’s tendency to challenge, and their success rate in doing so. I’ve charted out all of the managers using the data from Baseball Reference’s pages, with new managers from 2015 at the bottom of the chart to avoid small sample size confusion. Managers who were replaced mid-season last year are grouped together as one:

Joe Girardi is great at winning challenges, and the same could be said for embattled Marlins manager Mike Redmond. John Gibbons and Walt Weiss not so much, especially the former, who has one of the highest overall number of challenges and also the lowest rate of winning them. Down in our new manager section, we have three out of five off the mark with at least one overturn, then we have Paul Molitor and Kevin Cash. Molitor has only challenged once, so let’s key in on Cash, and his glaring 0-fer.

I’ve compiled the nine challenges he’s initiated this year below to try to see what’s behind his challenge strategy. I’ve also put the win probability shift for the team that eventually won the challenge, which is the team that isn’t the Rays in every circumstance.

#1: 4/6/15 vs. Baltimore, bottom 5, 2 out, down 0-3. Winning team win probability shift: 3%

James Loney was clearly out at home on a single to right field to end the inning, but this is one of those gray area calls that isn’t bad to challenge — the home plate collision rule still seems to be misunderstood, or rather it’s still subjective and ill-defined. Still, through most of 2014, home plate collisions were overturned only 15% of the time, one of the lowest rates of all types of plays. If won, the Rays would have been down 3-1 with a runner on first. However, Loney was obviously beaten by the throw, so unless the umpires thought Caleb Joseph was blocking the plate, it would stand. And it did.

#2: 4/8/15 vs. Baltimore, bottom 4, 2 out, score even. Winning team win probability shift: 1%

This was an incredible play by Manny Machado (it is one of the few plays labelled “remote” by our advanced metrics at 3B this year), and Logan Forsythe beats the throw. However, he was called out, and he steps in front of the bag without touching it, which is clear from many of the replays that were shown immediately afterward. Something that would become a recurring theme as I went through these challenges is that Kevin Cash pops straight out of the dugout after the play; there’s no waiting at the top of the steps while someone watching video tells him if he should challenge. There was no evidence that this out call should have been overturned, so it wasn’t, which is something the bench coach should have relayed to the manager.

#3: 4/14/15 vs. Blue Jays, top 6, 2 out, score even. Winning team win probability shift: 3%

Here is where Cash should’ve gotten his first overturn. This was Desmond Jennings getting called out stealing second base with a tag that clearly missed. In a tie game in the sixth inning, a runner in scoring position is pretty valuable, and this was a no-brainer. Even though the tag looked like it missed from multiple angles, New York sided with the original call, and Cash was spurned again.

#4: 4/15/15 vs. Blue Jays, top 2, 1 out, down 2. Winning team win probability shift: 3%

Another day, another Desmond Jennings out call at second base. The play was very close, but again, this is not a call that gets overturned unless there is clear evidence against the initial ruling, and there isn’t from the replays. This was another play that Cash popped out of the dugout on before having a chance to be briefed on video from a coach or staff member.

#5: 4/16/15 vs. Blue Jays, bottom 8, 0 out, down 4. Winning team win probability shift: 9%

The third day in a row Cash would lose a challenge, this high and tight pitch was judged to hit Josh Donaldson on the hand with two men on base. These are also difficult to overturn — FiveThirtyEight puts them at 43% overturned through most of 2014 — especially when the pitcher almost hits the batter in the head. Cash was ejected immediately after the call was confirmed, citing “frustration with the process” in a postgame interview. He also did not appear to wait for video review on this play in the dugout before deciding to challenge.

#6: 4/18/15 vs. Yankees, top 7, 0 out, down 2. Winning team win probability shift: 0%

Another play at the plate. Chase Headley was ruled safe, and is obviously safe on replay. Even the Tampa Bay announcers sounded resigned on this one:

“The way the Rays track record has been going…” “The track record has been nil.”

The call stood, like home plate/clear tag calls usually do. This wasn’t a bad point in the game to challenge given the circumstances, even if there wasn’t much hope for this one.

#7: 4/19/15 vs. Yankees, bottom 6, 1 out, down 2. Winning team win probability shift: 2%

Did Mark Teixeira come off the first base bag before he caught the ball? The initial ruling said he stayed on, and there wasn’t evidence to say otherwise. However, in the sixth inning with a leverage index around 1, it’s not a bad idea to challenge.

#8: 4/21/15 vs. Red Sox, bottom 6, 1 out, down 1. Winning team win probability shift: 5%

We’ll talk about the opportunity costs of challenges following the recap, but this was a good place to challenge, even if Cash wasn’t confident of winning. With a leverage index of 2.13 on the play, this runner being safe instead of out in a one out, one run game would have been huge. The Tampa Bay announcers picked up on a theme of Cash challenges: “This is another one of those where… is there enough there to have it overturned?” The answer to that was no, and the call stood.

#9: 4/22/15 vs. Red Sox, bottom 8, 0 out, up 2. Winning team win probability shift: 2%

This final challenge seems like a good point to start talking about those opportunity costs. The runners were ruled out at both first and third base (there was initially a runner on first that tried to get to third on the groundout). Every manager in the game should challenge this play if they still have one available, mostly due to the fact that it’s the eighth inning and they almost certainly won’t get another play to challenge in the game. Even if both runners were clearly out (which was slightly unclear from the replays), it would probably still be a good time to challenge, because you gain nothing from holding onto your challenge after the game. The opportunity cost for a challenge late in the eighth inning is very, very low.

This is also where the dominant unwritten guidelines for manager challenges come into play. Here’s a strange thought: Kevin Cash might actually be utilizing the challenge system better than other managers, simply because he’s not afraid to be wrong. If we grade Cash’s challenges by Jesse Wolfersberger’s method (leverage, which inning they were initiated, and possibility of winning), most of these plays are certainly worthy of a challenge. That they weren’t overturned is unrelated to whether they should have challenged, and we can see that at least one of them definitely should have gone Cash’s way. The issue isn’t that Cash has challenged often and lost; the issue is that other managers probably don’t challenge often enough.

So, does it matter that Cash is 0-9? Considering he probably shouldn’t be, and considering challenges are only a small part of a manager’s job, it matters very little. The issue for Cash is the public perception of him being wrong on the field so often, something that can have a way of growing against a rookie manager. Could Cash do a better job at having a bench coach look at video before he challenges? Probably, but that’s also in line with the dominant thought process — that challenges should only be made on plays a team thinks they have a very good chance of winning. As it turns out, the trigger for challenging should probably be much lower than it is across the league. To his credit, Cash seems to get that; but he still has an 0-9 next to his name, which has its drawbacks.