Twenty-four percent of all NFL games are decided by three-points or less. If that happens this weekend at the 51st Super Bowl, all the glory (or the blame) will fall on Matt Bryant (placekicker, Atlanta Falcons) or Stephen Gostkowski (placekicker, New England Patriots). It seems reasonable to give them the credit, but in this case reason has it wrong. Giving Bryan or Gostkowski the MVP for making a crucial kick is like giving a gambler credit for the roulette wheel landing on red.

In American football the team is generally a single unit, but the kicker is a unique position. Quarterbacks are the de facto leaders of the team, but a quarterback is only as good as his offensive line, receivers, and running backs. Unlike baseball or even basketball, measuring the performance of an individual player in football is notoriously difficult. Unless that player is the kicker. In that case, it’s easy.

Football placekicking is obviously a high-skilled occupation. But is luck involved too?

A kicker does need teammates — a “long snapper” and a “holder” to get the ball in place and an offensive line to guard him from being blocked — but generally his job is solitary. He is given the well defined, self-contained task of kicking the ball from a set distance through the goal posts. The difficulty of his task is easy to measure: How far from the goal posts is he required to kick? And his success or failure is equally easy to measure: Did the ball go through the goal posts or not?

Football placekicking is obviously a high-skilled occupation. But is luck involved too?

The Falcons’ Bryant has successfully kicked 91.5% of his field goal attempts this season. The Patriots’ Gostkowski has only an 84.4% average this season, but has a better lifetime success rate than Bryant. This is Gostkowski’s first season since 2011 in which he did not lead the league kickers in scoring. On the surface, it looks like Gostkowski has the long-term edge — but Bryant is on a hot streak and might be the one to take the crown in the 2017 Super Bowl.

We like to tell underdog/reigning champion stories, but what if those stories distract us from what’s really going on? What if there was no actual, measurable difference between Bryant and Gostkowski? What if I told you we could replace either of them with Roberto Aguayo of the Tampa Bay Buccaneers, who, at 71% success rate, has the worst performance in the league, and expect him to do equally well?

What if who’s kicking on Sunday doesn’t matter at all?

THE HIGHEST PAID PERSON ON THE FIELD

Kickers, who earn $500,000 to $4 million per year, do not command the salaries of quarterbacks.

However, when you calculate pay in terms of amount earned per play (per “snap”), eight of the top 10 highest paid players in 2013 were kickers. Using “per-snap” metrics, the highest paid person in the league was Adam Vinatieri, kicker for the Indianapolis Colts, who earned a whopping $47,000 per kick. The highest non-kicker was Eli Manning at $21,000 per snap. Vinatieri’s coach, Tony Dungy, was known to mouth “money, money, money” when Vinatieri went out for an important kick.

No pressure, right?

Case in point: October 27, 2013, was a bad day for Shaun Suisham, then the starting kicker for the Pittsburgh Steelers. His team had just lost to the Oakland Raiders 21–18, and Suisham felt directly responsible. He had missed two field goals, which, had he made, would have given the Steelers a three-point victory.

Afterward, it was no wonder Suisham felt terrible. Other factors could have helped his team win, but nothing was as obvious as his two missed field goals. And any other errors could be shared by a lineup of guys. The missed field goals were all on Suisham’s shoulders.

In his post-game interview, Suisham looked dejected. He could barely make eye contact with the reporters and by the end he had a hard time getting words out clearly.

Suisham: “We would have won the game if I made the field goals. Tough one to swallow. No choice but to do it. I should have been better today and we lost because of it.”

Reporter: “What would you say it was?”

Suisham: “I just missed.”

Reporter: “How tough is this loss to take?”

Suisham: “We would have won the game today if I was better. You know it’s. Ah. It’s hard to take. But. You know. Swallow it. Get ready for next week.”

Imagine if you were getting paid $40,000 to do one thing and you very clearly failed at that one thing and had no one else to blame.

Watching the interview, it’s impossible not to feel empathy for Shaun. He’s a young man under an unimaginable amount of pressure and he messed up in front of a national audience that blames him (as he blames himself) for his team’s loss. Imagine if you were getting paid $40,000 to do one thing and you very clearly failed at that one thing and had no one else to blame. It was 100% your fault. Now imagine failing like that twice in a row in a way that costs your employer 10% of its annual goal, since it takes about 10 wins in a season for an NFL team to make it to the playoffs. From that perspective, Suisham took it about as well as could be expected.

Unlike playing the lottery, kicking is clearly a skill. There’s no doubt that some people — through a combination of inborn talent, temperament, and training — are better at kicking than others. If I was a kicker in the NFL, I would be the worst kicker the league had ever seen. There would be no way for me to “get lucky.” So when Suisham failed publicly in front of millions of people — after the long snapper and holder did their jobs right and set him up to succeed — he couldn’t chalk it up to luck or chance. Or could he?

WHAT IS LUCK IN A GAME OF SKILL?

How do we know how much of an activity or competition is luck versus skill? Sometimes it’s obvious: If we played a game where we each rolled dice and whoever rolled higher won, it is pretty clearly 100% luck. Perfect skill activities are harder to identify. In chess, success is close to 100% skill. All of the information is fully visible and, in theory, players can plan ahead for every possible future contingency with complete ability to make choices without outside complications. But in practice, chess players are making decisions with imperfect information. Two options may look equally attractive but a choice must be made. When a player gives up calculating and “just chooses” one of those two options, how else do you define the two different possible future results if not “luck”?

Most activities fall somewhere between high-skill chess and random-luck dice games. If it is played perfectly, blackjack is a game of luck — but the ability to play it perfectly is a skill. Poker is far more skill-based, but still involves a lot of luck in any individual hand. The hand game rock-paper-scissors seems to be pure chance — but championship RPS players (yes, there’s a world championship rock-paper-scissors contest) say there’s skill in using psychology to predict an opponent’s next choice.

It turns out, there is an objective way to measure the relative amount of luck in, say, poker versus rock-paper-scissors.

If a history of success predicts future success (or a history of failure predicts future failure) then we can confidently say skill was involved. If the past does not predict the future, we can be comfortable calling the results “luck.”

In a game of pure luck, one should expect no predictability in future performance based on previous performance. If I win four dice games in a row, I am no more likely to win the next round than you are (nor less likely — believing “luck will come around” is known as the Gambler’s Fallacy). If I beat you four times in a row in head-to-head poker tournaments, I am not guaranteed to win the next one, but an outside observer would be crazy not to give me more than a 50% chance.

This ability to predict future performance based on historical results is the key to measuring skill versus luck. If a history of success predicts future success (or a history of failure predicts future failure) then we can confidently say skill was involved. If the past does not predict the future, we can be comfortable calling the results “luck.” The better the future prediction the more skill we can claim.

HOW PAST DATA HELPS PREDICT THE FUTURE

Good data analysis is using data about the past to predict the future. If you do it right, you will have unique insight into what is going to happen: How many sales your company will make tomorrow; the chances your favorite sports team will win the championship; how much it will rain in NYC in July. If you do it wrong, your analysis may “predict” the past, but have no use in telling you what will happen tomorrow. But sometimes the best you can do is know that you can’t do it well enough to predict anything.

Breaking out skill from luck gets complicated when there are two people of similar skill competing against each other. If I were to play tennis against Roger Federer I would lose every game. One could say the results of Federer vs. Nevraumont matches were perfectly predictable and therefore the game is 100% skill. When Federer plays Andy Murray he will usually win, but sometimes (like in the 2012 Olympics) he will lose. That doesn’t mean the game isn’t highly skilled. Instead of looking at single-player matchup, we can look at all the games from all the players and use the data to predict the likelihood that a given player will win a given match against a given opponent based on skill ratings we give them modeled on their historical performance. It turns out that when you do this, you can predict tennis matches with a degree of accuracy when player ratings are far apart — but much less so when ratings are close. One could say that tennis is a very high-skill game, but when two players are evenly matched it comes down to luck.

How does this help us with Shaun Suisham’s kicking in the NFL?

It is relatively easy to model kicking performance. We can look at each kicker and know whether they succeeded or failed on each kick attempt. And we know the level of difficulty based on how many yards he had to kick. The NFL keeps detailed records of every play, so we can go back to see kicker performance over time.

The first thing you’ll see is that kickers have been getting better. The website FiveThirtyEight ran the numbers to look at field goal success rate in the NFL by year going back to 1960.

Graph by Reuben Fischer-Baum via FiveThirtyEight.com

There are random fluctuations from year to year, but the overall trend is clear: Kickers in the NFL are much better today than they were a decade ago at every distance. This strongly backs up our assumption that kicking is a high-skill activity. If kickers were rolling dice to find out if the ball would go through the uprights, we would not see a consistent year-over-year improvement.

But this doesn’t dig into individual performance. For that we need to look at how well Suisham’s 2008 performance predicts his 2009 performance. More accurately, we should look at all the kickers in 2008 and see how correlated their field goal success rate was to that of the following year. To get an even better result, we could look at all the kickers from, say, 2008 to 2015, and compare their first appearance in that time period to their second appearance.

If we see a high correlation, we can assume the job has a high skill component and that we can be confident at predicting next year’s performance based on how they did in the past. If the correlation is low, that doesn’t mean the job is not skilled, but that when one gets to the level of the NFL, the skill difference between kickers is very small — like the tennis players who compete against one another.

The higher the correlation, the higher the relative skill gap between the individual kickers. The lower the correlation, the more we cannot differentiate which kicker is actually better than any other.

So what do we find?