The Ten Laws of Hockey Analytics

[This is an update of a piece written in 2008. The many updates show how much knowledge has changed since then].

Law #1: Winning is what matters

It may seem self evident that winning is all that matters, but a surprising number of analyses are not grounded in this law.

When measuring something, one should know the relationship of that something to winning in order to assess its usefulness. There is a very large amount of statistical futility out there. The other nine laws are designed to coach you towards useful information.

Most feel that winning in the playoffs is what really matters, that regular season victories don’t matter much. But with 30 teams in the NHL the planets must be exceptionally well aligned for any given team to win the Stanley Cup (see Laws #3 and #5) and the single best predictor of playoff success is regular season performance (even then, when a 110 point team faces a 105 point team in a playoff series the probability of an ‘upset’ exceeds 40%).

Let’s call this the Jacques Plante Law of Winning. Plante wore #1 and he won: six Stanley Cups and a regular season record of 314 wins, 133 losses and 107 ties.

Law #2: Goals for and against are the only factors that affect winning

In any sporting event, one wins by having more credits than debits. The accounting system of some sports is complex. In hockey it is not. Hockey teams win (collect points in the standings) by scoring more goals than they allow (including in the shootout).

In a single game it is an absolute truth that the team that scores the most goals is the winner. However over the course of a season a team with a positive average goal differential only has a tendency to win more than it loses (a consequence of Law #3).

Nevertheless the predictive power of goals for (GF) and goals against (GA) is very strong. Regression analysis tells us that about 94% of winning is explained by a sophisticated model involving just GF and GA. The remaining 6% seems to just be statistical noise. All other variables simply drive GF or GA.

The sophisticated model: Winning Percentage = GF n / (GF n + GA n),

where n depends on some things (but you can just assume n=2 and you are nearly there).

Taken together, Laws #1 and #2 tell us that ‘goals are what matter’.

This is the Doug Harvey Law of Goals. He knew how to create and prevent goals, and he wore #2.

Law #3: Goals are random events

It may seem that focusing on goals violates Law #1. If a team wins a lot of games in spite of a small average goal differential, we seem to be getting mixed signals. But this is where Laws #3 comes in.

The game of hockey is played by humans at high speed in tight quarters on a slippery, but imperfect, surface with a disk made of rubber. If this does not sound like a recipe for chaotic events, watch a hockey game.

There is a great deal of statistical proof that goals occur randomly. This does not mean that skill, strategy and execution give way to luck. It means that outcomes are uncertain, influenced by a myriad of factors including skill, strategy and execution.

In fact, that is why the games are played. If variations in individual performance and conditions (‘randomness’) did not exist, the game would have no interest.

Most people struggle to accept this. Human brains are hard wired to find patterns. We like good stories and want to believe in cause and effect. Yet there are too many chaotic factors in play. Randomness is everywhere in the game of hockey.

I will call this the Harry Howell Law of Random Events. He wore #3 and saw a lot of random events in a very long career with generally bad hockey teams.

Law #4: Winning has a nearly linear relationship to goal differential

Below is a plot of predicted winning percentages (based on goal differentials – goals for minus goals against) versus actual winning percentages of NHL teams since 1946:

If goal differentials perfectly predicted winning, all of the data (the blue dots) would have fallen on the red line. In fact the data is tightly clustered around the red line. This gives visual evidence of the very high correlation between goal differentials and winning.

This data does have a bit of an “S” shape to it. Extremely good teams tend to underperform a prediction based on goal differential and extremely bad teams tend to outperform the prediction. While the linearity breaks down for extreme teams, the preponderance of the data comes from teams with winning percentages between .300 and .700. Over that range, the linear relationship between goal differential and winning is very strong.

The simple model: Winning Percentage = .500+ (GF – GA) / (g x GP),

where g is the average number of total goals per game and GP is games played.

Goal differential by itself explains about 93% of winning. We get only about 1% of extra information from the more sophisticated (non-linear) model described in Law #2 (because it captures the “S” shape of the data).

Why does this matter? This ‘linearity law’ is a huge building block for hockey analysis. It means that goals saved and goals scored have the same kind of impact. Linearity means that the game is essentially the sum of its parts. It means that individual performances are basically additive. It means that team performances can be decomposed into individual contributions.

Laws #2, #3 and #4 are like the fundamental forces of the universe. But they beg the question: how are goals created (or prevented)? This turns out to be the really big question. The answer is much less clear, but the remaining laws help us to understand this better.

We have to name this the Bobby Orr Law of Linearity.

Law #5: Sample size matters

An NHL team participates in 82 games. Won / loss records contain the same amount of statistical information as 82 coin tosses. This means, for instance that if a team looks average (wins 41 games) we have 95% confidence that the team is neither better than .600 or worse than .400. Yet a true .600 team is a Stanley Cup contender and a true .400 team is a candidate for the top draft pick.

An average, modern NHL team participates in about 450 goal events, 5000 shot-on-goal (‘SOG’) events and 9000 shot-at-goal (‘SAG’) events. The statistical information from a sample is proportionate to the square root of the sample size. If there is 1 unit of information in a game outcome, then there are 2.3 units of information in goal events, 7.8 units in SOG events and 10.6 units in SAG events. Expressed differently, there may (see Law #7) be as much statistical information in 8 games of SAG data as there is in a full season of wins/losses or 35 games of goal data.

This is also important at the individual player level. An elite goaltender might face about 2,000 shots per season. An elite shooter might take just 300 shots. So we generally know more about goaltending talent than shooting talent.

But samples are just that. They are shreds of evidence of true capability. In many cases the evidence is slim. It takes several years to narrow the confidence interval for the true talent of a starting goaltender to the point that we can declare him to be something other than average. Even then we have the statistical problem of ‘trend’ – by the time we are confident of ‘true talent’, it may have moved on us.

We will call this the Nick Lidstrom Law of Large Numbers. His sample size is very large. We know he was very good.

Law #6: Expect mean reversion

This is really a corollary of Law #5, but it is so important that it has its own place in the pantheon of the laws of hockey analytics. You should always expect mean reversion.

If there is one stat that you really should try to appreciate it is “PDO” – the sum of shooting and save percentages for a team (or for a given player while on the ice). Research has shown that a PDO much greater than 1.000 is an indication of good luck and a PDO much less than 1.000 is an indication of bad luck. Both will have a strong tendency to regress to the mean. PDO outliers make for the easiest predictions in hockey.

At the team level it is pretty easy to accept mean reversion in shooting percentages. Talent and circumstances are averaged very quickly (note, however, Law #10). Individuals, on the other hand, have their performances revert to their personal mean and circumstances. Phil Kessel and Colton Orr do not have the same mean. This means that one source of variation in team shooting percentages is the variation in number of shots between talent levels.

Goaltending is a bit different. At the individual goaltender level, sample sizes are much larger and we know more about true talent. At the team level we quickly observe that one player may dominate team save percentages. This means that save percentage differentials are more credible. The caution here is that a 1% difference in save percentages (.925 vs .915) is the difference between very good and just average and that degree of discrimination in the mean requires a great deal of data to confirm.

Finally, we need another reminder about trend. Personal means move over time, sometimes very suddenly. And coaches adapt to both perceived and true trend, trying to ride the hot hand even though they may simply have been playing a lucky hand. Uncertainty in trend is another source of mean reversion.

This is the Toe Blake Law of Mean Reversion. Blake is probably the most famous #6 in NHL history but he is probably more famous as a coach. No doubt, as a coach, in his era, he understood that a hot hand turns cold.

Law #7: Respect the data

Data can only tell you what it knows. Law #7 encourages us to collect the right data, the right way. If we are working with flawed data we should sanitize it properly. We should take care to analyse data effectively. Finally, we always want to work with rich raw data because the summarization of data always destroys information.

The only unambiguous event in a hockey game is a goal – and even then a few are disputed. All other events are a matter of opinion. We need to be constantly vigilant for ‘measurement error’.

Law #3 told us that the game has a great deal of randomness. Law #7 warns us that there is also randomness and bias in the observation of the game. Shot totals (including misses and blocked shots) are affected by the judgement and practices of the official scorers. Shot information frequently contains errors or bias. Other records of the game, such as ice time, may be inaccurate. If it is recorded by humans, it may be wrong.

Law #5 helps us out a bit with this problem. If data contains random errors, the errors tend to average out and become part of the ‘noise’ of the game. A large enough sample size, if it is unbiased, will drown out the measurement noise.

However, if there are systematic biases in the data one needs to find a way to weed it out or correct for it. The most infamous example of bias is in the shot distances recorded at Madison Square Garden. The data says that shots are much shorter at MSG than elsewhere. We know this is wrong and (if you care about shot distance) you need to fix the problem. One solution is to throw the data away. Law #5 says that a better answer is to try to adjust the data for the effects of the bias. This is the more universal approach to data that suffers from context (see Law #10).

There are degrees to be earned in statistical analysis. In short, there are specific tools for each type of analysis. A better designed and controlled experiment yields better information. One important warning – do not confuse correlation with causation. The former is easy to prove, the latter is quite challenging. For example, carry-in zone entries yield more scoring chances than do dump-in zone entries. But this could mean that a carry-in is evidence of better neutral zone puck control rather than a cause of better offensive zone puck control.

Summarization is essential to communications, but it should always be a late step in an analytic process. Consider the humble average: when we average a data series we seek the ‘signal’, but it doing so we lose information about sample size and variation (remember Law #5). This could be a good or bad trade, depending on the circumstances. Example: A common way to summarize offensive or defensive control is by subjectively counting scoring chances. This is suffocating data before it is even allowed to breathe. Don’t do things like that. “Ten scoring chances” has much, much less information in it than the ‘when’, ‘where’, ‘who’, ‘how’ and ‘outcome’ of 30 shots on goal.

Record, adjust, analyse, summarize.

Let’s call this the Tim Horton Law of Data Awareness. Horton wore #7 for 17 years with the Maple Leafs.

Law #8: It’s a team sport

Law #4 says that we can add individual performances to get to team performance. But hockey teams always have players in roles. So it is critical to understand the impacts of context (see Law #10). In this sense, to go back to Law #1, winning is what matters.

Law #8 says that we should not care about who put the puck in the net. We should only care that it went in. This law tells you to think about something like plus/minus. Alex Ovechkin scored 51 goals in 2013-14 but concluded the year -35. Teammate Joel Ward scored 24 times and finished +7. We are conditioned to honour the goal scorers and finishing is a talent … but it is a team sport.

Plus/minus is a flawed statistic for a number of reasons. In particular, Law #5 tells you not to stop there as there is limited information in goal events, especially when restricted to even-handed play as is the case with plus/minus. So Law #8 is not about plus/minus but about seeking out contribution-to-winning (Laws #1 and #2) team measurements. It is about the question mark in Law #4.

Laws #5 and #6 also say that we should be wary of disaggregated individual results. There will be much greater variance in individual performance than in team performance. It is challenging to separate the player from the team (see Law #10).

This is the Mark Recchi Law of the Team. He wore #8 for 5 different teams (and played for two additional teams) in what will be a hall of fame career.

Law #9: Puck possession matters

Law #5 says that we should take our analytic efforts to the largest mountain of data – shot data. Rigorous analysis has shown that historic SAG data predicts future winning better than does historic winning or historic goal scoring. This is almost certainly due larger sample sizes – the signal is more evident through the background noise of randomness – but it is also almost certainly a commentary on the quality of the underlying ‘process’ of goal scoring and prevention.

SAG has also been shown to be highly correlated with, and therefore a proxy for, time of puck possession (something that is not readily available). It is evidence of puck control. So the thesis is that puck control is a material factor in the generation and prevention of goals. Note the flow of logic: SAG is a proxy for puck possession, which is believed to drive goals scoring and prevention, which are know drivers of winning. The evidence is that SAG predicts winning.

What is SAG? Otherwise known as “Corsi”, it is simply a count of the number of times the puck was directed at the net:

SAG = Shots At Goal = Goals + Saves + Missed Shots + Blocked Shots, either For or Against

SAGD = Shots At Goal Differential = SAGF – SAGA

SAG% = Shots At Goal Percentage = SAGF / (SAGF + SAGA)

How do missed and block shots better forecast goals for and against? After all, those shots could never have been goals. The answer is that there is process information present in this larger data set about puck control, and puck control is an observable factor in goal scoring and prevention (just watch some power plays).

When you remove blocked shots from SAG you get “Fenwick” (which I call SAM for Shots And Misses). The hypothesis for the removal of blocked shots is that these events are evidence of lower quality puck possession. The data does not really support the hypothesis but the thought process suggests that a better approach may be to give differential weights to goals, saves, missed shots and blocked shots (Weighted SAG or WSAG).

We know that SAG tells us something. But SAG is still a measurement of the result and only an indirect observation of the process. It is also a reflection of hockey culture and era. As an example, consider the Traditional Russian Style of hockey. This style absolutely emphasized puck possession, but in this era and culture the puck was never to be wasted on a poor scoring chance – the name of the game was high shot quality. SAG would not have been a useful tool

What contributes to puck possession? SAG describes it, today, to a degree. The search is ongoing for better measures of the underlying process.

This is the Gordie Howe Law of Puck Possession. Many great players have worn #9. None wore it better. Nobody else is allowed to wear it in Detroit, the birthplace of modern puck possession thinking.

Law #10: Context matters

Let’s start with ice time. A player with more ice time is being asked to do more. We can infer something from the ice time (the coach’s view of the player). In any case it is necessary to adjust for ice time differentials through the use of rate (per 60 minutes) statistics.

Situation, the kind of ice time, matters. We have completely different expectations of penalty killers and power players. A common approach is to ignore ‘special team’ play and focus on even handed play, but that ignores a good part of the game, throws away information and denies the value of some roles. A better approach is to make situational adjustments to the entirety of the game.

Zone starts matter. This is another piece of information about roles and the kind of ice time. Manny Malhotra’s role was to move the puck up the ice. He got 70% of the defensive faceoffs. The Sedin twins had the task of scoring the goals. They got 70% of the offensive zone faceoffs. Their metrics look very different. A common way to address this is to plot zone starts against SAG metrics. This makes it clearer why certain players SAG the way they do. A better approach is to adjust data for zone starts.

Score effects matter. It is observable that the nature of the game changes based on the score (and time to play). A common approach is to use only close game data. But that too throws away information (remember Law #5). A better approach is to make adjustments for score effects.

Quality of teammates matters. If you cruise alongside Sidney Crosby you are one lucky winger. This naturally encourages you to consider a With-Or-Without-You (WOWY) analysis. For instance you can compare Patrice Bergeron’s SAG% to that of the Bruins when Bergeron is not on the ice. One problem encountered with teammates, however, is the Sedin Syndrome. Players that rarely play apart are like statistical Siamese twins and are difficult to separate.

Quality of competition matters, including strength of schedule. Playing against Crosby is more difficult than playing against Brandon Sutter. Correcting for this is possible. The most sophisticated approach is regression analysis, however simple methods also work well. Sometimes this is less necessary due to Law #5 – competition averages out way better than teammates.

Other stuff matters too. The more we know, the more the details matter.

We have to call this the Dale Hawerchuk Law of Context. Several other great players have worn #10. But it turns out that Hawerchuk has probably advanced our understanding of context more than anyone else.

Share this: Email

Print

Facebook

Twitter

LinkedIn

Reddit

