Figuring out how well a team plays defense is a bit tricky. If we look at goals against, we end up crediting team defense when that number is due in part to goalie talent. Tuukka Rask makes Boston look better at defense than they actually are, and similarly, Ondrej Pavelec makes Winnipeg look worse. Additionally, we know that goaltender performance (and hence, goals against) can vary widely even over large samples. If we want to accurately measure team defense, we shouldn't use metrics that are vulnerable to so much variability.

We avoid these pitfalls by looking at shot attempts against rather than goals to evaluate a team's defense. Teams which give up fewer shots against than others are presumably better defensive units since they're limiting the scoring opportunities for the opposition. We see more consistency in these metrics as well, so they avoid the massive swings from trough to peak that you see in goal-based numbers. How this relates to the Penguins and the Kings and playoff defense is explained below.

The Backdrop

Some people object to using shots against as a measure of defense since those numbers don't account for shot quality. There are two things that need to be cleared up though before going any further. First, there is a difference between talking about shot quality as an individual talent and shot quality as a team talent. We have evidence that there are a few skaters who possess elite shooting talent, and in a sense have a "shot quality" edge to their game. Ilya Kovalchuk and James Neal are two of these guys. Technically speaking, we say this because we have models that can calculate the league average shooting percentage for a given shot after controlling for shot distance, shot type, whether it was a rebound, and a number of other factors. You'd look at all of Neal's shots, for example, and come up with an expected shooting percentage based on the characteristics of each shot. If his shooting percentage is constantly higher than what the model predicts, that would be evidence of a "shot quality" advantage at an individual level.

But just because there's shooting talent at the individual level doesn't mean that it exists at the team level. And this dovetails into the second point: no one disputes that a breakaway is a more dangerous shot than an unscreened wrister from the blueline. But in today's NHL, we don't see teams getting 25 breakaways a game while also limiting their opponent exclusively to weak shots from bad angles. If there isn't a meaningful difference, in the aggregate, between how often a team can generate dangerous shots, then we want to focus on the number of shots given up, not their quality.

There is mountains of evidence to support these propositions. On-ice shooting and save percentage heavily regress to 100.0, highlighting that teams can't maintain a consistent advantage in obtaining or preventing higher quality shots on average. There are possibly one or two teams that maintain a slight edge, but it's not significant over the long-term. This is largely why stat guys conflate PDO with luck. And we've known this for some time--early work by JLikens and Gabriel Desjardins confirmed that shot quality at the team level, assuming it even existed, was minimally important.

And at the individual level, we don't see any significant ability to actually diminish an opponent's shot quality on a regular basis. If teams were significantly affecting the shot quality of their opponents, we'd see tons of players with repeatedly higher/lower on-ice save percentages than average. But in this seminal work by Eric Tulsky, he highlighted how individual players have basically no control over their year-to-year on-ice save percentage. Here's the key portion, in my opinion, from that piece:

So while there may be a sliver of repeatable talent for defensemen preventing the opponents from getting high-percentage shots, after three years of data we aren't even close to being able to reliably tell who's good at it. And like with forwards, making judgments about defensemen based on the number of goals scored against them is a mistake that ends up crediting or blaming the player for factors entirely out of their control.

These results have been confirmed elsewhere.

Scoring Chances in the Playoffs

I've been counting scoring chances in the playoffs this summer to better understand what happened to the Penguins the last two years. As expected, the team scoring-chance numbers hewed very closely to their FF% and CF% in each series, illustrating that scoring chances weren't adding much to the discussion. This is true across the league as well.

Nevertheless, I thought it would be interesting to compare the Penguins' scoring-chance numbers in the playoffs the last two years to the Los Angeles Kings' scoring-chance numbers from this year. The Kings are an indisputably great team, which is evident from their overall possession numbers and their Stanley Cup this year. Additionally, most consider them to be excellent at defense. They set the bar high, but I wanted to see how the Penguins stacked up against them.

There's a guy on twitter named Nick who is an avid Kings fan. He's also a generally awesome hockey fan, and I highly recommend that you give him a follow. He sometimes even posts pictures of adorable baby ducks.

Anyways, he tracked scoring chances in the Kings games during the playoffs this year using the same parameters that I did (shots on goal and missed shots from the home-plate area in front of the net). He sent me the scoring chance data for the Kings, which I'm very grateful for since this project wouldn't have been possible without that. I looked only at even strength scoring chances against (to measure defense) and calculated the number of scoring chances against per game for the Penguins and the Kings. Please click all images to enlarge them.

It's a bit crazy to me that the Penguins were giving up less scoring chances per game in the playoffs the last two years than the Kings were en route to the Cup this year. The idea that the Penguins were better at preventing dangerous shots than the Kings would probably strike many as nonsense, but that's what we have.

To avoid extra time in the playoffs skewing the results, I also calculated the number of even strength scoring chances against per 60 minutes of play for the Penguins and the Kings.

That doesn't change anything--the Kings still gave up more scoring chances against per 60 minutes of even strength play.

I also thought that there might be some difference in how often the Penguins or the Kings give up scoring chances relative to their total number of fenwick events against (shots on goal + missed shots). So I simply divided the total number of even strength scoring chances against in each series by the total number of even strength fenwick events for that series.

Both the Penguins and the Kings were tightly clustered together. Between 29% and 34% of the total number of fenwick events against in each series ended up as scoring chances for the opposition.

I imagine that some folks might object that the Penguins had easier playoff opponents the last two years than the Kings, which would, as a result, skew the numbers. That's fair. But if you just look at how LA did against their weaker opponents (Anaheim and New York), they still didn't limit scoring chances against as well as the Penguins did.

There could very well be some minor discrepancy in how Nick and I track these events. He might record something as a scoring chance that I wouldn't, and vice versa. But I doubt that would change much. Even if we generously add ten scoring chances against to the Penguins in the Boston series, they still only gave up 12.3/game, which is close to what the Kings gave up during their Cup run this year. And if we add fifteen chances to the NYR series, the Penguins are still giving up only 10.6/game, less than what the Kings gave up to the Rangers this year.

This sample is too small for anyone to confidently say that the Penguins are better at defense than the Kings. When you look at overall shot totals (which I think is advisable), the Kings have been better than the Penguins at limiting shots against over the last five years. But the difference isn't that much. And coupled with the scoring chance data above, the Penguins and the Kings probably give up dangerous shots at roughly the same rate.

Team Defense Under Bylsma

We saw above that it's best to evaluate defense in terms of shot attempts against. As such, the chart below contains the Penguins' regular season numbers for 5v5 shot attempts against under Bylsma, with their corresponding rank among the 30 teams (1st being the best). The data is from hockey analysis.

SA/60 FA/60 CA/60 2009-2010 28.0 (6th) 40.0 (11th) 54 (15th) 2010-2011 28.0 (9th) 38.6 (7th) 51.8 (5th) 2011-2012 26.9 (5th) 37.3 (4th) 50.2 (6th) 2012-2013 29.3 (16th) 40.6 (17th) 56.2 (20th) 2013-2014 27.4 (7th) 39.4 (11th) 54.0 (13th)

This is quite good. Their 5v5 shots against/60 were in the top 10 each year other than 2012-2013. As for Fenwick and Corsi numbers, the Penguins only dipped below 15th in the league once in the last five years. I'm personally most impressed with their numbers from this year; to have as many injuries as the Penguins did, especially to their defensemen, and come out with those defensive numbers is a testament to the entire coaching staff. (As an aside, the Penguins 5v5 close shots against/60 were very similar to the numbers above).

I also wanted to see if this held true for the playoffs the last four years (Hockey Analysis doesn't have playoff numbers, and Extra Skater doesn't have data for 09-10 yet). The Penguins' 5v5 playoff numbers are below with their corresponding rank among the 16 playoff teams (1st being the best).

SA/60 FA/60 CA/60 2010-2011 25.1 (1st) 36.6 (2nd) 48.2 (1st) 2011-2012 23.3 (2nd) 32.5 (2nd) 46.5 (4th) 2012-2013 30.9 (11th) 42.3 (9th) 58.6 (9th) 2013-2014 26.3 (3rd) 36.8 (4th) 49.4 (3rd)

This was surprising. I wasn't aware the Penguins had been elite at suppressing shots against in three of the last four playoffs. What's ironic is that they advanced farther in the Eastern Conference in their worst defensive year (2013), which I think is (not surprisingly) due to Fleury's benching in the first round of the playoffs. The fact that the Penguins have gotten awful goaltending in the postseason shouldn't be news to anyone.

***

To avoid any confusion, I'll reiterate the key claims above based on current research. We have compelling evidence that in evaluating team defense, we should focus almost exclusively on shots rather than goals. While team effects might exist, any effect they have is small and dwarfed by randomness. To avoid complex models and highly technical calculations, it's much easier (and correct) to focus on shot totals.

During Bylsma's tenure with the Penguins, they were almost always a good to very good defensive team in this regard. And in the playoffs, they put up some truly elite numbers in three of the last four years. Their defensive competence can be seen by breaking down the scoring chance data for the LA Kings and comparing that to the Penguins.

I'm not arguing that the Pens play defense as well as the Kings do; the scoring chance samples above are too small to support such a claim. But that data is more evidence that the Penguins and the Kings are much closer in terms of defensive competence than many (including mainstream media in Pittsburgh) like to give the Penguins credit for.