SAP and the NHL entered into a massive joint partnership in Q1 2015. The plan was for SAP to ‘revolutionize the hockey stats world’. Let’s check in with how they have done so far:

1. NHL’s stats database currently suggests that seventeen regular goaltenders have not given up a goal on the penalty kill (This is because they have inverted special teams save percentage.)



2. NHL’s stats database currently features completely inaccurate/randomly generated numbers for team-level shot statistics. They have been compared to multiple independent resources (all of which are agreeable on what the numbers should be). Not in the same universe.

3. NHL’s stats database has either inverted the faceoff count or is calculating zone starts through, again, randomly generated numbers. (Also true for certain players at the individual level.)



4. SAP has convinced the NHL that they can deliver results in less than zero seconds:

5. It is believed that SAP is going to (inexplicably) need three years time to digitize the NHL’s old stat sheets.

6. One of their biggest ‘rollouts’ to date was automated data viz, which includes user-friendly graphs like the following:

7a. The capturing and incorporation of “clutch factor” measurements, paragraph below, which almost certainly do not exist (but certainly sound nice!)

7b. As an extension of that, the belief that faceoffs are so crucial in the determination of outcomes despite all evidence to the contrary.

8. The usage of ‘close’ statistics (as a way to mitigate score effects) despite them being phased out years ago in favor of score-adjusted metrics.

9. The creation of a ‘milestone tracker’ to capture huge accomplishments for active players such as … Chris Pronger.

10. “Deep statistical comparisons” which include little more than basic counting statistics that have existed for decades.

11. The searching of a database coordinator with proficiency in … Powerpoint.

12. A relationship grounded in the belief that SAP would deliver ‘never before seen ideas’, which includes re-hashing of years old data that lives (accurately!) on no less than a dozen sites across the internet. As an aside, it’s somewhat amazing the NHL had so much tunnel vision over the last decade that they genuinely believe some of the furnished metrics are property of SAP.

13. Speaking of those never before seen ideas, check out these unprecedented (and stolen) shift charts!

14. Randomly generated CorsiClose% numbers. CorsiClose% is a defunct metric and has been for quite some time. Even if CorsiClose% wasn’t defunct, the numbers are still wrong:





15. Their belief that what the market wants is ‘black box power rankings’, which was a key component of their ‘Phase Three’ rollout:

16. Totally inaccurate or randomly generated goaltending numbers with no source data from an old presentation:







17a. A comical, comical belief that they had built a predictive model that could accurately select 85% of post-season winners. A reliable source indicated that this is one area the NHL pushed back on in disbelief. SAP eventually purged any mention of it from their publications without clarification as to whhy.

17b. (We checked the model anyway. I stopped caring once they couldn’t clear 50%.)

18. Has their snake oil sales act and falsified data had a negative impact? Well, hockey media still cite their numbers, despite the fact that (a) their database is inundated with inaccuracies; and (b) their business model seems to be little more than pulling doing a dance and pulling wool over the eyes of Big NHL. The dangers of being perceived as credible despite no supporting evidence, I suspect.

19a. What do NHL employees think about the product? Here’s Andrew Thomas, consultant to the Minnesota Wild: “ ..What burns me the most isn’t at all "competition”: it’s extremely lazy work masquerading as professional innovation.”

19b. A league employee: “Up there with the first Lindros trade and the 1988 game with linesmen in yellow practice jerseys, only worse because the league would have us believe it is now competent.”

19c. Another league employee: “Outrageously bad.”

20. At the time of the first (and second) rollouts, NHL.com failed to separate game states from one another. I say this without hesitation: the separation of hockey statistics by game state is the FIRST STEP in hockey analysis.

In summary: there is no reason to go to NHL.com for anything related to hockey statistics. Their numbers are inaccurate. If not inaccurate, they are misleading. If not inaccurate or misleading, they aren’t capturing what they believe they are capturing. If none of the above, the site is a total hassle to maneuver through and the filtering/visualization looks like a child slapped the cursor on MS Paint a hundred times.

The NHL, at an early point in their business relationship with SAP, considered a liaison or oversight position to ensure that SAP’s deliverables were accurate. That never materialized. SAP’s ran roughshod since then. Many of the above numbers are data points pulled from months ago. These [inaccurate] numbers remain on the official web site of the NHL. There’s no quality control, no reliability, no incentive for SAP to satisfy their half of the partnership.

Why? Because at some point in time, the NHL realized that pushing the stats responsibilities out to a third-party meant a lot of man hours and headaches saved. They took no interest in vetting the information. And they ate up a ridiculous SAP song-and-dance about a litany of never-seen-before statistics that have existed in the foremost corners of the internet for close to a decade.

The NHL should terminate their partnership and find someone who actually cares about the work they are doing. And the SAP should find more big business to sink their teeth into before the well dries up. Monorail sales have never been better.

Lastly: if you ever want to conduct stats research, the following resources are worth your time.

Behindthenet.ca

War-on-ice.com

Hockeyanalysis.com

Nicetimeonice.com

HockeyStats.Ca

TSN.Ca (Naturally.)