Rating Hockey Players: Hits and other Statistical Jargon

So I decided to start blogging on a whim this weekend. Partially because I didn’t want anyone to think I’m the michaelbarba.blogspot.com that offered unthoughtful opinions in 2009 on legalizing weed as a solution to boost economic growth, and partially because I had opinions beyond 140 characters on things I read on the internet where the things I’m interested in – such as hockey, economics, statistics, technology and coffee – intersect.

There was an SAP-sponsored paper that was presented at the MIT Sloan Sports Analytics Conference in Boston last weekend that caught my attention titled “Total Hockey Rating (THoR): A comprehensive statisticalrating of National Hockey League forwards and defensemen based upon all on-iceevents”.  What the paper tries to present is a, “reliable methodology that can quantify the impact of players in creating and preventing goals for both forwards and defenseman” (Shuckers and Callo 1, 2013). After that, the paper identified nine on-ice action events to quantify THoR, including shots, turnovers and hits. The paper smartly isolated biases for several different considerations – how takeaways / giveaways are calculated (with home scorekeepers tending to favor the former statistic), offensive vs. defensive zone starts, talent level of goalies and other players on the ice, and shot location selection (providing higher weights to higher percentage shots). Overall, it’s an extremely well orchestrated study, but there are a few major criticisms I have for THoR, in addition to the grammatical mistake from the paper’s abstract I quoted above (defensemen should be plural). Chiefly: the inclusion of hits in THoR’s methodology.

The original purpose of measuring hits as a statistic was to treat it as a turnover metric – for example, if Shea Weber knocks Patrick Kane off the puck and Roman Josi gains possession as a result, Weber is rewarded with a hit. However, the stat has become a subjective tool that the scorekeepers can reward regardless of outcome. For example, say Kane and Patrick Sharp are on a 2 on 1 vs. Weber and Weber hits Kane right after Kane sets up Sharp for a goal. The scorekeeper can award Sharp the goal, Kane the assist and Weber the hit.

The statistic is further diluted by home scorekeeping bias. Consider this random Red Wings – Blue Jackets game from February 2012 in Columbus. Although the Blue Jackets lost 5-2, they managed to “out hit” the Red Wings 33-2. Take a look at team hits for the 2011-12 season; the Phoenix Coyotes were fifth in the league at home in hits, but only ninetieth on the road. Either the Coyotes are extra fired up to play to arenas where announced attendance was 72.5% last season, or the (more obvious answer) the scorekeepers are biased towards the home team.

Biased home scorekeeping can extend to other stats beyond hits as well that the authors did not account for. From the Sabres - Rangers game last weekend in MSG, Drew Stafford scored a rare goal off directly off a Mikhail Grigorenko faceoff that the Rangers “won”. The book Scorecasting finds that the “home court” advantage a team maintains is reflective of a) the away team’s travel schedule (in the NBA, MLB and NHL) and b) subjective decisions made by referees. In theory, the ability to draw penalties and get on the power play favors the home team as a result of the referee bias. The referee bias can also extend to non calls favoring the home team, a la this Matt Duchene goal against the Predators last month that was just a bit offside (Side note: in fairness, this memorable non call from last rewarded the away team).

In a more extreme case of biased home scorekeeping, Jeff Marek from Sportsnet in Canada recounted a story on the Marek vs. Wyshynski podcast a few weeks ago from the 1997-98 season when Glen Sather was trying to trade defenseman Dan McGillis. To boost McGillis’ trade value, Sather had the Oilers scorekeeper tally extra hits for McGillis to make potential suitors think he was a hitting machine – at the trade deadline, the Philadelphia Flyers acquired McGillis and a second round pick from the Oilers for Janne Niinimaa, who gave the Oilers five productive seasons and an All-Star game appearance in 2001. (Side note: In fairness, while McGillis was not the hitting machine the Flyers thought they had acquired on their blue line, McGillis did give the Flyers five quality seasons and won the Barry Ashbee Trophy awarded to Philadelphia’s top defenseman in 2001).

Hits’ lack of meaningfulness is not necessarily the study’s fault, but including it in THoR considerably weakens its usefulness.  However, what is not accounted for in THoR’s methodology hurts the study just as much as including hits. Part of this is beyond the study’s control – x y coordinates for where every single player is on the ice during a goal, which undoubtedly affects the probability a shot will go in, are not recorded. But there are a couple of measurable oversights that weaken THoR – dummy variables for power plays, and who is coaching the team.

Including dummy variables for power play situations (whether it be 5 on 4, 5 on 3, or 4 on 3) is a bit of an academic point. The study factors in time spent on the power play and multiplies it against the league average; but as of this writing, the Anaheim Ducks are more than twice as likely to score with a man advantage than the Buffalo Sabres are (29.2% vs. 12.2%). Because power play situations are mutually exclusive from regular situations, the effect can be easily isolated with a dummy variable.

Though no mention of it is made in the study, it should be notable that Alexander Ovechkin (who scored 70 goals over the course of the two observed seasons) is completely missing from the study’s Top 50 players list (just as it should notable that Tyler Kennedy is Number 3 behind Alexander Steen and Pavel Daystuk). But wouldn’t the affect of the Capitals firing their run and gun head coach Bruce Boudreau midway through the 2011-12 season in favor of the defensive minded Dale Hunter have a negative effect on Ovechkin and other Washington Capitals that didn’t make the list such as Niklas Backstrom, Mike Green and Brooks Laich? (Side note: Alexander Semin was the only Capital to make the list at #49, and is evidently not too missed in the Washington locker room these days). It is notable THoR treats players as separate if they switched teams over the course of the study; this logic should additionally apply to when their coach is switched.  

As well, measuring players by purely goals for and goals against is an idea that is a sharp departure from what Corsi (arguably the most popular advanced hockey metric) chiefly accounts for to rank individual players – shot attempt differential (which is also utilized as a proxy for puck control).  Of course, there would be little to no difference in findings between THoR and Corsi if they both measured the same metric. But by measuring players on creating and preventing goals alone, you can’t control for luck’s contribution as well as you can with shot differential, chiefly because goals are not independent of the goalie. Accounting for whether Jonathan Quick (who led all goalies with 40 or more starts last season with a 1.95 GAA) or Vesa Toskala (who is Vesa Toskala) is in net will have a difference on goal probability. But Quick still let in 133 goals last year; without insanely detailed information, you would have to go back and look at all 133 goals scored on Quick to determine if Player X created the goal, or scored as a result of luck (such as a good rebound, a defensive letdown or poor positioning on the goalie's part). 

Ultimately, any methodology will have its shortcomings. At the very least, this paper has presented an alternate way to think about what makes up a top NHL player, and is a good first pass of developing a “reliable methodology” it wishes to develop. But unless you’re Tyler Kennedy or his agent, I imagine this study will have little more influence on shaping any business decisions in the NHL as this blog will.