 |
| Source: Montreal Gazette |
Though the future of the goalie save data that drives this projection system is up in the air due to the War-on-Ice site’s founders joining the Minnesota Wild organization, my hope is that the momentum behind breaking down shot quality into save difficulty will continue in the hockey world, and that the model can be adjusted to work with future advanced save statistics down the road. So without further ado, below is the methodology broken down into four key areas:
Low, Medium and High Danger Zone (DZ) Opportunities
For me, what opened my eyes into improving goaltender evaluation was War-on-Ice pioneering the idea of adjusting goalies save percentages into three separate 'danger zones', based on how league-wide shooting percentages from three different areas of the ice: |
| Source: War-on-Ice |
So when leveraging past performance to project future performance, I wanted to make sure that I was evaluating each individual goaltender’s performance by these three zones, as opposed to overall 5v5 or even Adjusted 5v5 performance, for a couple different reasons. First off, controlling for danger zones (DZ) help to better isolate team effects in front of the goalies, such as focusing on 5v5 helps to not penalize goalies playing on teams that take more penalties than the norm vs. Regular Save %. Second, DZs help to provide another layer of analysis when evaluating differences between goalie projection and which DZ is relatively driving a higher or lower than expected projection. There is additionally room to add in differing levels of regression by neutral zone if necessary (more on that concept below), though for this initial model I have stuck with the historical league averages (which I refer to as save and goal stats from the 2007-16 seasons, i.e., the Behind the Net era).
Past Performance Weighting
Independent of DZ, how to weight past performance when projecting future performance is work that others such as Eric Tulsky have spent much more time researching. This is currently a blind spot for me, as I haven’t dug further into replicating and/or revalidating Tulsky’s work. But for this model, I think that sticking to the 100%/60%/50%/30% system to weight the previous 4 seasons' performance is reasonable enough. Chiefly, it creates a larger body of past work to base future performance off of while incorporating the most recent season’s performance more heavily than from a few years ago.It should also be noted that I have used regular season data only. While I plan on exploring playoff data in future iterations, my concern is that the effect of facing the same team 4-7 times in a row could skew some projections. To what effect, I'm not sure, but for now that data has been omitted.
Regression to the Mean
In addition to accounting for differences in save quality, an adjustment needs to also be made for differences in sample size, such that goalies with a more limited body of above average work like Martin Jones should have their projection more heavily regressed to the mean (i.e., historical average save percentage) versus goalies with a large body of above average work like Henrik Lundqvist and Cory Schneider. While I’d like to be able to account for other league data to help overcome some of this sample size issue, as far as I know a) the data is not available on a 5v5 level, let alone by DZ and b) using equivalency factors on top of having to translate overall Save % into an Adjusted 5v5 metric may just create more noise. So for now, only goaltenders that have faced at least 1,300 NHL shots in the previous four seasons are included in the below projections, which comes out to a sample of 49 goalies eligible to play in the NHL next season (i.e., no Jonas Hiller or Anton Khudobin).Yes, that endpoint is a little arbitrary, but it was selected to include John Gibson in the sample, an interesting example given that he has led the sampled goalies in Low DZ Save % opportunities in the smallest amount of work. That's a big positive for the Anaheim Ducks given the majority of shots historically come from that area of the ice (around 44%, vs. 28% each for Medium and High DZ opportunities). However, other work has suggested that High DZ Save %'s is the most significantly correlated with future success, which Gibson has performed slightly below league average in. While I considered using just the High DZ variable to project future performance, I have instead decided to include all types of shots for sample size reasons, allowing the historical number of each DZ faced, saved at the historical success rates of 97%, 93% and 83% of Low, Medium and High DZ opportunities respectively. While I need to revalidate that these save percentages still hold true for the most recent seasons (i.e., 2013 to present) due to the downward trend in goal scoring, for now, this is what I’ll be assuming.
By following this method, Low DZ save attempts are most heavily regressed to the mean, though not entirely discounted. As a result, John Gibson is projected to have around a league average Save % projection despite having the highest projection in Low DZ opportunities. While this may seem confounding, driving this is the fact that this the area that goaltenders show the least amount of differentiation across the three DZs. As an additional check, I have also evaluated all 52 goalies that faced at least 3,000 shots from 2007-16 and evaluated how the averages, minimum and maximum 5v5 Save %'s looked compared to the projection model:
| 3K Shots | Sv%L | Sv%M | Sv%H |
| Max | 98.12% | 94.64% | 86.10% |
| Min | 96.92% | 91.54% | 80.51% |
| Mean | 97.41% | 92.97% | 83.33% |
| Median | 97.38% | 92.88% | 83.24% |
| St Dev | 0.29% | 0.68% | 1.36% |
| 16-17 Goalies | Sv%L | Sv%M | Sv%H |
| Max | 97.84% | 94.26% | 86.42% |
| Min | 96.87% | 92.21% | 83.09% |
| Mean | 97.43% | 93.15% | 84.91% |
| Median | 97.47% | 93.14% | 84.91% |
| St Dev | 0.23% | 0.37% | 0.71% |
| Data from War-on-Ice | |||
The initial model proposed by Garik19 used 1,525 shots of regression, though I have upped that to 2,000 to ensure that the Low DZ projections fell within the bounds of the historical averages. While the mean Medium DZ Save % trends a little higher than the historical averages and the High DZ a lot higher, for now, I think the projections are reasonable enough given the maximums align. However, if future work suggests that more regression should be added to these, then that will be added in.
Aging
Perhaps the most elusive thing to account for in a projection model, aging is something that should be accounted for, but 'how much' to apply is very much up for debate. There's been no shortage of ink spilled about this topic, including work done by Pension Plan Puppets, Garik16 and Cam Charron. What these past studies have in common is they have quantified aging by evaluating the average change in goaltenders' save percentages from one season to the next at different ages.I have taken an alternative approach - using 5v5 Career Save %'s from 2007-16, I tried to estimate on average how many percentages points below or above the career average a goalie performed at for his age, and applied that estimate to the Projected Save % value. I limited the number of shots faced within an individual season to 400 to account for small sample sizes. For comparison, Charron used 500, though because I had fewer seasons to work with, I wanted a little bit of larger of a sample than 500 would have yielded. The end result looks like:
 |
| Data from War-on-Ice |
Now the important disclaimer - while I wanted an aging effect in my projection model, I really do not like the 'macroeconomic' approach to aging. As this InGoal Magazine piece highlights, applying a universal aging curve to all individual goalies is going to draw heavy skepticism, particularly when you have older goaltenders like the aforementioned Thomas and Roberto Luongo excelling at ages where they should be significant decreases. Going forward, I would apply a more 'microeconomic' approach to aging, such that each individual goaltender's aging curve is weighted by the similarity of previous goaltenders' aging. But how to go about that is something I will save for another time.
Results
For reference: upcoming UFAs are highlighted in orange. A Projected Save % of .927% (i.e., Jimmy Howard) would be considered around league average based on both historical data and the number of goalies in the sample. |
| Data from War-on-Ice |
Perhaps more surprising are the No. 2 and No. 3 goalies on this list. But when you start to peel the layers of the onion back, it's two of the three DZs driving Steve Mason's and Thomas Greiss' projections, with Mason projected to finish 31st in High DZ and Greiss 47th in Medium DZ opportunities. Conversely, Ben Bishop and Henrik Lundqvist have Top 20 projections across all three DZs. So while Greiss and Mason may in theory offer higher variance options in net, they also would appear to have relatively big holes in their game that, unless properly defended against, could leave their teams in a less favorable spot in a playoff series.
On the flip side, the goalie concerns in Dallas appear to be as poor as advertised. Though I don't think you need a projection system to confirm that statement, where I find most alarming is Nashville, where franchise goaltender Pekka Rinne is expected to perform essentially around a replacement-level goaltender. While the aging factor is contributing a little bit to this low of a projection here, the odds that he is able to justify his remaining 3 years, $7M AAV contract are extremely unlikely, and viable options to supplement Rinne from the UFA market are not at all encouraging.
Perhaps most fascinating to me the Carolina Hurricanes situation. I have high regard for Ron Francis as a GM, but I hope the fact that he is publicly entertaining a Cam Ward return is more to help ease the franchise's separation from the former Conn Smythe winner as opposed to being an actual consideration. Only two goaltenders - Niklas Bäckström and Curtis McElhienny - project to perform worse than Ward next season, both of which are older and have dealt with significant injuries in the past couple years. Granted, the model could be understating Ward's projection. But the projection model should only help to confirm that a league average goalie like Frederik Andersen would offer the Hurricanes significantly more upside than bringing back Ward.
Of the pending UFAs, James Reimer is the most interesting situation. Despite teams like Dallas and Nashville needing additional help in net around the trade deadline, it was San Jose that swooped in and acquired the long-time Maple Leaf for a relatively small cost. While he will likely be seen as the best UFA goalie on market, it's unclear whether any of Carolina, Calgary or Toronto see him as a starter, or if the trade options are more favorable (e.g., Frederik Andersen, Ben Bishop, Jimmy Howard, Marc Andre Fleury). Should he have to settle for a 1b spot (such as returning to the Sharks), his projection suggests he could offer a playoff bound team with a high variance level of play in place of the starter, such as Thomas Greiss provided the NY Islanders with this spring.
