Projecting NHL Goaltender Performance for the 2016-17 Season

Source: Montreal Gazette

With the NHL off-season set to ramp up in a few weeks, I wanted to revisit a topic I did a deep dive into last off-season: projecting goaltender performance based off of improving upon baseball's Marcels forecasting system as proposed a couple of years ago by Garik16. While I wrote a separate post on what I think the Sabres should do in net this summer, this post discusses the projection system's methodology in greater detail. The goal is to describe the components of the ‘minimum viable' Projected Save % model I have developed for the 2016-17 season, which I hope to refine and improve upon in the near future.

Though the future of the goalie save data that drives this projection system is up in the air due to the War-on-Ice site’s founders joining the Minnesota Wild organization, my hope is that the momentum behind breaking down shot quality into save difficulty will continue in the hockey world, and that the model can be adjusted to work with future advanced save statistics down the road. So without further ado, below is the methodology broken down into four key areas:

Low, Medium and High Danger Zone (DZ) Opportunities

For me, what opened my eyes into improving goaltender evaluation was War-on-Ice pioneering the idea of adjusting goalies save percentages into three separate 'danger zones', based on how league-wide shooting percentages from three different areas of the ice:

Source: War-on-Ice

So when leveraging past performance to project future performance, I wanted to make sure that I was evaluating each individual goaltender’s performance by these three zones, as opposed to overall 5v5 or even Adjusted 5v5 performance, for a couple different reasons. First off, controlling for danger zones (DZ) help to better isolate team effects in front of the goalies, such as focusing on 5v5 helps to not penalize goalies playing on teams that take more penalties than the norm vs. Regular Save %. Second, DZs help to provide another layer of analysis when evaluating differences between goalie projection and which DZ is relatively driving a higher or lower than expected projection. There is additionally room to add in differing levels of regression by neutral zone if necessary (more on that concept below), though for this initial model I have stuck with the historical league averages (which I refer to as save and goal stats from the 2007-16 seasons, i.e., the Behind the Net era).

Past Performance Weighting

Independent of DZ, how to weight past performance when projecting future performance is work that others such as Eric Tulsky have spent much more time researching. This is currently a blind spot for me, as I haven’t dug further into replicating and/or revalidating Tulsky’s work. But for this model, I think that sticking to the 100%/60%/50%/30% system to weight the previous 4 seasons' performance is reasonable enough. Chiefly, it creates a larger body of past work to base future performance off of while incorporating the most recent season’s performance more heavily than from a few years ago.

It should also be noted that I have used regular season data only. While I plan on exploring playoff data in future iterations, my concern is that the effect of facing the same team 4-7 times in a row could skew some projections. To what effect, I'm not sure, but for now that data has been omitted.

Regression to the Mean

In addition to accounting for differences in save quality, an adjustment needs to also be made for differences in sample size, such that goalies with a more limited body of above average work like Martin Jones should have their projection more heavily regressed to the mean (i.e., historical average save percentage) versus goalies with a large body of above average work like Henrik Lundqvist and Cory Schneider. While I’d like to be able to account for other league data to help overcome some of this sample size issue, as far as I know a) the data is not available on a 5v5 level, let alone by DZ and b) using equivalency factors on top of having to translate overall Save % into an Adjusted 5v5 metric may just create more noise. So for now, only goaltenders that have faced at least 1,300 NHL shots in the previous four seasons are included in the below projections, which comes out to a sample of 49 goalies eligible to play in the NHL next season (i.e., no Jonas Hiller or Anton Khudobin).

Yes, that endpoint is a little arbitrary, but it was selected to include John Gibson in the sample, an interesting example given that he has led the sampled goalies in Low DZ Save % opportunities in the smallest amount of work. That's a big positive for the Anaheim Ducks given the majority of shots historically come from that area of the ice (around 44%, vs. 28% each for Medium and High DZ opportunities). However, other work has suggested that High DZ Save %'s is the most significantly correlated with future success, which Gibson has performed slightly below league average in. While I considered using just the High DZ variable to project future performance, I have instead decided to include all types of shots for sample size reasons, allowing the historical number of each DZ faced, saved at the historical success rates of 97%, 93% and 83% of Low, Medium and High DZ opportunities respectively. While I need to revalidate that these save percentages still hold true for the most recent seasons (i.e., 2013 to present) due to the downward trend in goal scoring, for now, this is what I’ll be assuming.

By following this method, Low DZ save attempts are most heavily regressed to the mean, though not entirely discounted. As a result, John Gibson is projected to have around a league average Save % projection despite having the highest projection in Low DZ opportunities. While this may seem confounding, driving this is the fact that this the area that goaltenders show the least amount of differentiation across the three DZs. As an additional check, I have also evaluated all 52 goalies that faced at least 3,000 shots from 2007-16 and evaluated how the averages, minimum and maximum 5v5 Save %'s looked compared to the projection model:

3K Shots	Sv%L	Sv%M	Sv%H
Max	98.12%	94.64%	86.10%
Min	96.92%	91.54%	80.51%
Mean	97.41%	92.97%	83.33%
Median	97.38%	92.88%	83.24%
St Dev	0.29%	0.68%	1.36%
16-17 Goalies	Sv%L	Sv%M	Sv%H
Max	97.84%	94.26%	86.42%
Min	96.87%	92.21%	83.09%
Mean	97.43%	93.15%	84.91%
Median	97.47%	93.14%	84.91%
St Dev	0.23%	0.37%	0.71%
Data from War-on-Ice

The initial model proposed by Garik19 used 1,525 shots of regression, though I have upped that to 2,000 to ensure that the Low DZ projections fell within the bounds of the historical averages. While the mean Medium DZ Save % trends a little higher than the historical averages and the High DZ a lot higher, for now, I think the projections are reasonable enough given the maximums align. However, if future work suggests that more regression should be added to these, then that will be added in.

Aging

Perhaps the most elusive thing to account for in a projection model, aging is something that should be accounted for, but 'how much' to apply is very much up for debate. There's been no shortage of ink spilled about this topic, including work done by Pension Plan Puppets, Garik16 and Cam Charron. What these past studies have in common is they have quantified aging by evaluating the average change in goaltenders' save percentages from one season to the next at different ages.

I have taken an alternative approach - using 5v5 Career Save %'s from 2007-16, I tried to estimate on average how many percentages points below or above the career average a goalie performed at for his age, and applied that estimate to the Projected Save % value. I limited the number of shots faced within an individual season to 400 to account for small sample sizes. For comparison, Charron used 500, though because I had fewer seasons to work with, I wanted a little bit of larger of a sample than 500 would have yielded. The end result looks like:

Data from War-on-Ice

By my math, goalies peak around ages 26-28, with a drop-off relative to their career Save %'s starting to adversely impact them past age 32. There are, of course, some significant outliers in here - you have to be a damn good prospect at age 20 (i.e., Steve Mason and Carey Price), or uncharacteristically effective at age 36+ (i.e., Tim Thomas), to get to face 400+ shots in a season. However, some significant outliers in the 38-40 range that result in the fit and shape of the curve not changing too much vs. plotting the graph from 21-35. Though the fit is poor, both the R^2 and shape of the cure align relatively well with what others have found - i.e., what ages are most negatively affected by an aging curve. So for the first iteration of the projection model, this is what I'm using.

Now the important disclaimer - while I wanted an aging effect in my projection model, I really do not like the 'macroeconomic' approach to aging. As this InGoal Magazine piece highlights, applying a universal aging curve to all individual goalies is going to draw heavy skepticism, particularly when you have older goaltenders like the aforementioned Thomas and Roberto Luongo excelling at ages where they should be significant decreases. Going forward, I would apply a more 'microeconomic' approach to aging, such that each individual goaltender's aging curve is weighted by the similarity of previous goaltenders' aging. But how to go about that is something I will save for another time.

Results

For reference: upcoming UFAs are highlighted in orange. A Projected Save % of .927% (i.e., Jimmy Howard) would be considered around league average based on both historical data and the number of goalies in the sample.

Data from War-on-Ice

It's always good to have a projection system where the top goalie on the list is who you would expect it to be in Carey Price. There's no way to directly account for the risk that his knee injury from the past season will negatively impact his future performance (albeit Price's 2015-16 results are move heavily regressed towards the mean than if he had played the whole season due to the smaller sample size). But with a Top 10 projection across all three DZs, it's very easy to see Price coming back to play a 60 game season next year, Montreal winning the division and Price taking home his second Hart trophy in three seasons.

Perhaps more surprising are the No. 2 and No. 3 goalies on this list. But when you start to peel the layers of the onion back, it's two of the three DZs driving Steve Mason's and Thomas Greiss' projections, with Mason projected to finish 31st in High DZ and Greiss 47th in Medium DZ opportunities. Conversely, Ben Bishop and Henrik Lundqvist have Top 20 projections across all three DZs. So while Greiss and Mason may in theory offer higher variance options in net, they also would appear to have relatively big holes in their game that, unless properly defended against, could leave their teams in a less favorable spot in a playoff series.

On the flip side, the goalie concerns in Dallas appear to be as poor as advertised. Though I don't think you need a projection system to confirm that statement, where I find most alarming is Nashville, where franchise goaltender Pekka Rinne is expected to perform essentially around a replacement-level goaltender. While the aging factor is contributing a little bit to this low of a projection here, the odds that he is able to justify his remaining 3 years, $7M AAV contract are extremely unlikely, and viable options to supplement Rinne from the UFA market are not at all encouraging.

Perhaps most fascinating to me the Carolina Hurricanes situation. I have high regard for Ron Francis as a GM, but I hope the fact that he is publicly entertaining a Cam Ward return is more to help ease the franchise's separation from the former Conn Smythe winner as opposed to being an actual consideration. Only two goaltenders - Niklas Bäckström and Curtis McElhienny - project to perform worse than Ward next season, both of which are older and have dealt with significant injuries in the past couple years. Granted, the model could be understating Ward's projection. But the projection model should only help to confirm that a league average goalie like Frederik Andersen would offer the Hurricanes significantly more upside than bringing back Ward.

Of the pending UFAs, James Reimer is the most interesting situation. Despite teams like Dallas and Nashville needing additional help in net around the trade deadline, it was San Jose that swooped in and acquired the long-time Maple Leaf for a relatively small cost. While he will likely be seen as the best UFA goalie on market, it's unclear whether any of Carolina, Calgary or Toronto see him as a starter, or if the trade options are more favorable (e.g., Frederik Andersen, Ben Bishop, Jimmy Howard, Marc Andre Fleury). Should he have to settle for a 1b spot (such as returning to the Sharks), his projection suggests he could offer a playoff bound team with a high variance level of play in place of the starter, such as Thomas Greiss provided the NY Islanders with this spring.

Next Steps

My goal was to develop a rough model to help inform Buffalo fans on where they should be looking for a backup goaltender option (something I'd consider to be a critical concern, given Robin Lehner's horrendous projection and the lack of an established backup in the Sabres' pipeline). But with the infrastructure in place, I'd really like to improve upon the work I've done. In particular, I'd like to further study how to weight past performance when predicting future performance; how differing DZ opportunity save %s influence future projection; further aging studies as discussed above; whether workload within any of the three DZs has an effect on Save %; and equivalency factors to bring other leagues (i.e., AHL) and pre-2007 data into the projection model. Sure, projecting goaltender performance will likely always remain voodoo, and will likely be exacerbated if the two-goalie system trend continues and sample sizes diminish. But it can only help to try, particularly in a salary cap league where a bad goal-tending contract (or two in Dallas' case) can significantly diminish your chances of winning in future seasons.

Who Would the Buffalo Sabres Protect in an Expansion Draft?

Source: Montreal Gazette

This is a 'for fun' piece I also submitted as a fan post over on Die By The Blade. 

With the last of the Buffalo Sabres' remaining RFAs, Jerry D`Amigo, resigning last week, we have unofficially entered the quiet season in Sabres hockey. Save for an unexpected UFA signing, it's safe to say that hockey-related Sabres news will be minimal until training camp kicks off in mid-September. So to help fill the void, I lifted an idea that other fan bases (e.g.,Stars, Oilers, Lightning, Maple Leafs) have been throwing around the last few weeks: who would the Sabres protect in a hypothetical expansion draft?

Before getting into the details of how an expansion draft may work, some assumptions: I believe that there will be at least one expansion team (i.e., Las Vegas and/or Quebec City) that will start play in the 2017-18 season, which is when the league anticipates expansion will occur. As a result, the players that the Sabres will need to protect will be reflected by their contract statuses in June of 2017, which is the time of the year that the 1998-2000 NHL Expansion Drafts were held. All contract details will reflect what is available on General Fanager. Of course, the Sabres' roster makeup two years from now is impossible to predict, but divvying up the current roster of players into those the team would protect vs. expose turns out to be a fun and straight forward experiment if we assume the players are still on the roster. I will also make the assumption that the upcoming expansion draft(s) will also work how the previous expansion drafts did:

Teams choose to protect either nine forwards, five defensemen, and one goalie; or seven forwards, three defensemen, and two goalies

If protecting one goalie, there is no experience requirement for the goalie left unprotected. If protecting two goalies, they must leave one goalie unprotected who played in at least 10 games the prior season, or 25 combined games over the past two seasons

Teams must leave one defenseman and two forwards unprotected who played 40 NHL games the prior season, or 70 combined games over the prior two seasons

Players on entry-level contracts are automatically exempt, and every existing team will have exactly two players selected

One other thing that needs to be addressed is the introduction and proliferation of NMCs/NTCs since the last round of expansion. For this exercise, I will assume that players with a NMC in the 2016-17 season can block themselves from being drafted, but those with Full and Modified NTCs cannot. This assumption won't have much bearing for Buffalo, as Zach Bogosian is the only player that will have a NMC in 2016-17. Since late June is still technically in the 2016-17 season, pending UFAs are also treated as players that the Sabres can leave exposed (a scenario that occurred in 2000 when Dallas Drake, whose rights were claimed by the Columbus Blue Jackets, instead opted to test free agency and signed with the St. Louis Blues).

First, I will start off with next season's UFAs that I don't believe will be on the Sabres roster in 2016-17 or, if they are, the team won't hesitate to leave exposed. They include: David Legwand, Mike Weber, Carlo Colaiacovo, and Chad Johnson. Buffalo also would be favorably affected by an expansion draft happening in June 2017 as players on ELCs and unsigned prospects are exempt, meaning Buffalo wouldn't have to use a protection spot. Notables on that list would include Jack Eichel, Sam Reinhart, Nicholas Baptiste, Justin Bailey, William Carrier, Hudson Fasching, Evan Rodrigues, Anthony Florentino, Brendan Guhle, Linus Ullmark, Jason Kasdorf, Cal Petersen and whomever else the Sabres select in the 2016 NHL Entry Level Draft.

As to which scenario the Sabres would elect to use (9F, 5D and 1G, or 7F, 3D and 2G), the former is the obvious choices the odds are good that Robin Lehner he will be the only goaltender worth protecting. There's always the chance that Nathan Lieuwen or Andrey Makarov takes such a huge step in Rochester next season that the Sabres would feel one of them is worth protecting. However, neither of the goalies seems like they would be worth leaving an additional four roster players exposed. Even if Las Vegas Black Knights and/or the Quebec Nordiques were interested in Liuewen or Makarov, the precedent does exist that expansion teams will agree to not select certain exposed players in exchange for picks and/or players, as the San Jose Sharks pulled off with Evgeni Nabokov in 2000.

Given the current state of the Sabres' defense, it seems like a slam dunk that Bogosian, Rasmus Ristolainen, Mark Pysyk, and Jake McCabe would be four of the five protected defensemen. As teams must also leave one defenseman exposed that played either 40 games in the previous season or 70 in the previous two, I will leave the injury-prone Josh Gorges and his one remaining year of $3.9M exposed in this exercise. I think it's worth reiterating that just because a player is exposed he won't necessarily be selected, but having a 32-year old Gorges get claimed would likely not be the worst thing in the world. I personally believe that there will be a different fifth defenseman worth protecting once 2017 rolls around, but for the sake of this exercise I have decided to make Chad Ruhwedel the fifth protected player (though to be honest, I think it's a coin flip whether him or Matt Donovan is more worthy of protecting).

As for the nine forwards, there are four players that I would consider to be slam dunks to be protected: Ryan O`Reilly, Evander Kane, Tyler Ennis and Zemgus Girgensons. I would also consider Johan Larsson and Marcus Foligno to be locks, provided neither is involved in a trade in the next two seasons. Assuming the ELC provision will stay the same and the Sabres don't have to use a spot on Eichel or Reinhart, the Sabres shouldn't have any trouble protecting the forwards they want to keep. Though he's a 2016 UFA and would need to be resigned first, I've also added Jamie McGinn to my list of protected forwards just because there's room to shield him (though as with Ruhwedel, I think another RW will come into the mix by 2017 that will make McGinn expendable). As far as having two NHL forwards exposed goes, 2017 UFAs Brian Gionta and Cody McCormick would fit the bill as the odds that either of these players will be retained by the Sabres beyond 2017-18 are pretty low.

Ultimately though, the most decisive player to leave exposed or not is going to be Matt Moulson, who in 2017 will be 33-years old and have two remaining years at $5M per season. There will be a lot of variables that will ultimately shape this decision: will Moulson's regain his form as a 20-30 goal scorer alongside Eichel, or was last season's 14-goal performance a sign of things to come? Even if Moulson regains his form, would leaving him exposed in 2017 be worth getting him off the books for 2018-19, when Kane will be a UFA and Eichel and Reinhart are RFAs? As with all teams, the continued weakening of Canadian dollar is likely to negatively affect the Sabres' long-term cap situation. Granted, I am talking 2-3 years out for an expansion scenario that may or may not happen. As of today, I have Moulson in my keep list as I am bullish on his future production now that he has high end making centers to play with again. But assuming expansion does happen in 2017, I would expect protecting or exposing Moulson will become a #HotButton issue for Sabres fans.

So my final protected list and key players exposed looks like the following. While teams like Dallas will have tougher logjams to address, it's safe to say that expansion teams have it pretty rough just based on the players they would be able to claim from Buffalo. Feel free to leave comments on who you think should and/or would be protected in an NHL Expansion Draft.

Goalie (1): Lehner

Defense (5): Bogosian, Ristolainen, Pysyk, McCabe, Ruhwedel

Forward (9): O`Reilly, Kane, Ennis, Girgensons, Foligno, Larsson, Moulson, McGinn, Nicolas Deslauriers

Notable Exposed Players: Gorges, Donovan, Gionta, McCormick, D`Amigo, Cal O`Reilly, Philip Varone, Tim Schaller, Jerome Gauthier-Leduc, Lieuwen, Makarov

An In-Depth Robin Lehner Trade Analysis

Source: Jay Kopinski/Icon SMI

Also submitted as a fanpost on Die By The Blade.

When the news first broke that the Buffalo Sabres had traded the 21st overall pick in the 2015 NHL Entry Draft in exchange for Robin Lehner and David Legwand from the Ottawa Senators, a good portion of Sabres nation freaked out. Not only were the Sabres trading away a 1st round pick in a draft that many have been labeling as the deepest since 2003, they were also acquiring a starting goaltender that many fans didn't even consider to be the best option available on the trade market (myself included). But despite his struggles in Ottawa over the past two seasons, there's no denying that the soon-to-be 24-year old has excellent long-term potential.

Now that the dust has settled, the question I want to answer is whether Tim Murray made the best move by acquiring Robin Lehner. To evaluate this, I'm comparing Lehner to the other goalies that were traded and had been previously linked to the Sabres (Cam Talbot, Eddie Lack, Martin Jones and the negotiating rights for Antti Niemi) through the following lenses: projected performance, the cost of the draft pick(s) that each goaltender was exchanged for, and the value of each goaltender's contract.

Projected Performance

Let's start with the caveat that the easiest thing to predict about goaltenders' performance from season to season is that it's very unpredictable, though the more shots a goaltender has faced over the course of his career the better we are able to project his performance in the future. There's also the question of whether 5v5 / Even-Strength Save % is the best metric for evaluating goaltender performance, as well as the accuracy of shot location data for High, Medium and Low Danger Zone opportunities from War-on-Ice that will play a large part in this analysis. So given the limited tools available, how does Lehner stack up against the other recently traded goalies? 

A few days before the trade frenzy, the phenomenal InGoal Magazine ran a piece breaking down Talbot, Lack and Lehner by Danger Zone (or DZ). In summary, the article determined that Talbot was the most preferable of the three goalies, followed by Lack and then Lehner. By comparing all three goaltenders' Career Save %s by DZ and how that compared to the volume of shots and quality of goaltenders that Buffalo endured last season, the article estimated that Talbot would have saved the Sabres an estimated 36 goals last season (or an additional 6 wins, applying a thumb rule developed by Eric Tulsky a few years ago). As for Lack and Lehner, they would have respectively saved just 12 and 2 more goals than the Buffalo Sabres' 2014-15 goaltending battery of Michal Neuvirth, Jhonas Enroth, Anders Lindback, Matt Hackett and Andrey Makarov managed to stop. 

But when it comes to evaluating goaltenders' desirability based on past performance alone, Lehner should get a lot more benefit of the doubt than the article gives him for a few different reasons. First off, part of Lehner's performance likely can be explained by Paul MacLean's system simply not being the best fit for him, a possibility raised by both InGoal Editor / goaltending guru Kevin Woodley and Lehner himself. As Woodley notes in a seperate piece:

"[Lehner] has not progressed technically during his time with the Senators. That’s not shocking given the way Ottawa’s goalies play, with an emphasis on more aggressive positioning than in many other places and more flow and movement as a result."

Age also benefits Lehner favorably relative to the other recently traded goalies, as the soon-to-be 24-year old is just now entering the part of his career where most goaltenders peak. Conversely, Talbot and Lack are already both 27, still close to their primes but at the start of the gradual drop-off. Another theory that could also work in Lehner's favor is the idea that he will thrive now that he's out of Craig Anderson's shadow and will be the undisputed starter in Buffalo, ahead of Chad Johnson. This would be similar to how Sergei Bobrovsky showed flashes of potential in Philadelphia but didn't blossom until he was traded to Columbus and handed the starting job.

While Lehner's Career 5v5 Save % is disconcerting when compared to the other goaltending candidates (91.77%, or .4% below the league average from 2007-15 per data from War-on-Ice), a deeper look into Lehner's performance by DZ tells a different story:

When predicting future goaltending success, High DZ saves have been found to be the most correlated with overall success, while Medium DZ saves have some correlation and Low DZ saves have virtually none at all. Given the relatively large variance of High DZ Save % for 49 goalies that faced at least 3,000 shots from 2007-15, it's most likely that differentiable skill is surfacing in High DZ Save %, which is the one DZ area that Lehner comes out ahead of the league average (albeit in a pretty limited sample size). Conversely, Lehner's Low DZ Save % falls significantly below the 2007-15 average of 97.4%; since all but two goalies from 2007-15 had Career Low DZ Save %s within 97-98%, Lehner is a prime candidate to see his Low DZ Save % rise towards the league average over the next few seasons. To a lesser extent, we should also expect Lehner's significantly low Medium DZ Save % to rise towards the league average. Lehner also has a great opportunity to improve on his High DZ Save % due to his young age and the volume of work and development he will get this season.

While Lehner's career save percentages by DZ helps validate the idea that he was relatively unlucky during his time in Ottawa, some fans may still feel strongly that Martin Jones or Cam Talbot would have been better options. However, both goaltenders benefited from some luck in even smaller sample sizes than Lehner, with both Jones and Talbot accruing unsustainably High DZ Save %s on defensively stout teams during the peak years of their careers. Given that Cory Schneider has the best Career High DZ Save % of 86.09% from War-on-Ice's dataset and the only other goalies above 85% are Jonas Hiller, Tuukka Rask, Braden Holtby and Henrik Lundqvist, Jones (86.2%) and Talbot (88.3%) will almost certainly see their High DZ %s regress in San Jose and Edmonton, respectively. The question that is difficult to answer is how much regression we should expect each to experience over the next few seasons, and how their long-term performance will compare to Lehner.

To help evaluate this, I utilized a hockey version of baseball's Marcels forecasting system as developed by Garik16, which in turn is based off of Brian MacDonald's Bayesian approach to projecting goaltender performance. While the Marcels model is still very new and has its limitations (e.g., a below average goaltender is positively impacted by regression to the mean; it does not utilize Adjusted Save %s in its forecast), its use of regression and aging makes it the best available tool for projecting future performance. After refreshing and replicating Garik16's projections from February of 2015 for 5v5 Career Save %, I came up with the below ballpark values that we should see expect each goaltender to perform at over the next few seasons and compared it to their past performance:

I have a much more thorough examination of the Marcels methodology in a separate post. But it should be noted that the previous season's performance plays a considerable effect on how well a goaltender is expected to play in the future, so a significantly lucky or unlucky season the year before is going to have a large effect on the projection. Using the same weighting methodology following the 2012-13 season, Lehner would have been projected to be a long-term .924 5v5 goalie - i.e., where Talbot is now - but a variety of factors contributed to him being a .912 5v5 goalie the past two seasons.  Whether those factors were plain old bad luck or Lehner's initial success in the NHL (.938 5v5 goalie from 2011-13) being significantly higher than his actual ability will be played out in Buffalo this season.

One additional consideration that should be noted is that all the goaltenders in this analysis have switched teams, which could have an effect on projections. However, the odds that a new team will significantly impact any of the goaltenders' future performance are pretty low. Generally speaking, shot quality distribution for starting goaltenders has generally been found to only affect save percentage by 0.5% or less, while save percentages by shot attempt quantity has too much noise to claim that a higher workload has a positive or negative impact. So while I am particularly worried for Cam Talbot in Edmonton given the recent history there, individual talent and luck should be the two main factors driving future goaltender performance.

Speaking of luck, the role that it plays in goaltender save percentages and performance gives a lot of weight to the argument that the Sabres simply should've went for the goalie that was the best value. Knowing what we know about past performance and future projections, was the price that Buffalo paid to acquire Lehner worth giving up the 21st overall pick in the 2015 NHL Entry Level Draft?

Draft Pick Value

My initial reaction to the Robin Lehner trade was that Eddie Lack or Martin Jones would have been better trade options for the Sabres due to the relatively lower asking prices that Los Angeles and Vancouver would demand from Buffalo, as the Sabres were the only Eastern Conference team seeking a starter for the 2015-16 season. However, I was glad that Buffalo traded just the #21 pick to the Senators as opposed to #31 and #51, which Bryan Murray would have likely preferred based on his earlier comments. Additionally, I was glad that the #21 pick was used to acquire Lehner as opposed to an 18-year old goalie, which is an even riskier proposition in my opinion. So was I on point with my initial reaction that many Sabres fans likely shared?

For this analysis, I'll evaluate what the Sabres, Oilers, Sharks, Hurricanes and Stars respectively paid the Senators, Rangers, Canucks, Bruins and Sharks into a couple different draft pick valuation systems to help determine trade value. It should be noted that I'm only evaluating the Bruins' trade of Jones to San Jose, as the initial trade of Jones to the Bruins was part of a package the Kings used to acquire Milan Lucic. I also omitted David Legwand's inclusion in the Lehner trade from this analysis, as the team's current cap situation makes it easy for Buffalo to either keep or waive him at any point this season.

The first valuation system I utilized was developed by Scott Cullen of TSN and assigns each player selected from the 1990 to 2010 drafts a value of 1-10 as an indicator of his success, with 1 being a player that has played 10 NHL games or less and 10 being a 'Generational' player. Each draft slot is then given an average expected score based on past success of the players drafted in those spots alone. It should be noted that this system has a bunch of disclaimers that I dedicated a separate post to when it came to evaluating these trades (including my work around for valuing the Sharks' future first round pick, but more on that in a bit). However, what makes Cullen's methodology advantageous is it creates a ballpark value for each pick in the draft that can then be translated into in plain English, such that #1 overall pick is expected to be a 1st liner while the 21st is expected to fall somewhere between a Fringe (200-349 NHL games) and Regular (350+) NHLer.

The second valuation system I used was developed by Eric Tulsky, who looked at 46 draft pick trades from the 2006 to 2012 drafts to estimate the prices GMs had to pay to move up and down in the draft  them and placed it on a scale of 0-100 (similar to the NFL Draft's Value Board). As shown in the above graph, I mapped Tulsky's graph into an exponential equation that gives us an R^2 of over 96%, a near perfect fit outside of the Top 10. What separates these two systems is the end goal - Tulsky's is about the expected cost for moving up and down in the draft, while Cullen's is about the expected value of the player that you draft in those positions. 

While trading a first rounder for Lehner may have sounded bad initially, both systems have it as being superior to trading two second round picks. The only way trading #21 instead of #31 and #51 can be considered a loss for the Sabres is if the Avalanche would have demanded a lesser prospect than JT Compher  in the Ryan O'Reilly trade if #21 was still play (side note: I highly doubt Colorado makes the trade with Buffalo unless Nikita Zadorov is included, given their need for left handed defensemen). Additionally, Robin Lehner is well on his way to being a Regular NHLer, meaning Buffalo gave Ottawa fair value to get the goalie they considered to be the best on the market (from an in-division rival, no less).

Depending on the valuation system employed, different conclusions come out over who paid the higher price between Buffalo and Edmonton. While Tulsky's draft value board would indicate that the Oilers spent half the amount of draft capital than Buffalo did, the Oilers gave up about the same amount of expected NHL talent as the Sabres did according to Cullen's system. This is not an unreasonable conclusion: the odds that any one of the Rangers' three draft picks becomes a Regular NHLer are probably about as good that the player that ultimately went #21 to Ottawa, Colin White, does. Though more research should be done into this, a market inefficiency could exist in teams overvaluing position in the latter (4th through 7th) rounds, meaning that teams should be trying to move down as often as possible to gain additional picks once they are in the back nine of the draft.

What is also important to remember is when these trades occurred. While many hockey pundits thought Cam Talbot would be the first domino to fall, Buffalo made the first move before another team that lost out on Talbot would ante up to acquire Lehner. While Murray's aggressiveness means he didn't get the best value on the trade market (that distinction goes to Ron Francis), the Sabres avoided the fate of the Sharks. Having missed out on the other trade options and seeing their previous starter Antti Niemi' sign with Dallas after his negotiating rights were traded there, the Sharks were forced to pony up a 2016 1st round pick and a prospect for Jones. A future 1st first pick is a significantly higher risk than moving #21 given that teams that miss playoffs (which is definitely a possibility for the Sharks) will have three chances to move into the top of the draft starting next summer. 

In any scenario, the price that Hurricanes spent on Eddie Lack was the best value of the goalie trades, as Jim Benning had to settle for a 3rd round pick after shopping Lack for a 2nd rounder. Additionally, the price that Ron Francis paid for Eddie Lack is the same price that Tim Murray could have gotten Lack for, given Vancouver's preference to move him east (versus Calgary or Edmonton). What is not in Lack's favor is the fact that he has the worst Career High Danger Zone (DZ) Save % of the traded goalies, and is a strong candidate to see his Medium Save % regress back towards the mean. But Lack has also seeked out Lyle Mast to train with this summer, the founder of the Head Trajectory puck trucking philosophy for goaltenders that has been identified as one of the contributing factors to Devan Dubnyk's bounce back season. Should this training help Lack realize just some of the gains that Dubnyk made in High DZ Save % from 2013-14 (81.3%) to 2014-15 (85.8%), then the Lack trade could rank alongisde Noah Hanifin falling to #5 as the steal of the summer. Given that Lack (as well as Talbot) are entering contract years, great seasons will respectively force Carolina and Edmonton into tough contract decisions - and this is where Lehner's value to Buffalo really comes into play.

Contract Value

As stated before, the more shots a goaltender has faced over the course of his career, the better his future performance can be projected. Of the goaltenders traded this summer, only the soon-to-be 32-year old Antti Niemi had faced more career shots, and Dallas will be paying him $13.5M over the next three years to provide league average goaltending in a best case scenario. With Lack and Talbot, who are both set to be UFAs in 2016, Carolina and Edmonton will have just a few seasons worth of data to determine whether to extend them long term or not.

The Sabres and Sharks, meanwhile, will respectively have four and three seasons until their newly acquired goaltenders are eligible to be UFAs, giving both teams plenty of time for their goaltenders to get experience and for the two teams to make more informed decisions than the Hurricanes and Oilers will be able to next season. Though they both cost first round picks, the fact that Jones and Lehner both have high ceilings, are in their primes age-wise and are on $2.5-3M AAV contracts for the next few seasons makes them less risky bets in the long term (i.e., beyond next season) than Lack and Talbot. And in a worst case scenario that Lehner and/or Jones don't pan out, both will have relatively easy contracts to move on from as opposed to, say, the $10.4M tandem in Dallas, or what Talbot and Lack could hypothetically get paid if they have career years. The only other question left to answer is whether Buffalo made the right bet on Lehner.

Verdict

I had initially hoped that the Sabres would land Martin Jones, thinking he offered the best trade off between talent, youth, and cost to the Sabres. However, the fact that Jones was ultimately traded for a higher price has me thinking that Robin Lehner is a slightly better bet given he is two years younger than Jones and his recent run of bad luck should turn in Buffalo, as should Jones' good luck in San Jose. Additionally, Jones has just 1/3 of the body of work that Lehner has in the NHL. But what I am ultimately happy about is that the Sabres didn't acquire Talbot or Lack due to their upcoming contract situations nor a UFA goalie like Niemi or Karri Ramo, as average goaltending is readily available every summer. Great goaltending has to either be nurtured or acquired at a steep price (hence why Anaheim isn't pushing John Gibson or Frederik Andersen out the door). With no other goaltending prospects in the Sabres' pipeline being close to NHL-ready, Lehner provides the Sabres their best chance of developing a top 10 goaltender in the NHL within the next few seasons.

So I will trust that Tim Murray has done his homework on Lehner, recognizing the role bad luck has played thus far in his career, and did not acquire him simply because he's familiar with Lehner from his Ottawa days. I am also optimistic about the role Andrew Allen can play in Lehner's development, given his work with the Blackhawks' young goaltenders in Rockford and the fact Dan Bylsma hired him with Robin Lehner's development in mind. But even in a worse case scenario where Lehner does not pan out as hoped, his affordable contract would make it easy for Buffalo to move on from (and hey, the Lightning did have to go through Anders Lindback before they got to Ben Bishop and Andrei Vasilevskiy). When it comes to risk taking, I would rather see the Sabres swing for the fences in net than to be perpetually stuck in a state of goaltending mediocrity.  

NHL Draft Pick Valuation

When it comes to measuring NHL Entry Level draft pick value in the form of the NFL's draft value board, a ton of work has been done on this. But what I really cared about in when writing my analysis on the recent Robin Lehner, Cam Talbot, Martin Jones and Eddie Lack trades was the expected contributions of the players that those traded picks were selected with, as opposed to what those picks would have been worth to move up and down in the draft. So when I stumbled upon Scott Cullen's, my work was done for me. But I do think there are a lot of caveats that should be disclosed here in addition to the ones he highlights.

First off, the 1 to 10 scale is very subjective once you get up into the 8-9-10 range, particularly when we are separating out Generational from Elite players. There's also some scale comparability issues (e.g., the gap between a 4 and 1 is significantly less than a 10 and 7), but since the picks I'm dealing with in my analysis are further down the scale, I think the system does a nice job of getting in the right ballpark. It should also be noted that there's probably not much difference between a 4.7 and 4.8 in the below table (i.e., the Lehner and Talbot trades); but it's interesting to note that there's a clear hierarchy between the goalie trades, with Jones, Lehner/Talbot, Lack and Niemi's negotiating rights rating from most to least costly.

Where I think this system could use further refinement is making it more mathematical to account for the fact that we're only working with 20 drafts worth of data. Though the #21 pick that went over to Ottawa for Lehner is rated a 4.7, it actually is higher than several picks above it, but #21 should not be projected to have a higher expected value than #15-19 in future drafts. Smoothing this out with a best fit line would more definitively help teams value draft picks in terms of future vs. current NHLers as well as to identifies potential trading market inefficiencies such as position in lower round draft picks.

As far as my own application of Cullen's model I had to make a few estimations with the 2016 futures to get the Cam Talbot and Martin Jones trades into order. The 2016 7th round pick that the Rangers trade sent with Talbot to Edmonton was easy enough to estimate at around 1.5, given there's little variation in the expected value of players drafted in the 7th round. The 2016 1st rounder that the Sharks traded Boston is considerably more difficult.

For the first time in a decade, the Sharks can't be penciled in for a playoff spot. What was once a dominant possession team fell to being 'just' above average last season, and it's anyone's guess whether the infusion of Pete DeBoer, Paul Martin, Joel Ward and Martin Jones into next year's squad will be enough to get them back. Though I would predict they get back in given how unlucky the team was this past season, next year's first round pick is potentially worth even more than in past seasons due to the NHL's draft lottery format getting retooled to more closely resemble the NBA's (i.e., the top three spots are subject to a lottery vs. just the top pick). With all that said, I pegged the pick to be worth somewhere between a 1st round forward and a 1st round defenseman (assuming Boston doesn't take a goalie with that pick) at a 4.85. I also used Sean Kurlay's draft pick position (#133 in the 2011 draft) to help highlight just how much the Sharks had to pay to acquire Jones.

Hockey Marcels Evaluation

As part of a separate piece that I wrote evaluating the Robin Lehner trade, I used a hockey version of baseball's Marcel system as developed by Garik16 to project the save percentages of goaltender trade options the Sabres had to choose from before they pulled the trigger on a swap with Ottawa. In the interest of space in that other piece, I saved my discussion of the projection system for those interested in this post.

For starters, my main deviation from the original Marcels hockey system was that I wanted to project 5v5 Save % as opposed to regular Save %, with the thinking that 5v5 was a slightly more reliable indicator of goaltender quality. Speaking of sample size, the considerably small amount of High Danger Zone (DZ) Saves that Lehner, Martin Jones, Cam Talbot and Eddie Lack have had to make thus far in their careers made it far from the best tool to project going forward, despite it being the most correlated with overall goaltending success.

Some may also take up issue with the use of weighting the most recent season more heavily than the seasons before, something that affects Lehner much more negatively than the other goalies in this sample due to his tumultuous 2014-15 season. But even when I messed around with increasing the importance of saves from more than one season ago, the effect it had on the goaltenders in this sample was pretty minimal. In addition to past performance, the regression to the mean methodology (which adds 1,525 shots saved at the league average rate to each goaltenders prior four years) has a huge impact on projected performance in that a goalie that's below the league average has his projection pulled up - i.e., Lehner. Some may disagree with regressing a below average goalie to the mean, but if we are to account for exorbitant amount of good luck that Jones and Talbot experienced in their stops, then we should also adjust for bad luck Lehner likely experienced in Ottawa.

Hockey Marcels is also based on a Bayesian approach developed by Brian MacDonald, for which has a niftier way to account for the small sample size issue. The lesser number of shots a goaltender has faced in his career, the higher the variance is for a save percentage that a goaltender is expected to have with 95% confidence. As a result, the Bayesian approach would forecast goaltenders averages in the ballpark of the following values, highlighting that it is impossible to deem any one of the five goaltenders as being a significantly superior option based on forecasted performance alone:

But what I like better about Hockey Marcels is a) it applies a weighting methodology that places higher value on saves made in more recent seasons and b) it accounts for aging. While I am interested in further exploring the aging methodology, I will save that for another time; what's important to me about any projection system is that younger goalies' future performance should be regressed relatively less than older goalies. As a result, a soon-to-be 32-year old, .924 5v5 Career Save % goalie like Antti Niemi is projected to have a lower save percentage over the next three seasons than soon-to-be 24-year old, .918 5v5 Career Save % goalie like Robin Lehner.

As also highlighted in the other article though, there's many downfalls to simply using Save % to evaluating goaltender quality - hence ongoing research into and development of alternate measures of goaltending quality such as Adjusted Save % and Stephen Valiquette's Royal Road project. In addition, luck can play a huge role in early career save percentages - hence why an undrafted goaltender like Martin Jones can fetch a 1st round pick for Boston despite having an unsustainable High DZ Save % in 36 career NHL games.

While I have analysis on the Lehner trade to finish up, the direction I would like to toy with tacking Marcels is a weighting methodology by DZ, and then apply aging to that Adjusted Save %. Long term, all goaltenders should have Small DZ Save %s in the 97%-98% - of the 49 NHL goaltenders with 3,000 shots against from 2007-15, only Tim Thomas (98.12%) and Chris Mason (96.98%) fall outside of that range - while the weighting regression methodology currently being applied to overall Save % should be applied to High DZ Save %s. Medium DZ may be more of a hybrid between Low and High DZ, and is where much of the experimentation will come in.