With the NHL playoffs starting this week, all eyes will be on the goaltenders, hockey’s masked men. A goalie’s time on the ice is a solitary existence, with flurries of activity punctuating long periods of inaction. Hockey has been called the ultimate team sport, but all too often the goaltender finds himself alone as the puck approaches.Because the goalie is a team’s last line of defense, it’s no surprise that strong performance in net is incredibly important to winning a hockey game. During the regular season, save percentage (the generally accepted shorthand measure of goaltending effectiveness) explains a higher proportion of team performance than any other fundamental factor1Think of hockey’s “Four Factors” as the following: generating shots (as measured by shots per game), scoring on a high percentage of those shots (shooting percentage), preventing opposing shots (shots allowed per game), and stopping shots (save percentage). Together, these factors explain over 99 percent of a team’s goals-per game-differential, which in turn explains 92 percent of point percentage (a team’s standings points divided by the total number of points available in its games). in hockey.2The measure of relative importance used here is the Lindeman, Merenda and Gold (LMG) method described in this paper. In NHL regular seasons since 1988, team save percentage has a 29.5 percent LMG value when regressed against goal differential/game, compared to 28.9 percent for shooting percentage, 24.0 percent for shots allowed/game, and 17.7 percent for shots/game. In the playoffs, the emphasis on goaltending only intensifies; save percentage is easily the most important determinant of a team’s goals-per-game differential in the postseason.3In postseason play since 1988, team save percentage has a 43.3 percent LMG value when regressed against per-game goal differential, compared to 34.9 percent for shooting percentage, 13.3 percent for shots allowed/game and 8.6 percent for shots/game. A hot goalie really is the key to a successful playoff run.But herein lies a great paradox: Despite goaltending’s outsize impact on the outcomes of hockey games, it’s extremely hard to say exactly which goalies are truly good or bad at their jobs.This perplexing point was raised by the authors of the 2010 book “Stumbling on Wins,” and it still stands today. Using Hockey-Reference.com’s adjusted version of the save percentage statistic, adjusted Goals Against Percentage (GA%-),4I used GA%- because it’s useful for historical analysis since it compares a goalie’s save percentage to the ever-changing league average. GA%- is scaled to represent the percentage of the league’s rate of goals per shot that a player allows, so lower is better. For example, a GA%- of 84, like Henrik Lundqvist had last season, means he allowed only 84 percent of the number of goals a league-average goalie would have allowed on the same number of shots. the correlation of goalie performance from year to year is so low5A correlation coefficient of 0.296, to be exact, for goalies who qualified for Hockey-Reference’s leaderboards in back-to-back seasons. that, in practical terms, only 30 percent of the difference we see between a goalie and the league average in any given season actually “belongs” to the goalie himself. The rest is just random.6So when Ottawa’s Craig Anderson led the NHL last season with a 67 GA%- (the second-lowest mark of any goalie since 1984), the best expectation of his talent going forward was still only a GA%- of 90 — the assumption being that the other 23 points of GA%- were probably due to random variance. (If you’re doubting that assumption, Anderson snapped back to earth this season with a 104 GA%-.)The poor correlation of save percentage from one year to the next also indicates that goalies are extremely volatile commodities. For instance, if a goaltender is above average in a given season, there’s only a 59.2 percent chance he’ll be above average again the following year. And if he’s below average now, don’t worry: There’s a 47.2 percent probability that he’ll be above average next season.7Some of this is admittedly due to selection bias; by zeroing in on goalies we knew had a “next season,” we’re implicitly weeding out the ones who played poorly and were never given another chance — presumably because scouts decided they were as bad as the numbers said. But the threshold to qualify for Hockey-Reference’s save percentage leaderboard is a mere 26 games in a normal season, so the selection effect shouldn’t influence the results too much.Take Brian Elliott of the St. Louis Blues. During the 2010-11 season, Elliott was the NHL’s second-worst qualified goalie — only Nikolai Khabibulin was less effective at stopping pucks — and in 142 career games he had a lifetime GA%- of 111 (which translates to 11 percent worse than league average). If any goalie seemed unlikely to play well in the future, it was Elliott, but the very next year he led the NHL with a 69 GA%- (31 percent better than league average), at the time the third-best single-season performance by any goalie since the NHL started tracking save percentage in 1984.8Elliott’s 2011-12 season now ranks fourth because Craig Anderson put up a 67 GA%- last season. And how did Elliott follow that brilliant campaign? By posting a below-average 106 GA%- last season, and a 90 GA%- this year.It’s the kind of thing that will spin you around faster than a Pavel Datsyuk deke.So is Elliott a good goaltender or not? We can say he is probably a slightly below-average netminder who happened to have an all-time outlier of a career year in 2012. But that’s just because we have a relatively large amount of data on him by now. His career GA%- is 103 after nearly 250 games and 6,000 shots faced. A goalie’s save percentage only begins to stabilize after facing around 3,000 shots, at which point we would expect half of his observed performance to be talent (the rest is still luck). The busiest goaltenders each year face roughly 2,000 shots, so it takes about a season and a half for GA%- to offer insight on even the biggest goaltending workhorses.This does not mean that there is no difference in talent among goalies. It just means there’s a great deal of uncertainty around how any one goalie compares to another, and that the distribution of talent among NHL-caliber goaltenders is significantly more narrow than would be expected from looking at season-level save percentages alone.9The spread of which is artificially inflated by luck in small samples. As a consequence, the “replacement-level” save percentage for goalies (to borrow a term from baseball’s sabermetrics, referring to the production a team could expect from a minimum-salary player freely available on the waiver wire) is remarkably close to league average.10In keeping with the ratios of cap dollars devoted to each position, a save percentage .006 below average is probably the optimal replacement level. In 2013-14, that would set the replacement level at .908, a number that was average just five seasons ago. This, too, is a product of the uncertainty surrounding the true talent level of any given goalie — with such high levels of volatility, teams don’t need to accept bad goaltending performances for long. Given what little information we have about any goalie’s actual talent, a backup is almost as likely to give above-replacement production as a struggling starter is.If chance overwhelms skill in an entire regular season’s worth of goaltending statistics, imagine what can happen in the playoffs, when the leading goalies face but 800 shots at most. The Vancouver Canucks have experience with this: Kirk McLean, a nine-year veteran with a perfectly average 100 career GA%- going into the 1994 playoffs, backstopped the team to within a win of the Stanley Cup on the strength of a stellar 78 GA%- in the postseason. So do Capitals die-hards: Olaf Kolzig led Washington to the Finals in 1998 with a playoff GA%- that was 27 points lower than his career average. And Mike Smith nearly did the same for the 2012 Phoenix Coyotes. Playoff history is littered with seemingly nondescript goaltenders who suddenly became incredible puck-stoppers come springtime.But history can also cut the other way. In 2001, the great Patrick Roy had a regular-season GA%- of 90 — 13th in the league, if slightly down from his peak numbers of a few years earlier — and during the playoffs he had one of the best performances of his career with a 75 GA%-, leading the Colorado Avalanche to the championship. The following year, Roy was quite a bit better during the regular season (81 GA%-), ostensibly setting himself up for another strong playoff bid. So what happened next? Roy put up a terrible 110 GA%- in the playoffs, capped off by an embarrassing, season-ending loss to Detroit in which he allowed six goals before being pulled from the game in the second period. In the minuscule sample of the playoffs, even Hall of Famers are at the capricious whims of variance.It’s something to keep in mind during this year’s playoffs. Just as we found the correlation for regular-season GA%- to be quite low from year to year, the correlation between a goalie’s regular-season and his playoff GA%- is even smaller, as is the correlation between his previous career GA%- and playoff GA%-. We can’t predict who will fluctuate, just that somebody likely will.It’s not just goalies who are unpredictable; hockey’s stats holy war over shot quantity versus shot quality has shown us that an offense’s shooting percentage is just as inconsistent. The whole sport is especially vulnerable to random fluctuations, something that shows up most once a puck starts moving towards the net.In 2008, an Edmonton fan named Brian King left a comment at a now-defunct Oilers blog suggesting that the best way to understand luck in hockey was to look at a team’s shooting percentage on offense and the collective save percentage of its goalies. If you combined those stats for a team, and compared them to the league average, you could tell whether a team had been lucky or unlucky — and how far it had to go to regress to the league’s mean. The metric became known as PDO (its namesake was King’s online pseudonym), and the more it strays from its baseline of 1.00 (above 1 means lucky; below, unlucky), the more likely the team’s record, and even its goal differential, has been tainted by randomness.A great deal of recent hockey research has shown that, if given a large enough sample, every team’s PDO will more or less regress toward the league average of 1.00. The big implication of PDO is that a team has virtually no long-term control over shooting percentage, just like it can’t predict the efficacy of its goalie. The key to good defense, then, is simply to keep the opposition from shooting, because it’s impossible for a goalie to maintain a consistently high save percentage.PDO can now tell us the extent of hockey’s chaos, but goaltenders have always grappled with the randomness of their position. In his classic book, “The Game,” the Montreal Canadiens’ Hall of Fame goalie Ken Dryden wrote that a goaltender’s mental focus is key:If you were to ask a coach or a player what he would most like to see in a goalie, he would, after some rambling out-loud thoughts, probably settle on something like: consistency, dependability, and the ability to make the big save. [ … ] Because the demands on a goalie are mostly mental, it means that for a goalie the biggest enemy is himself. Not a puck, not an opponent, not a quirk of size or style. Him. The stress and anxiety he feels when he plays, the fear of failing, the fear of being embarrassed, the fear of being physically hurt, all are symptoms of his position, in constant ebb and flow, but never disappearing. The successful goalie understands these neuroses, accepts them, and puts them under control. The unsuccessful goalie is distracted by them, his mind in knots, his body quickly following.Stats portray the goalie’s job as a nihilistic one. Chaos mounts, pucks fly, muscles react. What happens beyond that is so random that, as Dryden writes, the only way for a goalie to cope is to focus on what’s immediately in front of him: a stretch of ice with an ever-changing landscape of variables.