Putting error bars on NHL equivalencies
Eric T.
November 09 2012 07:40AM
After I posted the prospects wing of our reference library, I got a couple of interesting replies on Twitter:
I love what the #fancystats community have done, lots of useful stuff, but confidence in their goalie and prospect analysis a bit too high.
@steffeG @Sens_Army_ @NHLnumbers not in that library are any columns pointing to the limits of prospect analysis.
I asked both of them to explain further. (I also defended myself -- my Tweets announcing both of those articles made light of how iffy the stats are in those areas, so I think it was clear that I know there are limitations.) The essence of their feedback, as I understood it, was that by not actively discussing the uncertainty, we as a community have implied that we know more than we do.
This strikes me as a fair criticism, at least in part. It seems like half the articles in the goalie section talk about how unpredictable goalies seem to be, so I'm not sure I'd accept the critique there. But it's pretty rare for us to put actual error bars on our projections. For prospects in particular, there have been a lot of articles written where we give league translation factors to two decimal places; I am pretty certain that the authors did not mean to imply that we can project results to within 1%, but we haven't explicitly laid that out for people.
Can we actually estimate the uncertainty on those projections?
Calculating the translation factor
The league equivalencies are normally calculated by looking at players who changed leagues from one year to the next and how much their point total went up or down. We could simply look at the spread in those results as an estimate of the error in our translation model. However, that would be misleading.
Players' point totals fluctuate up and down all the time; from year to year, there could be large differences in their usage, their teammates, their nagging injuries, or simply how many bounces they got -- not to mention whether one 18-year-old grows more than another. If we just look at the spread in our numbers, we would be attributing all of those sources of error to uncertainty in the league translation factor, which is clearly wrong.
Instead, we'll do a bit of simple analysis. Here is a plot of the performance of every forward who played at least 40 NHL games in a season by age 22 between 2005-06 and 2011-12 and how they did the previous year in the AHL:
Here, we see a translation factor of 0.55 for these players. This is a bit lower than the ~0.47 previously calculated for players in this age range back in 2004, so perhaps the AHL has gotten stronger.
Estimating the uncertainty
However, with only 88 players included in this study, there is considerable uncertainty in the slope of that best-fit line. We just haven't sampled enough players to nail things down precisely -- it's entirely possible that we just happen to be in a little bubble where some players happened to make the transition well or have good shooting luck in their first NHL season or whatever.
When we calculate the confidence interval for this analysis, we find that there is a 10% chance that if we had a zillion players to analyze, the true conversion factor would turn out to be greater than 0.66 or less than 0.42.
I believe that 0.55 represents a better guess for the current translation factor for 18-21 year olds than 0.47, that it's more likely the difference is from changes in league strength than random noise. However, the error in this estimate is large enough that either answer could be right -- the conversion factor should probably be described as 0.55 +/- 0.1
It's worth emphasizing here that this is an estimate of the error on our conversion factor, not an error on our projection. If you look at the players who averaged a point per game in the AHL, you will see that they fall in a much wider NHL band than 0.55 +/- 0.1. That is, 0.1 is the error on the slope of the best-fit line, not a measure of how far from that line the average player might fall.
The latter comes from the correlation coefficient. With R^2 = 0.38, we can say that about 38% of the spread between these players' NHL results is accounted for by looking at their AHL results. The other 62% would come from all of those other factors that might affect the players' scoring: usage, teammates, luck, etc; those are the things that pull any given player away from our fit line.
In a simple NHL equivalency projection that doesn't account for any of those factors, about 2/3 of players who score a point per game in the AHL would be expected to fall within the range 0.35-0.77 in the NHL the following season. This is obviously an enormous range, but what fraction of that spread comes from error in the translation factor itself?
Where the errors come from
We can isolate the translation errors by comparing this outcome to what happens for players the same age who did not change leagues. Here are the players who were in the NHL two years in a row at this age:
In this case, we see a slight upward slope -- on average, the players' point total improves by about 8% from year to year at this age.
We might also expect that an NHL season would be a better predictor of NHL point total than an AHL season would be. After all, not only is it a whole different level of competition, but things like a change in how much ice time a player gets are likely to be larger when you switch leagues than when you play a second year in the same league. And indeed the correlation is a bit stronger, but just a bit -- those factors not included in the model (plus luck) account for 53% of the difference between players here, compared to 62% in the AHL-to-NHL translation.
In practice, the difference is scarcely perceptible. Let's imagine three groups of players who have the same 0.5 points per game projection for this year:
- Group one is a bunch of young guys who each had 0.91 points per game in the AHL last year
- Group two is a bunch of young guys who each had 0.46 points per game in the NHL last year
- Group three is a bunch of veterans at their peak (age 24-28) who each had 0.50 points per game in the NHL last year
AHL kids | NHL kids | NHL vets | |
---|---|---|---|
90th percentile outcome | 0.85 PPG | 0.88 PPG | 0.78 PPG |
70th percentile outcome | 0.65 PPG | 0.61 PPG | 0.62 PPG |
Median outcome | 0.50 PPG | 0.53 PPG | 0.50 PPG |
30th percentile outcome | 0.40 PPG | 0.43 PPG | 0.41 PPG |
10th percentile outcome | 0.26 PPG | 0.32 PPG | 0.30 PPG |
The AHL group spreads out very slightly more than the NHL kids, but the difference is probably imperceptible in practice. When any individual player goes over or under their NHL equivalency projection, just a sliver of that is due to the unpredictability of making the transition to the NHL.
Moreover, comparing the young players to the veterans shows that very little of it is due to the unpredictability of a player's development. The lion's share of the error just comes from luck and factors like ice time.
Simply looking at a player's point total in the previous season isn't a particularly precise way to estimate how many points a player will score, regardless of what stage of his career he's in. But using NHL equivalencies to convert a point total from the AHL doesn't seem like it adds appreciably to the challenges of making such a forecast.
Previously by Eric T.
- Review: Zone start adjustments to shot differential
- Stats article reference library
- Stats are ruining hockey
- Impact of changing teams on a player's point total
- NHL lockout: Breaking down a new CBA proposal
November 09 2012, 10:30AM
Great stuff Eric.
As you point out here, the greatest uncertainty is what happens to the guy when he gets promoted - who does he play with, whats his role and how do the bounces go?
For me, equivalencies are less about projecting a prospects outcomes in the NHL, because I am always acutely aware of the potential confounding variables. It's more a "correction factor" that helps me compare kids across leagues when they are in junior, college, SEL etc.
On another note, with your finding re: AHL translation factor, perhaps we should re-run NHLE for the various leagues again. Sounds like they may need an update.
November 09 2012, 10:52AM
Great work, Eric. I think comparing players making the AHL-NHL transition to players of the same age playing NHL games is a creative way to strip out factors other than league adjustment. A couple of questions I have:
Do you think the fact that the NHL-NHL group is likely to be more talented than the AHL-NHL group should have any impact on the analysis?
How many players did you have in each group?
November 09 2012, 11:46AM
@Kent Wilson
I went into this planning to do that (which is why it covers age 18-22 even though there aren't 18 year olds in the AHL). The problem is one of sample size -- if the error on the AHL translation using seven years of data was +/- 0.1, it'd be a lot larger on any other league.
I could do one for Canadian major juniors as a whole (neglecting differences between the leagues), but for anything else, we'll need to look at a lot more than seven years and therefore wouldn't really capture the changes over time and probably wouldn't see much difference from the existing numbers.
November 09 2012, 11:54AM
@Scott Reynolds
There's no question that the NHL-NHL group is more talented -- they averaged over 0.5 PPG in the NHL, while the AHL-NHL group averaged under 0.4 PPG.
While it's certainly possible that that could have some effect, it's not obvious to me that it would; I'm not even sure which way such an effect would go. (My guess, I suppose, would be that the NHL-NHL players would fall in a narrower band of TOI and therefore see less variance -- the majority of the NHL-NHL players probably stay on the same line or maybe move up a line, whereas the AHL-NHL players include a wide spread from a 1st-to-1st transition to a 2nd-to-4th transition.)
I had 129 NHL->NHL guys and 88 AHL->NHL guys in my sample.
November 11 2012, 05:00PM
@Eric T.
The only way you're going to find out the equivalencies of European leagues is to cross reference between people that switch from one Euro league to another, like Kent did for the Allsvenksan to Elitserien. You increase the sample by expanding the scope of the search.
Also, always, ALWAYS include playoff scoring on top of regular season. Increase the games played sample whenever you can.