November 09 2012 07:40AM
I love what the #fancystats community have done, lots of useful stuff, but confidence in their goalie and prospect analysis a bit too high.
@steffeG @Sens_Army_ @NHLnumbers not in that library are any columns pointing to the limits of prospect analysis.
I asked both of them to explain further. (I also defended myself -- my Tweets announcing both of those articles made light of how iffy the stats are in those areas, so I think it was clear that I know there are limitations.) The essence of their feedback, as I understood it, was that by not actively discussing the uncertainty, we as a community have implied that we know more than we do.
This strikes me as a fair criticism, at least in part. It seems like half the articles in the goalie section talk about how unpredictable goalies seem to be, so I'm not sure I'd accept the critique there. But it's pretty rare for us to put actual error bars on our projections. For prospects in particular, there have been a lot of articles written where we give league translation factors to two decimal places; I am pretty certain that the authors did not mean to imply that we can project results to within 1%, but we haven't explicitly laid that out for people.
Can we actually estimate the uncertainty on those projections?
Calculating the translation factor
The league equivalencies are normally calculated by looking at players who changed leagues from one year to the next and how much their point total went up or down. We could simply look at the spread in those results as an estimate of the error in our translation model. However, that would be misleading.
Players' point totals fluctuate up and down all the time; from year to year, there could be large differences in their usage, their teammates, their nagging injuries, or simply how many bounces they got -- not to mention whether one 18-year-old grows more than another. If we just look at the spread in our numbers, we would be attributing all of those sources of error to uncertainty in the league translation factor, which is clearly wrong.
Instead, we'll do a bit of simple analysis. Here is a plot of the performance of every forward who played at least 40 NHL games in a season by age 22 between 2005-06 and 2011-12 and how they did the previous year in the AHL:
Here, we see a translation factor of 0.55 for these players. This is a bit lower than the ~0.47 previously calculated for players in this age range back in 2004, so perhaps the AHL has gotten stronger.
Estimating the uncertainty
However, with only 88 players included in this study, there is considerable uncertainty in the slope of that best-fit line. We just haven't sampled enough players to nail things down precisely -- it's entirely possible that we just happen to be in a little bubble where some players happened to make the transition well or have good shooting luck in their first NHL season or whatever.
When we calculate the confidence interval for this analysis, we find that there is a 10% chance that if we had a zillion players to analyze, the true conversion factor would turn out to be greater than 0.66 or less than 0.42.
I believe that 0.55 represents a better guess for the current translation factor for 18-21 year olds than 0.47, that it's more likely the difference is from changes in league strength than random noise. However, the error in this estimate is large enough that either answer could be right -- the conversion factor should probably be described as 0.55 +/- 0.1
It's worth emphasizing here that this is an estimate of the error on our conversion factor, not an error on our projection. If you look at the players who averaged a point per game in the AHL, you will see that they fall in a much wider NHL band than 0.55 +/- 0.1. That is, 0.1 is the error on the slope of the best-fit line, not a measure of how far from that line the average player might fall.
The latter comes from the correlation coefficient. With R^2 = 0.38, we can say that about 38% of the spread between these players' NHL results is accounted for by looking at their AHL results. The other 62% would come from all of those other factors that might affect the players' scoring: usage, teammates, luck, etc; those are the things that pull any given player away from our fit line.
In a simple NHL equivalency projection that doesn't account for any of those factors, about 2/3 of players who score a point per game in the AHL would be expected to fall within the range 0.35-0.77 in the NHL the following season. This is obviously an enormous range, but what fraction of that spread comes from error in the translation factor itself?
Where the errors come from
We can isolate the translation errors by comparing this outcome to what happens for players the same age who did not change leagues. Here are the players who were in the NHL two years in a row at this age:
In this case, we see a slight upward slope -- on average, the players' point total improves by about 8% from year to year at this age.
We might also expect that an NHL season would be a better predictor of NHL point total than an AHL season would be. After all, not only is it a whole different level of competition, but things like a change in how much ice time a player gets are likely to be larger when you switch leagues than when you play a second year in the same league. And indeed the correlation is a bit stronger, but just a bit -- those factors not included in the model (plus luck) account for 53% of the difference between players here, compared to 62% in the AHL-to-NHL translation.
In practice, the difference is scarcely perceptible. Let's imagine three groups of players who have the same 0.5 points per game projection for this year:
- Group one is a bunch of young guys who each had 0.91 points per game in the AHL last year
- Group two is a bunch of young guys who each had 0.46 points per game in the NHL last year
- Group three is a bunch of veterans at their peak (age 24-28) who each had 0.50 points per game in the NHL last year
|AHL kids||NHL kids||NHL vets|
|90th percentile outcome||0.85 PPG||0.88 PPG||0.78 PPG|
|70th percentile outcome||0.65 PPG||0.61 PPG||0.62 PPG|
|Median outcome||0.50 PPG||0.53 PPG||0.50 PPG|
|30th percentile outcome||0.40 PPG||0.43 PPG||0.41 PPG|
|10th percentile outcome||0.26 PPG||0.32 PPG||0.30 PPG|
The AHL group spreads out very slightly more than the NHL kids, but the difference is probably imperceptible in practice. When any individual player goes over or under their NHL equivalency projection, just a sliver of that is due to the unpredictability of making the transition to the NHL.
Moreover, comparing the young players to the veterans shows that very little of it is due to the unpredictability of a player's development. The lion's share of the error just comes from luck and factors like ice time.
Simply looking at a player's point total in the previous season isn't a particularly precise way to estimate how many points a player will score, regardless of what stage of his career he's in. But using NHL equivalencies to convert a point total from the AHL doesn't seem like it adds appreciably to the challenges of making such a forecast.
Previously by Eric T.
- Review: Zone start adjustments to shot differential
- Stats article reference library
- Stats are ruining hockey
- Impact of changing teams on a player's point total
- NHL lockout: Breaking down a new CBA proposal