Shot quality matters, but how much?
Eric T.
July 03 2012 12:18PM
Last week, I delved into the always-controversial realm of shot quality, looking at how tight the link between scoring chances and shot differential is. In the comments there and in some subsequent conversations, I was reminded that the current understanding of shot quality is often oversimplified and misrepresented.
I don't think anyone believes that there is no such thing as shot quality, that all shots are equal. The argument is actually that most shot quality effects are smaller than people think, and that over the sample sizes we normally work with, differences in shot quality tend to be dominated more by noise than talent. In this article, I'll walk through some examples and try to emphasize the subtle difference between saying shot quality is meaningless and saying it is often negligible.
Team-level shooting percentage
Even after 82 games, about 2/3 of the shooting percentage differences between teams comes from simple variance. The result of this is that when a team leads the league with a 8.86% shooting percentage at 5-on-5 (0.74% above average), the best guess at their long-run true talent is only 8.32% (0.20% above average). Most of the difference you see between teams in a season is just chance.
- The point is not: "All teams shoot for the same percentage, so ignore shooting percentages completely."
- The point is: "Differences in shooting percentage are small and require a very large data set to overcome noise, so you won't be wrong by much if you ignore them."
Individual shooting percentage
Some players definitely shoot for a higher percentage than others. This appears to be driven as much by where they shoot as from how well they shoot, but shooting locations are much more reproducible. In fact, people's shooting skill is so transient that only Ilya Kovalchuk and Alex Tanguay can be unambiguously identified as being good shooters. The result is that players' shooting percentages vary a lot from year to year, and you should look at several years' stats to estimate how a player will shoot going forwards.
- The point is not: "There's no such thing as a good shooter, so ignore shooting talent completely."
- The point is: "You can tell shot location skill quickly, but differences in shooting talent are significant and require a very large data set, so look at several years of shooting percentages to make your predictions."
Linemate shooting percentage
Some players have a higher on-ice shooting percentage than others. (On-ice shooting percentage is the team's shooting percentage with that player on the ice.) Even including the player's own shooting percentage, four years of data only identifies 10% of players as having unambiguously high or low on-ice shooting percentages (see comments section). We already said that some players have higher shooting percentages than others, so remove the players' own shots and the spread shrinks. Remove the tendency of good shooters to play together, and it looks like over multiple seasons we see only a few top playmakers improving their teammates' shooting.
- The point is not: "There's no such thing as a good passer, so ignore linemates completely."
- The point is: "The ability to improve teammates' shooting is small and requires a very large data set to overcome noise, so you won't be wrong by much if you ignore it."
Save percentage
The average change in a goalie's save percentage from year to year when he stays with the same team is just 0.0005 larger than simple random chance would predict, and when a goalie changes teams the sv% difference is just 0.0011 larger than random chance. The best shot-quality-influencing system of this era (Jacques Lemaire's) reduced Fenwick shooting percentages by about 0.0015. The result is that any team effect on a goalie's save percentage doesn't add up to more than a goal or two per season.
- The point is not: "All teams face the same shots, so ignore shot quality completely."
- The point is: "Differences in shot locations are small and require a very large data set to overcome noise, so you won't be wrong by much if you ignore them."
Conclusion
Not everyone agrees with what I've laid out here, but I've tried to capture what the majority of the leading analysts believe, emphasizing the nuanced distinctions that we usually skip for the convenience of writing readable prose. David Johnson's work is a good place to start if you're looking for the dissenting viewpoint, focusing much more on goals than shots. But as Hawerchuk has summarized:
Together, Fenwick/Corsi and Luck account for around 3/4 of team winning percentage. What's the remainder? Goaltending talent - which Tom Awad estimates at about 5% - and special teams, along with a very small sliver that's due to shooting talent and the oft-mentioned "shot quality."
In general, shot quality factors tend to be small enough that they don't grossly alter our understanding of the game, and they tend to be swamped by noise during in-season analysis. The best possible understanding obviously requires more than a cursory glance at shot totals, but shot-based analysis has consistently proven to be a strong approach to identifying talent and predicting outcomes.
Recently by Eric T.:
@Oilanderp
Ah, sorry, that wasn't clear. In this context, I mean statistical noise -- random fluctuations up and down away from what the true long-term talent is.
I'm a 50% coin-flipper. If I flip a coin a million times, I'll get heads somewhere around a half-million times; our measurement of my skill will end up really close to 50% (give or take a few hundredths of a percent). If I flip a thousand times, I'll get heads somewhere around five-hundred times, but it might be 520 or 490 -- we'll measure my skill at 50% give or take a percent or two. If I flip ten times, I might get three heads or I might get eight, and our measurement of my skill will be way off.
If we're trying to tell the difference between a 60% coin flipper and a 40% coin flipper, a couple hundred flips is probably plenty. This is the situation for shot locations.
If we're trying to tell the difference between a 52% coin flipper and a 48% coin flipper, we need a couple thousand flips. This is the situation for shooting ability.
If we're trying to tell the difference between a 50.5% coin flipper and a 49.5% coin flipper, we need several thousand shots. This is the case for goalie save percentages.
Statistical noise is the extent that a goalie's save percentage (for example) will vary up and down over time just because of this simple random chance. Because we're looking for very small differences in our measurement, we need a very large sample to get the noise to average out to where we can tell how good someone really is.
I think we can add Steven Stamkos to the small list of true talent shooters along with Kovalchuk and Tanguay.
Great stuff, especially the links added in to the older articles.
@draglikepull
Yeah, my source there was a couple of years old. An updated look (with the resulting larger data set) probably gives us a few more names. But whether it's two or three or ten, the point is it's countably few.
I am unclear as to what you mean by 'noise'.
~If I scream really loudly from the stands will that make Kovalchuk miss?~
Seriously though can you talk a bit about what you mean by noise in this article?
As the 'dissenting viewpoint" let me make a few comments.
1. I haven't spent much time looking at team data but when I did a few years ago I did find pretty much the same evidence that Gabe did. Corsi/Fenwick/Shot differentials explained winning percentage in a significant way. So, on the team level, I am willing to grant you that corsi is significant.
2. At the on-ice level, my observations are clearly different. Players can drive on-ice shooting percentage and to some extent it really is irrelevant how (i.e. because they are great shooters, because they are great passers and improve line mates shooting percentage that way, or some other factor). It's nice to know the how, but simply knowing they can makes taking it into account important.
How much does on-ice shooting percentage matter? Well, I took all forwards with 2000 minutes of 5v5 zone start adjusted ice time over the past 5 seasons. There are 337 such players, so approximately 11 per team so we are pushing into 4th liners. The average fenwick shooting percentage among these players is 6.34% with the standard deviation of 0.9518%. The average fenwick for per 20 minutes (FF20) of these players is 12.81 fenwick for events per 20 minutes with a standard deviation of 1.12. The average goals for per 20 minutes is 0.813.
Now, if we take the FSh% standard deviation and multiply it by the average FF20 we get 0.122 which means one standard deviation in FSh% equates to 0.122 GA20. If we take the FF20 standard deviation and multiply it by the average FSh% we get 0.071 which means on standard deviation in FF20 equates to 0.071 GF20. Thus one can conclude variances in FSh% have a 72% larger impact on GF20 than variances in FF20 do.
BTW, the average player was on the ice for ~2250 fenwick for events so we aren't talking small sample sizes. The minimum number of fenwick for events is 981.
If we limit the sample group to players with over 2000 fenwick for events we find that one standard deviation in FSh% has a 0.113 impact on GF20 and one standard deviation of FF20 only has a 0.063 impact on GF20. In this case FSh% has a 79% greater influence on GF20.
I have also shown that when running 3 year (2007-08 to 2009-10) Sh% is an equally good predictor of future 2 year (2010-11 to 2011-12) GF20 as 3 year FF20 is. This tells me that 3 year shooting percentage is an equally good evaluator of offensive talent as 3 year FF20 is. It should also be noted that 3 year GF20 is a far better predictor of future 2 year GF20 than 3 year FF20 is (see http://hockeyanalysis.com/2012/04/19/on-ice-shooting-percentage-is-sustainable/). Thus the better evaluator of a players offensive talent is 3 year GF20, not 3 year FF20 or 3 year Sh%.
The question that arises from this is why the difference between team level observations and on-ice level observations. Why is shooting percentage significant (if you believe what I just wrote) at the on-ice level and not significant at the team level. I have a few theories on this.
1. The best offensive players are the players who drive shooting percentage (just look at the players that top the 5 year shooting percentage list: http://stats.hockeyanalysis.com/ratings.php?disp=1&db=200712&sit=f10&pos=forwards&minutes=3000&type=goal&sort=ShPct&sortdir=DESC). The best offensive players also get paid the most money and since teams have limited budgets teams can't stock up on high shooting percentage players.
2. High shooting percentage players often achieve high shooting percentage because they take risks offensively as many also have low on-ice save percentages while the reverse is true for defensive players who don't generate high shooting percentages. Because teams generally build a balanced team that includes a mix of offensive high risk high reward players and defensive low risk low reward players team shooting percentages get driven towards a mean.
My guess is it is some combination of the two factors.
So that is my "dissenting viewpoint".
Interesting well written article Eric, and very strong comments by D. Johnston.
Don't strong score effects also play a role in balancing out noticeable differences?
Interesting well written article Eric, and very strong comments by D. Johnston.
Don't strong score effects also play a role in balancing out noticeable differences?
What role, if any, do you see league competitive parity play in these results?
Would shot quality increase in significance if the NHL President trophy winners played in say, the ECHL for a year or so?
Is league parity responsible for the effective insignificance of shot quality as a predictor of anything?
Thanks Eric. Great post.
I've been looking to see if shot trajectory/location/targeting has any effect on shooting %age, but I've come up dry. I'm pretty sure that I understand the gist of your post, but, do you know of anyone who has looked at where the shot is headed and the effect that it has on %ages? I realize that this gets to be much more difficult to quantify, but just wondering if there's any data available.
Again, thanks to you and all of the other NHLnumbers writers.
"Even including the player's own shooting percentage, four years of data only identifies 10% of players as having unambiguously high or low on-ice shooting percentages (see comments section)."
Where do you come up with this comment? Based on the fact that 10% of the players studied aren't 95% sure to have an average ONSH% outside the 7%-8.5% range? Not sure why this range is at all significant?
The average ONSH% for guys who have played a minimum of 10 games has been 7.9% for the last 5 years. What should be looked at is a one tailed test that a players average shooting% is greater or less than the average.
Looking at only guys that have all 5 qualifying seasons (366 players), I come up with 120 players with 95% probability of having above average ONSH%, and 34 having below average ONSH%. That would be ~40% of players significantly away from average.
@TrentonL
Yeah, fair enough. I think it's two different ways of expressing the same thing, but it's worth making it clear what my statement is supposed to express.
You're right that a lot of players can be confidently identified as above or below average. But not a lot of players can be confidently identified as being well above or well below average -- and the point of this article is that the corrections, while nonzero, tend to be small.
So what I'm looking for are players who are well outside the normal range. Merely being confident that a player is above average -- before we correct for the very significant influence of his own individual shooting percentage and his tendency to play with other good shooters -- isn't enough to make me feel like I need to correct for his tendency to elevate his linemates' shooting.
@gongshow
It's been done a little bit (http://www.arcticicehockey.com/2011/8/22/2374005/shot-target-visualization for example), but generally the data we have is pretty limited.