Shot quality revisited: A look at the correlation between scoring chances and shot totals

Eric T.
June 26 2012 10:08AM

 

 

- graphic courtesy of The Russian Machine Never Breaks

 

For two years, my standard explanation for why we pay a lot of attention to shot differential (Corsi or Fenwick) is that it correlates strongly with scoring chances, zone time, and puck possession, and is the best predictor we have of future goal scoring. Each of those claims is based on seminal work by Vic Ferrari and JLikens.

The claim about scoring chances was based largely on this plot from Vic, showing a correlation between scoring chances and Fenwick of 0.92:

09-10 Oilers SC vs Fen

A comprehensive data review

If scoring chances correlate that closely to shot differential, then that would suggest that shot quality differences are negligible, that tracking the scoring chances isn't really adding much information. With more and more bloggers tracking scoring chances these days, we actually have a pretty large body of work to draw from; let's see whether this tight correlation is a general phenomenon.

I managed to dig up 18 teams' scoring chance data: The 2009-10 and 2010-11 Canadiens, Oilers, Maple Leafs, and Flames, and the 2011-12 Canucks, Stars, Flames, Hurricanes, Blue Jackets, Canadiens, Rangers (~3/4 season), Sharks, Capitals, and Flyers. (Thanks to Canucks Army, Defending Big D, Shutdown Line, Jackets Cannon, Leafs Nation, En Attendant les Nordiques, The Copper N Blue, Flames Nation, Blueshirt Banter, Fear the Fin, The Washington Post, and Broad Street Hockey for either publishing or emailing me their results.)

Across those 18 teams, the correlation between a player's shot differential and his scoring chance differential is about 0.80-0.85 (depending on whether you set the cutoff for games played at something like 20 or something like 70).

Correlation plot for scoring chances and shots

There is a strong correlation between shot differential and scoring chance differential, so the simple shot differential tells us most of what we need to know. And since others have shown this for a team here or there, this isn't news; the analytical community has long known that shot differential is very important and shot quality effects are minor.

The thing that surprised me is that the spread was actually larger than I had expected -- some players do lie a reasonable distance above or below the fit line, getting scoring chances at a rate like a player whose shot differential is a few percent better. Does that mean that we need the scoring chance totals to get the full story? Or is that just random noise?

Reproducible differences or random fluctuations?

To address that, we'll look at year-over-year correlations; if players are showing an actual talent in this area, we'll expect their numbers to be at least somewhat reproducible from year to year.

Dividing a player's scoring chance differential by his shot differential (SC% / Fenwick) gives us a metric assessing how much of a shot quality edge the team had with him on the ice that year. We can look at the same shot quality index for the following year and ask whether players who did well one year were likely to do well again the next year. With 105 players who had games tracked two years in a row, we should be able to tell whether this shot quality index is measuring a persistent talent.

Having lots of your shots be scoring chances is not a reproducible talent.

That's not much of a trend. With an R^2 of 0.049, the correlation is strong enough to be statistically significant, but weak enough to be insignificant in practice. If a guy is at 0.9 this year (one of the worst in the league), you can't even say that he'll be below average next year -- just that he won't be one of the very best.

Whatever tendency certain players might have for driving their team to get more scoring chances than a simple shot differential predicts is small and swamped by random noise. This suggests tracking scoring chances isn't adding much information to the readily available shot differential numbers.

Recently by Eric T.

2654ef2681c88bc3252431ec45e30590
Eric T. writes for NHL Numbers and Broad Street Hockey. His work generally focuses on analytical investigations and covers all phases of the game. You can find him on Twitter as @BSH_EricT.
Avatar
#1 Kent Wilson
June 26 2012, 10:10AM
Trash it!
3
trashes
Props
4
props

The chart showing the correlation between fenwick and scoring chances is rather definitive.

Avatar
#2 Jared Lunsford
June 26 2012, 11:08AM
Trash it!
2
trashes
Props
0
props

Great work, as always, Eric.

I hope instead of being discouraged or sticking to it, people shift their hard work to other areas that have shown promise like tracking zone entries and exits.

Avatar
#3 Khaos
June 26 2012, 01:45PM
Trash it!
1
trashes
Props
0
props

What do the special teams numbers look like? I feel that on the PP, teams that just shoot from anywhere (as every fan in the rink would like them to do) does worse in comparison to a team that sets up quality scoring chances.

Avatar
#4 JLikens
June 27 2012, 11:16PM
Trash it!
1
trashes
Props
0
props

Great work, Eric.

The next question is: Is there any effect at the team level?

My guess would be yes.

Avatar
#5 Ben Wendorf
June 26 2012, 10:29AM
Trash it!
0
trashes
Props
0
props

Honestly, this information on the one hand makes me feel good and the other pretty bad. You guys and others have put so much work into the scoring chance data; it's really been a remarkable thing. I guess the question is what are the suggestions going from here? Do we continue with recording chances, or do we move towards recording something else?

Avatar
#6 Derek Jedamski
June 26 2012, 10:36AM
Trash it!
0
trashes
Props
0
props

This is kind of depressing as somebody who was looking forward to beginning to track scoring chances next season. Interesting findings nonetheless, and more ammo to explain why shot differential is so key.

Avatar
#7 Kent Wilson
June 26 2012, 10:40AM
Trash it!
0
trashes
Props
0
props

@Ben Wendorf

We wouldn't have know this without tracking scoring chances, so it still proved to worthwhile.

And for the (large) proportion of the population that doesn't want to read about correlations and weirdly named stats, scoring chances are basically a more intuitive, more accessible version for them.

Avatar
#8 Phil S
June 26 2012, 10:42AM
Trash it!
0
trashes
Props
0
props

Hunter Hockey. Ugh.

Great work Eric. So many still don't want to believe shot quality is such a minor, minor influence on who wins games, but data like this just reinforces that the general media and fans are completely out to lunch on how to evaluate hockey.

Washington never should have changed their style. Alas.

Avatar
#9 draglikepull
June 26 2012, 11:05AM
Trash it!
0
trashes
Props
0
props

Here's something I've been wondering though:

We know that career SH% does show some difference in talent - it's not an accident that Franzen or Stamkos consistently scoring on a higher percentage of their shots than Mike Brown. So if all players get scoring chances on a roughly equivalent proportion of their shots, doesn't that suggest that shot quality is a fairly significant factor (ie. that some players taking the same shot from the same location are better than others)?

If the difference in SH% between Stamkos and Mike Brown isn't that Franzen gets a higher proportion of his shots as scoring chances, doesn't that mean that Stamkos is actually a better shooter?

Avatar
#11 Neil Greenberg
June 26 2012, 11:35AM
Trash it!
0
trashes
Props
0
props

This is bittersweet. For the past two years I know a LOT of work (by myself and others) has gone into this, and while it showed us something we prob wished wasn't the case, it also showed how valuable it is to have a grassroots effort like this.

I am definitely proud to be part of this effort.

Avatar
#13 Olivier
June 26 2012, 12:50PM
Trash it!
0
trashes
Props
0
props

So, what's the next big thing?

That zone entry thingy looks nice, no? A bit more work, tough. Jumping from 30-35 events a game to 180+ means a few adjustments, but still...

Avatar
#14 Olivier
June 26 2012, 12:53PM
Trash it!
0
trashes
Props
0
props

Also, I should add that I don't think I'll stop recording scoring chances. Whatever we think of it, "Shot Quality" is here to stay for a while and recording chances on top of Zone Entries really amounts to nothing.

And the whole Gomez saga taught me that scoring chances are useful when it comes to dealing with the dreaded crappy on-ice shooting % / shot quality combo.

Avatar
#16 Jonathan Willis
June 26 2012, 01:13PM
Trash it!
0
trashes
Props
0
props

@Olivier

Yeah, I think zone exits/entries are the place to go next.

Avatar
#18 Jim
June 26 2012, 02:08PM
Trash it!
0
trashes
Props
0
props

The year-to-year correlation plot is great. I saw one by Kanneman? (sp? - he won the nobel prize in economics a few years back) which showed a similar thing with fund managers in investment banks. They have no yearly correlations, i.e. it's all dumb luck.

I think that a lot of the "Shot Quality" debate could be defused with a change in vocabulary though. Everyone knows that the quality of an individual shot matters; ES sh% ~10% but a penalty shot/2on1 is ~30%, an empty net from 3ft should be 100%.. The important point is that the average distribution of crappy to good shots is constant and corsi/fenwick is a good proxy. I think fewer people would complain with the latter statement if they didn't get confused by the idea of the quality of a single shot.

Avatar
#19 danl1988
June 26 2012, 02:11PM
Trash it!
0
trashes
Props
0
props

Even knowing that quality makes such a small difference I still think it'd be quite interesting to see which players continually produce a SC%/Fenwick% greater than 1. It appears that some players can reproduce higher quality scoring chances for at least two straight seasons. I think it'd be interesting to see which players can replicate it over even longer stretches, even if there aren't that many of them.

As for where to go next, I think zone exits and entries would be awesome. Is it more effective to make a breakout pass or skate the puck up the ice? Use the boards or go up the middle? Skate the puck in vs. dump and chase? What portion of scoring chances come from each of these methods? Which players do each of these things best/worst? There's lots of potential there.

Avatar
#20 David Johnson
June 26 2012, 02:11PM
Trash it!
0
trashes
Props
0
props

@Eric T.

"There's no question that some players have individual talent for getting their own shots from more dangerous locations.

So the question is why that doesn't show up in their on-ice totals."

It does. It just can't be seen using the small data sets most people use (i.e. one year vs one year) but use a larger data set and on-ice shooting percentage can be shown to be a repeatable talent. I have shown that the r^2 between a players 2007-10 3 year on-ice shooting percentage and their 2010-12 2 year on-ice shooting percentage is 0.2854 (see http://hockeyanalysis.com/tag/shooting-percentage/). In that same post I show that 2007-10 shooting percentage predicts future goal scoring as well as 2007-10 fenwick for rates.

Avatar
#21 Tach
June 26 2012, 02:20PM
Trash it!
0
trashes
Props
0
props

Eric,

I think your conversion rate measure of SC%/Fenwick% might be a bit flawed for assessing the relationship you are looking for. There are too many conflating factors at play.

The measure you are using is a ratio between two ratios. Really it is SC For/SC Against/Fen For/Fen Against.

But how much control does one player have over all of these events, particularly over two seasons? A player could be demonstrating a consistently high rate of SC For/Fen For but the ratio doesn't demonstrate inter-year correlation because of changes in SC Against or Fen Against.

If you have raw SC For and Fen For for each year, I would be very curious what the interyear correlation is for that ratio, particularly for forwards. That would tell us if players are able to turn more positive possession events into scoring chances.

Avatar
#24 David Johnson
June 26 2012, 02:57PM
Trash it!
0
trashes
Props
0
props

@Eric T.

Not sure about 1, but I do believe 3 is definitely a factor. Top offensive players often have weak defensive stats (including shooting percentage against) and top defensive players often have weak offensive stats (including shooting percentage).

As Tach points out, splitting out scoring chances for and against and looking at them separately may tell us something. But, as you point out, scoring chances are somewhat subjective so this may lead to scorer bias. Still, it may be worth looking into.

Ultimately I think there is a fairly fine line between converting shots/chances into goals and the difference between converting and not converting can come down to just a slightly better timed or accurate pass or a slightly harder or more accurate shot. Two plays may be almost identical scoring chances but the players that pass and shoot quicker, harder and more accurately will more likely get the goal. Until we can accurately define and record scoring chances that take this into account, I am not sure scoring chance data will be a significant improvement on shot data.

Avatar
#25 John Fischer
June 26 2012, 03:58PM
Trash it!
0
trashes
Props
0
props

I think it's fantastic that there is evidence that a correlation exist.

As a consumer of the data, I'd like to chime in and say I think recording scoring chances is very valuable at the game level. It provides further context on how a player performed on a given night than just their shot count, number of attempts, or Corsi. I use the prior three in recaps, but chances are the next step beyond. I personally think that knowing how many chances actually generated by the player has some value, even if Fenwick is a good estimator in the long term. Ideally, I'd like to see totals like the QMJHL does with "Dangerous Shots." So from my perspective, I hope the project doesn't end. For the long term, I would think having more than 2 seasons worth of data and more teams involved can help confirm the extent of the correlation.

Avatar
#26 George E. Ays
June 26 2012, 07:08PM
Trash it!
0
trashes
Props
0
props

I think the differences that appear in career shooting% are in fact coming from differences within the chance area.

We now know that shot differential and scoring chances correlate strongly, but location within the scoring chance zone still has a lot of variability. We also intuitively know that a scoring chance resulting from an odd-man transition is generally more dangerous than one resulting from a forecheck. (and I do have a little evidence of that as well, http://www.blueshirtbanter.com/2012/1/16/2706434/midseason-ny-rangers-stats-scoring-chance-types)

I think that's the direction the project could go next, getting more detail about each chance rather than recording that they happened. Whether it is a fruitful endeavor is another thing, but that's my two cents.

Avatar
#27 matt
June 26 2012, 07:29PM
Trash it!
0
trashes
Props
0
props

Hi, Ive followed this blog forever and ive recently been inspired to start out my own blog! If you could put it in your blog roll to help me get started it would mean the world to me. The link is: https://fiveholegrinders.wordpress.com/ Thanks and keep up the amazing work!

Avatar
#28 gongshow
June 26 2012, 11:41PM
Trash it!
0
trashes
Props
0
props

Thanks for all of your combined efforts on this project.

Avatar
#29 Kent Wilson
June 27 2012, 12:34AM
Trash it!
0
trashes
Props
1
props

Glad to see so much interest in this stuff.

Next steps wouldn't just be recording more events - it would compiling them in accessible, searchable databases I'd say.

Avatar
#30 Tach
June 27 2012, 08:29AM
Trash it!
0
trashes
Props
0
props

@Eric T.

Eric,

Not trying to be dense, but what do you mean by the scoring chances didn't add value? I get that the scoring chance ratio correlated highly with the shot metric possession ratio. But we also know there were players in your upper right quadrant of your SC ratio/Fenwick ratio that had year over year ability to translate possession to scoring chances more efficiently, and the bottom left quadrant vice versa. The problem is the top left and bottom right data where the ratio change season to season.

And I also take your point on data quality, but by your own admission we have the same challenge in shot data and we still use it.

Avatar
#32 Tach
June 27 2012, 02:03PM
Trash it!
0
trashes
Props
0
props

'If half of the coins that were heads this year come up tails next year, you don't point to the half that were heads two years in a row and say "we've found some loaded coins."'

I agree, but if I suspected the coins were loaded I might flip them X number of times again and see if the same ones didn't come up heads some number of times that would give a statistically signficant result. Not just give up after two trials and say "No correlation to see here."

While I appreciate there is a time cost to investigating this stuff, I think this is more a data issue than an actual description issue.

Do you really, truly believe that no two players, given the exact same amount of offensive possession, against the exact same quality of opponent and teammates, have distinguishably different abilities to have shots derive from positions on the ice more likely to score?

To put it another way, if I gave Wayne Gretzkey and Raitis Ivanans the same 4 teammates, against the same 5 defenders and goalie, and gave them 100, or 500, or 1000 offensive zone possessions, you think that each team would score the same (within statistical significance) number of goals? You think the shots would come from (on a statistically signficant average basis) the same place on the ice?

I am fairly confident they would not, and it is only our lack of this type of data that prevents us from measuring the scope of the effect.

Avatar
#34 Tach
June 27 2012, 06:53PM
Trash it!
0
trashes
Props
0
props

While I still think there may be useful data in Scoring chance per shot attempt data I totally agree with this statement:

"And I'm thinking that while there is in fact a statistically significant difference from player to player, it is small enough at the 1-2 year level that the thousands of hours being put into collecting scoring chance data are resulting in negligible corrections to the freely available shot data; that if we're going to try to correct for shot location at all, we should do it with the published shot location data, which while often imprecise and having limited predictive value, is probably still better than a binary yes/no on whether a given shot is a scoring chance."

Maybe we should turn the scoring chance counting hours into a letter writing campaign to get the NHL to buy the shot data recorders some new glasses.

Thanks for a frank and useful discussion.

Avatar
#35 David Johnson
June 28 2012, 06:27AM
Trash it!
0
trashes
Props
0
props

@JLikens

I suspect there won't be an effect at the team level either. If the individual components (players) aren't affected, it's unlikely the sum of the components will be.

Avatar
#36 dan
June 28 2012, 08:03AM
Trash it!
0
trashes
Props
0
props

1. Great work Eric!

? wouldn't 'score effects' be an issue... do we need to adj. for them to say for certain there is negligible effect?

2. Great work by all those who collected Scoring chances. 3. I encourage all recorders to continue...I will be doing Nux chances myself this year.

IMO - There are still many legitimate reasons to record Scoring chances

a) to further the understanding of the game - generating scoring chances are the primary purpose of the game/teams

b) it will help us with goaltending evaluation I plan a post on this soon

c) what I would like to see in the future is a movement in collecting chances and other stats is to separate - luck from skill/

example: there are three ways a player/team get a chance 1. generated by skill of team (skill) 2. unforced error - received by mistake 3. lucky bounce (luck) because we know luck is such a huge part of hockey its time we design all our stats to eliminate this bias as much as possible!

Comments are closed for this article.