July 29 2016 10:52AM
Graphic by Matt Henderson: @mhenderson95. The best!
Who doesn't love a good discussion? It's been quite a while since we've done a roundtable at NHLnumbers, so we figured we'd go big for our return to the game.Today, we focus on a quite broad topic: The state of hockey analytics, and their place in the modern game.
We've scoured the web for some of the brightest minds in hockey and looked to grab a number of differing opinions about the sport we know and love from people in a number of different backgrounds, who've all worked in the game in one way or another.
Heads up before you go any further! It's about 10,000 words long!
(But it's well worth the read.)
If you must, proceed forward and see what our panel has to say.
Manny Perry (MP1) @MannyElk
Catherine Silverman (CS) @CataCarryOn
For the last two years, I have covered the Arizona Coyotes and the NHL at large as credentialed media for Today's Slapshot, the Nation Network, and InGoal Magazine. I also work as a coach and features blogger for the Arizona Coyotes in the Department of Hockey Development, teaching Arizona youth how to play hockey and sharing stories from the community through the organization.
Scott Cullen (SC) @TSNScottCullen
I'm the fantasy sports and analytics writer for TSN.ca, covering hockey, football, baseball and basketball, but hockey gets the bulk of my attention. I played hockey and majored in Economics in university (Wilfrid Laurier), which is a little different background than most sports writers.
Prashanth Iyer (PI) @Iyer_Prashanth
I'm a long-time Red Wings fan, having spent the better part of my childhood living outside of Detroit. I graduated from the University of North Carolina at Chapel Hill in 2015 with his doctorate in pharmacy and is finishing his cardiology residency at UNC Medical Center. You can find my previous work and Red Wings-focused work over at Winging It In Motown. I'm always willing to spend the time to explain concepts, so please do not hesitate to ask!
Carolyn Wilke (CW) @Classlicity
I'm the Managing Editor for Today's Slapshot, an unabashed fan of non-traditional markets and various baked goods. Best known for my work on evaluating player salaries and contracts with respect to productivity.
Dom Luszczyszyn (DL) @omgitsdomi
I write about hockey stuff occasionally for The Hockey News, The Leafs Nation and Hockey Graphs. Pretty much everything I know, I've learned from smarter people on hockey Twitter or on my own through random googling.
I'm a freelance writer, hockey enthusiast, wannabe hockey player. My work is Blue Jackets focused, on the team website and BuckeyeStateHockey.com. I love to make puns – they are mostly bad. When I’m not talking about hockey, or thinking about hockey, I’m on my bike, traveling or enjoying good food. / CV:
Matt Pfeffer (MP2) @MattPfefferHky
My name's Matt Pfeffer. 54th most influential person in hockey according to The Hockey News.
(Editor's note: You may remember Pfeffer from his role in a recent trade in the NHL.)
Tom Hunter (TH) @PuckDontLie
I love hockey. I sometimes write about hockey. More? Um... I often gravitate towards junior hockey more so than the NHL. I fell in love with the OHL during high school when I started working for the St. Michael's Majors (before Melnyk ruined the franchise by moving them to Mississauga). A job with the Guelph Storm in university gave me another perspective and an appreciation for who two teams with the same goal, can operate very differently.
The most fun thing I've experienced in a long time is teaching my 3-year old about hockey this past season. She sees the game with an innocence that is completely lost on most of us and it reminds me that at the end of the day, for us fans, analyzing what just happened shouldn't be as important as enjoying it as it happens.
What was your introduction to hockey analytics? Consider time, place, and initial feelings.
MP1: I would have been in my late teens. I would frequent hockey message boards and occasionally read about Corsi. I actually thought it was silly – until I learned what it was. It prompted me to take a deeper look at the work that had been and was being done. In particular, Eric Tulsky's work on zone entries intrigued me. In the lockout-shortened 2012-13 season, I began tracking entries for the Ottawa Senators. Over the course of my next two seasons, I expanded the scope of my research on zone entries and began to publish my findings on various blogs. The rest, as they say, is history.
CS: I actually got my first taste of analytics through capology, not pure on-ice numbers. I did a 25-page thesis in college on the economics of the 2012 NHL lockout, and ended up diving into how numbers can better provide teams with efficient ways to utilize their cap space. As someone who grew up on and off of skates and playing the game, I didn't come in looking at analytics as a way to either replace or refute what you can learn about the game on the ice or behind the bench; rather, I came in looking to see what it could give me as a launching point for quantifying what was good and bad about different players. To me, stats were always - and still are - a way to help predict player sustainability in certain areas, which helps build your team (both financially and from a roster composition standpoint) now and down the road.
SC: I started learning about hockey analytics by checking out Behind the Net, where I would go to find rate (per 60) stats, but then started to see Gabriel Desjardins writing about Corsi and its predictive value. Numbers have always been of interest to me, so suddenly having new ways to evaluate NHL players was a most welcome development. Since I was already a fan of other sports (specifically baseball) that had a more robust statistical background, it was easy to open my mind to new numbers that matter in hockey.
PI: My initial introduction to hockey analytics was from reading work published by Neil Greenberg on ESPN.com around 2011-2012. I was fascinated by how I could follow the logical train of thought and arrive at the same conclusions from the data presented. From there, I tried to read all the available analysis to familiarize myself with new concepts. Once I felt comfortable with the topics, I began conducting my own analysis on the Red Wings before slowly branching out this past season.
CW: My very first blush with analytics was in a hotel room in Chicago, right before the 2014 Winter Classic. My friend and I were nursing our New Year's Eve hangovers, and she was explaining the basic principles behind Corsi to me as best she could. Obviously, I had to do some follow up reading later because it didn't quite stick the first time. From there on, though, I was hooked. I'd just come out a very analytical role in marketing, so using data to evaluate success was second nature, and something I personally had found very valuable.
DL: My introduction to hockey analytics is probably similar to many others who've grown up in Toronto: I saw the Leafs win a lot of games and I didn't quite understand why because they looked generally terrible. I followed a lot of other Leafs fans who started referencing things like corsi and PDO and it just made sense to me. I've always been pretty big on numbers in any sport and was immediately drawn to some of the new ways of explaining the game.
AL: I remember first hearing about Corsi while discussing hockey online with friends and thinking, "well if that doesn't make the most sense ever!" That compelled me to start finding thought leaders to read and sites that were sharing discovery and data. I thought then - and still feel - like each little analytical evolution is quite an exciting step towards understanding hockey as a science and an art.
MP2: I became obsessed with sabermetrics in grade 12. I knew it was for me about 2 pages into Moneyball. When I went to Trent University I saw my chance to get into it with the Peterborough Petes. Have been in hockey ever since.
TH: It was years ago, I'm not certain when exactly ,but it was definitely on Twitter. I remember being put off initially by some advocates of advances stats in hockey because of the method by which they were delivering their knowledge. Often times, the work of smart people are tainted by a less than ideal manner by which they carry themselves.
How does your career experience (both inside and outside of hockey) give you a unique perspective that others may not have in relation to hockey analytics?
I don't have a great deal of experience. That, in and of itself, allows me to have a uniquely naive perspective at times. I think it's fuelled my creativity and mitigated bias in my analysis. That said, the knowledge and experience I gained from my partnership with WAR On Ice has been invaluable. Andrew, Sam and Alexandra each helped me a great deal and introduced me to the inner workings of their website. Corsica wouldn't exist otherwise.
CS: I work as a youth coach, but I get close access to coaches at multiple levels of the game - and for all the mocking we do in stats of 'old school' hockey pundits, I think knowing how skills are developed and improved upon gives me an edge when looking at stats (particularly with goaltenders, my specialty). Stats tell us the what - what outputs is this player excelling at or struggling with? - while coaching experience tells you both the why and the how to improve. For example: player X may have an abysmal high danger save percentage, and stats quantify that. Coaching, though, helps me understand why that HD Sv% might be so bad - and whether it's possible to fix that. Maybe this goaltender is being overly aggressive with his depth on his blocker side, and it's burning him; stats tell me that there's a problem to be identified, and coaching tells me A) what that problem is and B) how I can help that player fix it. To me, that's the difference between simply identifying good and bad players in the moment and being able to predict whether that player can be useful to you down the road.
SC: There aren't a lot of people that I've come across that are into hockey analytics and have the same combination of playing experience (three years of Canadian university hockey) combined with a lifelong affinity for numbers. I'm a statistical oddity in my own right, I guess. I don't have the pure math or computer background that some analysts have, nor do I have real pro playing experience that ex-NHL players have, but there aren't many in the hockey analytics community (Mike Johnson would be one) that have both. I like to think that at least gives me a reasonable perspective on how to apply hockey analytics, though while I strive for balance, I'm sure my analysis tends to tilt toward the numbers. That I'm also interested in other sports helped me approach hockey analytics with an open mind, and that goes a long way.
PI: My background in pharmacy has ingrained the importance of identifying a process that is logical and consistently repeatable. From reading countless medical journal articles, I've developed the ability to look at the data presented and evaluate whether or not the reported conclusions are appropriate. I've taken this process and applied it to the available data in order to generate logical conclusions.
CW: I come from a tech marketing background. One of the core tenets of how our company worked was "data driven decision-making". When you've got millions of dollars on the line, you want to know that the sales you're running aren't just moving units, but which kinds of units, and what kinds of profit margins you're seeing.
The first project I became really "known" for was my initial work in "salary bands", that is breaking down a player's production compared to the NHL average for each cap hit.
This kind of exercise is second nature in marketing, but was pretty rarely done. Most analysts prefered to focus on the on-ice product, but I find the business side of hockey fascinating.
AL: While I'm a "hockey writer," I'm not a classically trained journalist so in some cases, that may allow me to be more open to new ideas and ways of thinking about sport and hockey specifically because I'm making up my own rules. Outside of hockey, I'm a consultant. It's my job to take in as much information as possible and find a good solution for a problem or an area of improvement - without bias. My goal in writing has always been to educate and engage people in hockey with a similar approach - there's so much we *don't* know about hockey. The mental and physical abilities these athletes possess is mind-boggling when you really think about it. Learning the alchemy of how it all fits together is fascinating. That's what motivates me...and what I try to communicate through the writing I do.
TH: Working for more than one OHL team gave me two very different perspectives on how to analyse a team. Now this started almost 15-years ago when a) I was a high school kid and b) analytics in hockey was a long way from being common practice.
The Majors focused on building a team that fits what the franchise did well and had a specific style of play in mind, where as the Storm focused on high end talent and letting a player's skill be more important than any system the franchise wants to adhere to.
Both had the same goal of an OHL Championship as their driving force but each team took a very different approach to it. Both had some degree of success while I was there and it lends to the idea that just because a team approaches things differently than another, it doesn't mean that one is right and one is wrong.
As a professional in the financial industry, I deal with risk/reward and intrinsic value models on a daily basis. From a hockey perspective, I see the same sort of theories being used in the application of advanced stats. The thing is hockey - and sports in general - are far more chaotic and more reliant on chance than the products I deal with in my professional life.
Predictive models can be a valuable tool whether it be in hockey or in finance, however, the less static your subject is, the less efficient the models will be. This is why the advanced stats approach to baseball has grown so well. As a sport, baseball is far more static than the rest.
What are the biggest challenges moving forward into integrating analytics into professional sport, in both hockey and otherwise?
MP1: Bettering the understanding people have about analytics. The idea that this is a novelty in hockey is bothersome and the sooner we stop compartmentalizing hockey analysis, the better. Marrying numbers and video is natural. Statistical analysts are unfairly isolated as a sect when, in reality, they differ only from traditional scouts or evaluators in that they employ a different means to the same end.
CS: Cooperation. Period. It's important not to be flippant and condescending when introducing a new measure of success to a community. Even though stats make the likelihood of accuracy and success stronger for those who integrate them, it's hard to accept that the way one has always done something isn't the best way of doing it. It's hard to remember that when introducing something to someone who is hesitant to embrace it; even though you, personally, know that your method is beneficial, you still have to sell the method on your listener. Statisticians aren't exactly known for being good at PR or salesmanship - and that lack of ability to market themselves well is a huge obstacle for the stats community. After all, you can have the best invention in the world - but it's worthless if you can't convince people to buy it.
SC: Finding the proper organizational support is paramount. If a team is going to hire an analyst (or three if they're serious), then management needs to take their opinions into account when making decisions. It doesn't have to rule in all cases, obviously, but the analyst can't be there to just fill a chair either. Sometimes, they will have different opinions than a coach or GM, and that's okay. Teams could use a dissenting opinion -- especially one backed with evidence -- from time to time. While I'm sure there is room for improvement in other sports, I get the impression that teams in other sports are better at incorporating their analysts' work. In baseball, it may simply be a matter of having a head start but, in basketball, the organizational buy-in has been more readily apparent. They've moved ahead quickly, though there's also a stronger statistical base in such a high-scoring sport. Understanding that the use of analytics can improve decision making also means understanding that they don't provide a guarantee and won't necessarily fix a team's problems in a year or two. It's an ongoing process of gaining as much information as possible before making decisions and just because the information isn't perfect, in many cases, it's still better than what existed previously. Imagine thinking that stats created 40, 50, 60 years ago represent the peak of data information available in a sport. From the analysts' side, improving communication is the challenge -- knowing how to speak the language of hockey lifers. It's one thing to have the numbers and know what they mean. It's another to be able to get the point across to a GM or coach in a way that they can understand and then apply it to either personnel or tactical decisions. That doesn't bring a guarantee either, because there are old-school decision-makers who don't want to be bothered by numbers, particularly those that don't conform to the opinion that they already hold but, in those cases, that's a management shortcoming that is ripe to be exploited by teams that do appreciate the value added by their analysts.
PI: In my opinion, the biggest challenge is taking all the excellent work that has been done and finding a way to explain it in a manner that demonstrates why each analysis is capturing and why it is important. I believe there is a negative association with the terms "advanced stats" and "analytics", and it is primarily based on the others not having a complete understanding of what is being analyzed and why it is important.
CW: Communication is key. I've attended several conferences now, and this has always come up as a sticking point between the analysts, and whoever they're passing off the findings to, whether that's coaching staff or front office. Often there's a perceived arrogance on both sides of the aisle - analysts vs hockey ops staff - and that can really put up walls between the two, when the entire goal is to ensure a team is at their peak performance. If you can't get both sides talking openly and honestly, and making sure they're understanding both points of view, analytics will have a hard time finding champions in any organization.
DL: The biggest challenge really is and always will be buy-in. People are very set in their ways and an opposing viewpoint that challenges them is threatening to their beliefs and their own way of analyzing the game. Because of that, even if they do see some value, if there's a disagreement they'll generally go to what they know instead of combining both sides.
AL: For me, the biggest challenge is effective communication and it's a two-fold issue. There's obviously still a hurdle that "traditional" names for stats like Corsi, or concepts like "zone starts" presents. People will tune that out, or dismiss it. So finding people who are willing to hear something new is challenge number one - just getting the communication to happen. The second challenge is making the communication that comes after the door is opened meaningful. Nine times out of ten, after you explain a stat or concept in "plain English" a person will say "well of course! That makes sense" but, the nature of scientific process demands a communication method when presenting new findings or research that doesn't always make that easy. We have to find a way to adhere to the rigors of method and scientific integrity and THEN turn those results into easily translatable tools that anyone can understand and apply. I very much believe ideas become better through examination and debate, but we need to be able to get to a place where people can engage in that because they are willing and able to discuss analytics in order to move forward. I think it's important to note that this applies to fans, athletes, coaches - anyone in and around the game.
MP2: Teams have a lot of people telling them a lot of different things about analytics. At this point in it's development in hockey, anyone can play expert.
TH: Melding the world between old school and new school thinking. Finding the proper analytical approach while being open to the possibility that there might be flaws in your model that can't be explained through a tangible value. The perfect example of this is goaltending. Far too often, people in the 'analytics world', dismiss the significance and value of goaltending. Since they're not able to come up with a model or statistic that explains what makes a goalie good, they write it off as luck. The running joke is that 'goalies are voodoo' and at this point isn't mostly goalies that carry on that idea sarcastically. While it's a joke now, it's rooted in the reality that far too many in the stats community badly down play the importance of the position. Advocates of advanced stats in hockey fought with the argument that 'just because you don't understand them, doesn't mean they're not significant and important.' Now many of those same people are on the flip side and of the argument when it comes to goaltending. Just because your models can't quantify if or you don't understand it, goaltending shouldn't be chalked up to 'voodoo'.
Hypothetical situation: you've never seen a player play before and know next to nothing about him, but he's a regular NHLer with readily available information that would be available for all other NHL players. You're given 45 minutes to do as much research as you can to become an “expert” on him. How would you do so?
MP1: I have a pyramid policy – start at the top and work your way down to the bottom blocks. I begin with macroscopic analysis before investigating how it's all supported. My yet-to-be-unveiled model K is a single number metric that approximates player quality. I can try to reverse-engineer this rating by further digging into a player's stats and find out what makes them tick. Given enough time, I would look at everything. From Corsi to goals to penalties to quality of teammates to points. I'd spend any remaining time breaking down video.
CS: Assuming we're talking about skaters, I'd start by pulling up his stats, both raw and advanced, for various situations. Glancing over that data, I'd check - just doing a quick overview - for areas that look significantly above or below the league norm. Maybe his shot suppression numbers are unbelievable, but his scoring is low. I'd check his zone starts, then take a look (if available) at quick videos of him either scoring or being scored against. I'd want to see, even if it's just a quick sample, how he plays in certain situations to yield the results he has.
For goaltenders, I'd go about it in reverse. Goaltending stats are still so far behind player stats, they're almost worthless without context - so I'd try and get ahold of about 30 minutes of collective tape of him playing in a few different games. Once I had an idea of what he looks like in net, I'd take a look at his stats in regards to save percentage in certain situations. More importantly than his stats, though, I'd try to find out as much as I could about his most recent goaltending coaches. The coach for a goaltender matters more than anything else - and if you know certain preferences for that coach, it can go a long way towards helping you predict whether certain data is likely to be sustainable or not for a certain goaltender.
SC: I'd pull up statistics. Start with basics (Goals, Assists, Shots, Time on Ice, Hits, Blocked Shots) then go beyond (Corsi%, CorsiRel%, O-zone start%, on-ice SH%, on-ice SV%, Quality of Competition and Quality of Teammate metrics). Being able to put those stats in context with other players would be helpful too. At that point, I would have a pretty good idea what I'm expecting to see out of this player. If it's possible to find zone entry and zone exit data, that would be great. With those numbers in hand, I'd then seek out video to see if I could find some explanation for how he's generating either especially positive or negative results.
PI: I'd look to identify his impact on possession and individual production over the past 3 seasons to identify what he has been capable of doing. Specific statistics I would look at are 5v5 time on ice per 60 minutes, 5v5 relative Corsi for and against per 60 minutes, 5v5 relative goals for and goals against per 60 minutes, 5v5 individual primary points/60, 5v5 quality of teammates (time on ice) %, 5v5 true zone start % (factoring in on-the-fly shifts), and then 5v5 quality of competition (time on ice). Based on his age, I would factor in the likelihood that he sustains, increases, or decreases production and possession effects over the next 3 seasons. Resources and references I would use would include data from Corsica.hockey and HERO charts from Domenic Galamini.
CW: Jeez, difficult question. I think I'd start by looking at the numbers, because that, more than watching 45 minutes of tape would give me a sense of role. First, I'd probably delve into some basic production stuff (points, TOI, 5v5 vs special teams results), then look at some relative numbers and WOWYs. I like to check a guy's career over time (is he trending up or down?). I would definitely want to watch some clips of him in action, because skating and on-ice decision-making tend to be two of the most important factors in today's game. However, the time constraints would likely mean this would be less thorough than I'd like. It's easier to find highlights than gaffes, and it wouldn't give a good sense of the guy's "average" game.
DL: Hit up corsica.hockey (sponsored) and look at their TOI, point production at 5-on-5 and the powerplay, and see if they drive play at evens. I'd also look at their play over the last three seasons to see any variations in what's happened in the past and how they're trending lately.
AL: I think first I would go to a few key visuals - Dominic Galamini's HERO charts are invaluable; as are Micah Blake McCurdy's three-year skater cards and year over year charts (WOWY etc). I'd use Corsica.hockey to pull up historical data - both individual and "on ice" (possession related) to evaluate consistency and any trending in the players performance. And, of course, I'd do a google search or two to find what people have written on the player.
MP2: I have a model with final answers on how good a player is. A model does a much better job then I ever could of evaluating different factors against each other. You should evaluate players with the exact same formula every time. If you have to consider something outside that formula, then it's not good enough. Everything that you weigh the importance of in your head can be calculated. So do it.
Is it important for fans to have readily accessible statistics? If so, why, and do you feel it makes the game more interesting?MP1: It's paramount that information be accessible but fans shouldn't be bombarded by it. I believe the NHL has failed to satisfy the former to this point. With Corsica, I'm following in the footsteps of WAR On Ice and their commitment to openly sharing data. Proper infrastructure and access to the fullest depth of data is conducive to better research, and I'm trying my best to provide that. I've personally found that my interest in statistics has enhanced my enjoyment of hockey.
CS: Yes and no. On one hand, anything you can provide to fans to help them understand the game better can only help you as an organization. On the other hand, stats are just a piece of that puzzle; it's no more important to provide stats than it is to provide player accounts, coaching insights, and systems breakdowns. Providing one without the others doesn't hurt you, but it can only help to add more to the story you present the fans. I think it's more important to remember that.
SC: I feel like it's important to understand what's really happening in a game and using advanced statistics helps tell that story more accurately. It dispenses narratives not backed by evidence and can tell a more accurate story about why a team won a game or series. For me, that means digging into alternative hockey stats sites all the time, because they have information that I can't get from a mainstream box score. And just imagine what a broadcast might be like if they raised the statistical bar for their audience. That doesn't mean bombarding them with numbers, but when a team is getting drilled when it comes to possession, it's fair to mention as much. There are real stories that can be told with data, especially involving matchups, but rare is the broadcast that is forward-thinking in this respect.
PI: This is a challenging question. I absolutely love having readily accessible statistics from websites such as Corsica.hockey, hockeystats.ca, and hockeyviz.com. At the same time, it is important that each person learns what each statistic captures and how to correctly apply that information. A smart person once told me that outside of the top 1% of players and the bottom 5%, statistics can be used to paint a player in a positive or negative light. It is up to the analyst to determine the direction they should lean. I believe having readily accessible statistics in the hands of people who know how the use them creates insightful and interesting discussion.
CW: I think so. For me, it really enhanced my fandom because I could relate to the data in a way I couldn't to the game. I didn't grow up a hockey fan, like so many did, so it really enhanced the experience for me. That said, I think there's plenty of room for traditional analysis, and some of my favorite parts of broadcasts are "when I played" stories, or when they do a quick breakdown of the X's and O's of a play.
DL: It's important for fans, but not important for all fans. Some want to dig deeper and analyze and readily accessible stats are huge for that. I don't know if it makes the game more interesting necessarily, but it does help people understand the how's and whys behind it.
AL: It's incredibly important. First, people use the data they have access to. We cite box scores because they are what's in a mainstream game recap, or on a league's website. Second, access makes people want to understand what they are looking at - this ties back to communication. I think it makes the game experience more interesting because we're again challenging the sport to move the ball (or the puck in this case) forward by looking at what *really* matters in understanding play.
MP2: I think a very small percentage of hockey fans actually care about advanced stats. Most people just aren't interested in going into that kind of depth. Nothing wrong with that.
TH: I don't know that it's important. It would be nice and many people would benefit from the statistics being readily available but if they aren't, it's not the end of the world. The statistics being available to the common fan would help those who want to learn, but if you can't enjoy watching hockey without them, then I think you're missing something.
Is it a problem if formerly public data gets “bought out” by teams? If so, why?
MP1: Sure it is. It means many sleepless nights and upwards of $200 spent on coffee for some poor sap like me.
CS: Teams are a business, no? If they're willing to purchase the rights to access data that could give them an exclusive advantage, I don't see that as a problem. Harvested and analyzed data is a bonus for fans, not a right - if teams want to make it their own, I don't see a problem with that.
SC: It's a problem, but a selfish one for the consumer. Would I prefer to have access to Extra Skater or War on Ice? Of course. They had some features that haven't been replicated at other sites, but it seems that others (Corsica.Hockey, Puckalytics, Natural Stat Trick, HockeyViz , HockeyStats.ca) have been able to fill the void. Now, if those sites are bought up, that would leave a pretty gaping hole in how I consume my hockey information. Presumably there are others out there that could take over, but there is such a feeling of uncertainty when at a moment's notice the site could be gone. My preferred sites seem to change, whether I like it or not, every couple of years. It would have been great if NHL.com's offering wasn't a train-wreck. Providing this data to consumers keeps a certain percentage of the fan base more engaged, but there are teams that won't like that so much because that engaged fan base is more apt to question team decisions.
PI: Over the last couple of years we've seen ExtraSkater and War-On-Ice, among others, shut down as their creators were hired by NHL teams. Each time, we lamented the fact that the loss of the websites created a massive hole in hockey analytics. However, each time a major website has shut down, a new one has popped up. When ExtraSkater shutdown, War-On-Ice popped up and with it came a whole wave of new statistics. And as War-On-Ice shutdown, we've seen the emergence of Corsica.hockey and hockeyviz.com. While each loss of a website is huge, we've seen hockey analytics continue to advance as new members of the community step up.
CW: I think we've been lucky in hockey with so many people being interested enough to build their own public databases. I think the problem isn't so much that the teams *shouldn't* be doing this or anything like that, but it does make work more difficult. For instance, War-on-Ice, which recently went dark, used a slightly different method of score adjustment than Corsica.Hockey. So though the results are very similar, they're not *quite* the same. While that's not an issue for any of my projects, I could see how it'd set someone else's research back. But ultimately I think that's just a natural consequence of good people getting recognized for doing good work.
DL: Something else will always come up, so until that ends I'm not worried. If all we have is NHL.com then it's a problem.
AL: Depends which team! But seriously, eliminating public access to data can create setbacks in the education and innovation that is yet to come. Taking away access slows down the ability for that data to become mainstream, (and now is of course the time when we note that if the league is going to take on this challenge they MUST talk to subject matter experts to get the data right). But buying out the data also implies that the mind that generated that presentation is now in-house as well and that voice is lost. We have been fortunate to have more smart minds in the community take on the replacement of public data, for example War on Ice and recently Corsica.Hockey. The former of course also gave more than ample notice that they would be going away. But that can't always happen and when that doesn't, we have to focus on knowledge resources on rebuilding the data and access before we can continue to innovate and take it to the next step.
MP2: The people who own it have the right to do whatever they think is best for themselves.
TH: No. If the person responsible for tracking and compiling the data finds a way to profit from their work, they deserve to. It's annoying that it's no longer available but it shouldn't be viewed as a problem.
What are some flaws you find in the practical use and application of analytics in hockey?
MP1: I lament the fact "analytics" has become synonymous with "Corsi" but we're partially to blame. I find people rely entirely too much on Corsi as a measure of quality, often at the expense of more thorough evaluation. I think the use of analytics in hockey is so much better than the alternative that it's easy to lose sight of the limitations statistical methods impose on our ability to conduct analysis. People acknowledge these limitations, yet often stubbornly oppose alternative possibilities or viewpoints.
CS: Using analytics to predict goaltenders is almost always flawed. So few in the hockey community - both from an 'old school' perspective and a stats perspective - have bothered to try and understand goaltending as anything other than a voodoo-riddled corner of the game that doesn't make much sense to them. Few parts of the game have evolved as quickly or as drastically as goaltending has in the last 20 years, and neither stats nor eye test analysis have kept up with it. Still, people discuss goaltender stats as if they have the same level of effectiveness as player stats, and that's led to quite a bit of misconception surrounding the position.
SC: The biggest flaw is that they aren't used and applied consistently. Some teams do and, especially in recent years, it's easy to see how it applies to their decision-making. If a team is intent on using analytics, they won't have much difficulty in finding ways to make smarter decisions, so the biggest flaws for the practical use of analytics involve the humans using (or not using) them. Of course data can, and will, get better, but there is no point throwing out or ignoring good information just because it's not perfect (as though that standard of perfection applies to other player evaluation tools).
PI: I think the biggest flaw in the practical use of analytics is that many people want to boil every player down to a single number. The most common question I get asked is "what is the most important statistic in your opinion?". There isn't a single statistic that paints the entire picture about a player. There are many different inputs and outputs that go into hockey and thus, the identification of a singular metric to represent a player is impossible. It is imperative that we consider what is captured by each statistic, what the benefits and limitations are, and what context clues need to be used in order to properly evaluate a player.
CW: I really dislike the way WAR is used. For people who work with the numbers a lot, we know that it's calculated by weighting several factors and each of those factors can be broken out and looked at by itself. However, most people don't look at fancy stats every day, and then they see a single number and want to hang their entire evaluation of a player on that.
Maybe one defenseman has better WAR but is a LHD, and your team is weak in RHD.
Should the fans (or even the team) be fixated on this one number? My other major gripe is with the term 'replacement-level' itself because it makes it sound like it's easy to rework a roster or replace an underperforming player. It's not. That's not how the real world works. There are not just opportunity costs (trades, buy outs), but pipeline/development concerns, and the human element to consider too. Recently, Micah McCurdy suggested "baseline" as an alternative, and I think that better encapsulates the concept.
DL: Communication is vital. If no one knows what you're talking about they probably won't care. Making things as clear as possible is a clear benefit to furthering the discussion. Also visualization is very important and the same rules apply. Clear, simple, effective, and informative. If any piece of that is missing the info is lost.
AL: We still have so much work to do in terms of getting micro stat data (passing, zone entries) and, excepting Ryan Stimson's passing project, so much of this micro stat work currently is being done in pockets, and not in a coordinated, league-wide way. Additionally, the lack of application of analytics to leagues outside the NHL is sorely lacking and can stunt our ability to look at those players in a meaningful way. Finally, there's the dollar signs we all see coming. "When we can track players" is a common answer to questions about when we'll really know things for sure and start to *really* know what matters - but if you look at some of the quotes that are already out there today for tracking companies, it's crazy expensive and I believe especially as we progress through still understanding what matters and how to measure it, there's going to be a lot of manual intervention - and man power is expensive. It will be interesting to see how this is implemented team by team and league-wide.
MP2: I think most of the community still believes in being able to distribute credit for on-ice events on an event by event basis, especially offensively. I think that we are running into the limits of the play-by-play data's usefulness here. I've had this conversation with a couple people in private (if Micah's reading this, DM me what you think). Shots, assists, and everything else that seeks to give credit to a particular player for an event may be descriptive and predictive, but systematically under and overrates certain players. In my opinion, in the long run we're much better off leaving this stuff alone and evaluating offensive contribution with differential equations, just as we have to do with defense anyway.
TH: As mentioned above, the fact that the application doesn't always allow for the amount of chaos you see in the game of hockey. Baseball is a static game, a pitcher throwing to a batter who is trying to put the ball in play. As a sport, baseball lends itself to advanced stats very well - which is why we've seen the explosion of it throughout MLB. In hockey, the goal remains the same for every team - score more goals than your opponent - the problem is, it can be achieved in very different ways from game to game and shift to shift. There is incredible value to the application of analytics in hockey, the flaw in my eyes is that people want to use them to explain far too much of a given game. Luck and chaos is a huge factor, far beyond what PDO can explain.
Is there a culture issue in hockey analytics in the way they tend to be discussed? Consider tone and audience of how both you and others interact.MP1: If there is, I'm certainly part of the problem.
SC: If you've been part of the online Hockey Stats Wars, and the ongoing battles, you recognize that there is an issue with how they are discussed, but the arguments are almost always the same. Old-school hockey types (whether ex-players or long-time MSM) don't like being talked down-to by someone that's never played the game, but of course that is the same kind of thing that bothers a stats analyst -- being talked down-to because they haven't played the game (or, more ridiculously, haven't been in a dressing room). It's the appeal to authority vs. appeal to numbers, forever. Does having NHL experience provide a more informed opinion? For sure. Is it a requirement to be an analyst? Obviously not. Consider the skills necessary. Ex-players don't generally make the NHL because they think the game better than anyone else. They might, but when weeding out players to find the elite, things like skating ability, physical strength and skills play such a huge part in which players climb the ladder in pro hockey. They might also process the game at an elite level too, but that's in addition to all those other factors. A sharp analyst doesn't need to have played, especially if they approach their analysis understanding that there may be information that they don't have because of that. It's not like there are ex-players (or journalists) running around with a Ph.D in math and collecting their own data. We all have limitations, and it wouldn't hurt if people on both sides of the discussion acted as though they understood that. In any case, both sides get defensive, both sides troll on occasion and (as someone who's been successfully trolled many times) I've tired of both. We'll really know that hockey analytics have taken a step forward when those debates cease to exist. I don't come across the same issues pertaining to baseball or basketball analytics.
PI: I think there is a perception that many of the proponents of analytics are "condescending" to others who ask questions. I think that statement is inaccurate, although I do believe that a lot of people in the analytics community have a chip on their shoulder. After years of being told to "Watch the game, nerd" or that "analytics are stupid", certain people, including myself, have developed a small chip on their shoulder when anyone challenges or questions our work. By no means should that be used as an excuse for condescending or rude behavior. I believe that a majority of people in the analytics community are more than happy to explain their work and why it is important.
CW: I think there can be a tendency in any area to be a bit arrogant when you're convinced you're right. But overall, I think most people in the hockey analytics world are just really excited by all this nerdy stuff and want everyone else to be excited, too.
DL: There’s far too much condescension on both sides. Everyone would be better served if they just chilled out a bit.MP2: Many in the NHL who are interested in analytics are seriously taken aback by hockey analytics twitter, or whatever you'd like to call the community. Their loss. I don't think anyone on twitter should have to change to be more inclusive to more mainstream hockey types. There is a tradition of debate and being a little bit nasty to one and other. So what, it's fun. I remember getting roasted by Dellow. It was good for me.
TH: Absolutely. There are many smart people who do a lot of great work on analytics in the hockey world, the problem is that their message is often lost by the way they interact socially. We have to remember though that just because someone is being a giant jerk, it doesn't mean their work should be dismissed, it just means less people will be willing to give it attention. This works both ways. There are those in the hockey world that are far too closed minded towards the people trying to discuss analytics - Glenn Healy being the prime example. There are still far too may people who hear 'analytics' and chalk it up to 'some nerd that needs to watch the game more’. This isn't something localized to analytics or hockey. It's the case in any setting, professional or social. Whether your analysis is correct or not, people are going to be far less receptive to it if you're being abrasive when interacting with them.
Is there anything else you’d like to add?
MP1: Love each other.
CS: I come into covering and working in hockey from a PR and pre-law based college education, which was all about how to present information in a way that makes your audience want to learn that information. As a result, it may sound like I'm being chastising to the stats community; that's not my intention. There's been an increase in dischord between stats and old-school hockey communities, though - and to me, that marketable way of presenting new information is absolutely why.
SC: I do think there's generally a generation gap when it comes to hockey analytics, that younger people are more interested in the use of statistics to provide evidence, but that may be an impression based on my readership.
CW: Thanks for including me in this discussion!
AL: For me, the importance of visuals in analytics can not be underestimated. The work of Micah Blake McCurdy, Carolyn Wilke, Sean Tierney and others takes full advantage of the "a picture is worth a thousand words" concept. Turning complex thought of evaluation and comparison into easy to understand visuals is huge in terms of translating how to apply analytics to the game and to players for people who follow the game.
MP2: Can anyone name an online sports community that has a higher percentage of women participating? I think we can all take a little bit of pride in that.
TH: It's amazing that the divide still exists when discussing analytics in hockey. It's hard to believe that there are still people out there who don't understand the value of advanced statistics and the impact they can have on an NHL team. The combative nature of the topic has driven people to extremes on both sides and it has created the idea that you must be an advocate of advanced statistics all the time Personally, I enjoy that the advanced stats are there. I appreciate the work behind then an understand that my knowledge of hockey is enhanced by it.
Where I have the issue is when I'm told I must only take into account the analytics and that I don't really know the game without them - it's happened far more often than you'd expect. There are a ton of great people who do a ton of great work with analytics. The key is to find the right ones and ignore the ones that cause the friction. Analytics have incredible value from a team building perspective and they help the fans that are interested in growing their knowledge. But for those out there that watch the innocent joy of the game, analytics shouldn't be forced on them.
Many people still just want to eat the hot dog and not worry about where it was made. If we know and interact with people like that, it's perfectly fine, no one should be judging them for it. Analytics might be necessary for building a successful team, but it is far from a necessity when it comes to being a fan. If a deep knowledge of analytics is what you need to enjoy the game, that's fine. They help explain the game and some people want that. We just shouldn't act like you're somehow missing out as a fan of the sport if you don't understand the difference between Corsi and Fenwick.
Thank you to everyone who participated, and a special thank you to Manny for coming up with the name for this feature. If you have ideas for future roundtable topics or contributors you'd like to see, feel free to leave them in the comments below and we'll see what we can do.