Can we make predictions in hockey with Machine Learning? A simple experiment

Josh W
May 18 2013 11:23AM

 

 

Introduction

You probably don't know me, my name is Josh Weissbock and I am a Masters student at the University of Ottawa up in Canada and I study Machine Learning.  Many of the other authors on the Nations Network, who use advanced stats, typically use statistical models to analyze hockey where as I use algorithmic models. What I can do is use Machine Learning techniques which is an application of these same statistical ideas.  Machine Learning is simply the ability for a computer to take a whole bunch of data, learn from it and be able to make decisions. 

Part of my research is combining Sports Analytics and Machine Learning; can we use these algorithms to learn from hockey data and try and make accurate decisions or predictions?  This area actually has very little academic research; in fact I could find no peer reviewed published works on predicting anything in hockey in any field.  Basketball, American football and soccer all have plenty of research in machine learning but none have been found in hockey. 

This post is the first of many on applying Machine Learning to hockey and the NHL.  In this simple experiment to see if it is possible to make simple predictions on who will win the game, using known stats on each team.  The goal is to beat 50%, or a coin flip.  This type of predicting model is valuable both to team managers and for betting.

Data

I've been collecting data daily on every game that happened over the NHL season.  This experiment used games over a ten week period fro February to April 2013, a total of 386 games.  Before each game I collected advanced stats and traditional stats on the teams.  After the game I collected the result of who won, the score and the shots for and against.  This gives me the flexibility to calculate more stats in the future.  

The traditional stats that were collected were: Home/Away Status, Goals For, Goals Against, Goal Differential, Power Play Success Rate, Power Kill Success Rate, Shot Percentage, Save Percentage, Winning Streak and Conference standings.  For the advanced stats I collected the Fenwick Close %, 5-on-5 Goals For/Against ratio and the season PDO of each team.  These stats have been shown to be much more correlated with wins than traditional stats.

Classification & Results

After pre-processing the data I plugged the vectors of game data into the algorithms to let the algorithms do their learning.  The algorithms I used were Neural Networks (NN) because they are good with noisy data; Support Vector machines (SVM) because it does well in machine learning; Naive Bayes (NB) to learn on past data to predict the future; and Decision trees (J48) as they create a human readable output.  I won't go into the math of how these algorithms work but I have linked them to their Wikipedia page which gives further detail. 

Testing of the algorithms was done using 10-fold cross-validation.  The results of the learning with these algorithms are presented in the table below.  I ran the algorithms on just the traditional stats, just the advanced stats and a mixture of both.

  Traditional Advanced Mixed
Baseline 50% 50% 50%
SVM 60.10% 55.05% 60.10%
NB 59.07% 54.53% 57.25%
J48 58.03% 52.85% 58.29%
NN 58.68% 55.83% 58.03%

Results of the J48 Decision Tree on traditional data

Conclusion & Future Work

60% accuracy, that's pretty decent for a very simple experiment.  That would probably make profit in the long run if used in betting.  I can't confirm which algorithm is statistically best without running a t-test.  What I find most interesting is that we see this advanced stats on the Nation Network are the best for forecasting future success but here they didn't seem to make any improvement in the forecasting rate.  That doesn't mean they aren't valuable in machine learning, it most likely stems from the fact I only used three of them.  Also interesting is if you look at the decision tree generated ML finds the most value in the teams Goals For, Power Play % and Conference standing in predicting a game winner.  I don't think PDO is that useful in predicting over the long run as it regresses to 100% but it would be much more useful in predicting the short term. 

There is much future work to build off this, first I would start with using a different variation of PDO.  Machine Learning can predict more than just who will win a game, it has been used to predict the scores of games, who will win a tournament, which prospects are likely to work out, and there has been plenty of ML research in baseball on predicting award winners and hall of fame nominees. 

If you are interested in my data for your own work let me know.  If you have any questions or discussion feel free to send me a line.  This experiment is a positive result in making predictions in hockey using Machine Learning and future posts will look at it in more depth.

172ff756e336b4deef407cc7fc644369
I am a Van Fan in Bytown. Living in Ottawa for work, I research Sports Analytics and Machine Learning at the University of Ottawa. I play hockey as well as a timbit but I compete in rowing with hopes of 2016 Olympic Gold. Follow me on twitter at @joshweissbock and feel free to give me a shout.
Avatar
#1 Ty
May 18 2013, 01:27PM
Trash it!
0
trashes
+1
1
props

Hello, Josh. I think it would be best to not use this season to calculate a result. The lockout season will most likely be dramatically different from previous seasons, which will hurt the results in the future going. Maybe you could try using the 2012 season to calculate a result. As an example: In the NBA lockout season, last year, the teams generally played very poorly at the beginning, and then got better as the season went on, but not quite enough to where the usual talent level would be at.

Also, I find it interesting that the ML has the GF, and PP% as the most value, as they aren't as predictive as GA, and PK%. Maybe they do have more value, but they seem to fluctuate more than the defensive metrics.

For PDO, do you account for teams with a PDO below 1000 to improve and teams above 1000 to regress?

I'm very interested in this project. I am doing something like this myself (although it is a bit more crude).

Avatar
#2 RexLibris
May 18 2013, 05:30PM
Trash it!
0
trashes
+1
1
props

Has anyone emailed this link to Brian Burke or perhaps Glenn Healy? It might be enough to finally make their heads explode.

Can I be the first to dub the Machine Learning side Team Turing?

I found the results very interesting and something of a surprise, considering the depth and breadth of data available in advanced stats versus the traditional data.

The Machine Learning test is a great experiment to run, and I am very interested in hearing your results as you go along here.

One thing that stands out to me though is that the end result may be that the test provides more information on the advanced stats than the initial goal of the experiment, that is teaching a computer to make predictions off of those stats.

You mention that you expect to use a difference variation of PDO in future, and I use that as an example. PDO was considered a fairly telling statistic, or even meta statistic as it correlates two other performance numbers. Yet here you suggest that, as it is formulated today, it doesn't provide a great indicator of future performance.

I'm excited about this idea and look forward to reading more from you. Thanks!

Avatar
#4 Louis Pomerantz
May 26 2013, 09:20AM
Trash it!
1
trashes
+1
1
props

I have actually been betting NHL games using the premises that advanced stats follow such as regression of PDO, but it seemed like this year more so than the last few, PDO's were more sustainable. If you bet using the regression to the mean premise you would have ended up with a lot of bets on the Devils, Panthers, Hurricanes, Lightning and a lot of bets against Pittsburgh, Toronto, Anaheim and those combinations were not very successful this year. Last year was a god send if you hopped aboard the LA Kings Express after they dumped Jack Johnson. It would be awesome if someone smarter than I could do the same analysis JLikens did over at objectivenhl and apply it to this year (link at bottom) Obviously there is a smaller sample this year but it would be interesting to see how EV GR through 20 or 30 games did in predicting the remaining sample of games in comparison to Corsi T or any possession metric, ideally Score adjusted Fenwick. I would guess EV GR would perform stronger, and the possession metrics weaker, than in previous seasons. It should also be interesting to follow the Penguins if they can keep this group of players together and healthy and see what kind of PDO they are capable of posting as a team, especially with Vokoun in net. No team since the lockout appears more capable of sustaining off the charts shooting percentages as this one.

I cant manage to copy and paste exact chart so link only leads you to blog. The post I am referencing is about 60% of the way down and the title of it is "Loose Ends- Part 1: Predicting Future Success" http://objectivenhl.blogspot.com/search?updated-max=2011-04-13T13:54:00-07:00&max-results=7&reverse-paginate=true

Avatar
#6 scm
May 19 2013, 11:18PM
Trash it!
0
trashes
+1
0
props

Thanks for the article. Couple of questions:

What about injuries / game day roster? This wouldn't be simple random noise in the data.

For the advanced stats did you correct for score effects? or correct for game day roster?

Avatar
#7 Hendy
May 21 2013, 04:29PM
Trash it!
0
trashes
+1
0
props

Is there a 40% failure rate on these statistics? So in as to say you will be wrong 40% of the time?

Comments are closed for this article.