MIT Sloan Sports Analytics 2012

A final perspective...

Sun, 04 Mar 2012 16:54:26 -0500

It has been a fantastic opportunity here in Boston for discovery.

I’ve discovered that there is a HUGE wave of excitement about the whole concept of data analytics across a range of domains. From my observations I think you can classify the activity into four types:

“Moneyball” analytics - the concept popularised by Michael Lewis’ book (and now the movie) about Billie Bean, Paul DePodesta, and the Oakland A’s. How do you identify players undervalued on a small budget to compete against the rich free agent market?
Performance analytics - the exploration of “big data” in search of competitive insight that can help improve performance and win games.
“Fan-alytics” - an entire industry of fantasy games and fan-based engagement in player statistics and ratings, that seems to me to be almost as important as the competition itself!
Ticketing analytics - much closer to the traditional financial statistics genesis from which many econometricians have emerged into sport.

Where does Australian sport fit into this world?

We talk a lot at the AIS about innovation, and it has long been my view that the only sustainable competitive advantage in sport (or anything for that matter), is the capacity to be perpetually innovative.

We have nothing to hide in Australia if we are forward looking, because anybody who copies our best practice is forever not quite keeping up.

I saw plenty of “exciting”, “new” and “ground breaking” startup companies here that are selling ideas or technology that we have long ago achieved, and in some cases left behind. It leaves me with some confidence that we are doing good work in some of these areas.

Kirk Goldsberry’s presentation on spatial analysis included some great visualisations and a fantastic data set, and that is reaffirming also, since it is the essence of Pattern Plotter, and we started out on that nearly 10 years ago.

The presentation by Jason Sada likewise was enthusiastically received by much of the audience, but again, it mirrors work that we have done some years ago.

We have some exciting projects bubbling away at the AIS in collaboration with groups including Disney Research, Carnegie Mellon University, University of Canberra, Victoria University, and Technical University Munich. I think that much of that work is genuinely new, and possibly even unique.

Everything I’ve seen here leads me to think that in most cases we are at least as far advanced as anyone else, and in a few instances, I’m convinced now we lead the pack.

But, an underlying theme that permeated the entire conference was the communication gap between analytics and coaches. I don’t think that is something we do as well as we could.

When two mathematicians discuss an idea, an equation is the most succinct method to convey the detail. It makes sense to communicate in an elegant and efficient language - if the other end of the conversation understands what you are saying! Exotic machine learning techniques, bootstrapping, and various statistical circumambulations are a foreign language to most people, but too often the analyst reverts to using this mode of one-way communication, and it doesn’t help. Instead, it reinforces that other barrier to coach “buy-in” - the sentiment that some geek with a PhD in calculus is telling the 30 year coaching veteran how it is.

One-way languages do that - the receiver is being told, but they can’t respond, engage or adapt. So of course, they resist.

Even the MIT Sloan conference was not without dissenting voices from coaches. Todd Martin repeated the mantra that communication and context is fundamental, and mostly lacking. On the soccer analytics panel, Alexi Lalas noted that as a manager at L.A. Galaxy, he was acutely aware of the business-side analytics (sales, revenues, forecasts), but dismissed the performance side. But my favourite was Brian Burke (Toronto Maple Leafs) who said:

“There has not been a statistical breakthrough in (ice) hockey yet. Sure, baseball was made for this but in hockey, stats are like a lamp post to a drunk - they’re useful for support, but not for illumination.’’

Since most of the time, most coaches reach the elite level without the aid of any kind of sports science, much less of the mathematical variety, they understandably maintain a view that they must know a thing or two. And they do. The challenge for analytics, in my view, is to find ways to enhance coaching expertise with evidence-based decision support. It is not to supplant a coach’s expertise, nor to relieve them of their responsibility to make critical and complex judgements.

So my sense is we are doing a great job on the technology and research front. With Keith Lyon’s insight and Tim Kelly’s enthusiasm the AIS is leading the way in building virtual communities of expertise such as the Competition Advisory Group (CAG), and there are others.

But we are not there yet, and I don’t think we are even close to fulfilling our potential of a culture that embraces analytics and capitalises on the current opportunities it creates.

Speaking the same language would be a good start.

Soccer analytics

Sun, 04 Mar 2012 16:48:06 -0500

Soccer is easily the largest sport in the world, so it is strangely US-centric that the discussion about soccer, and of the huge quantities of statistical and tracking data that is available in the European leagues (not to mention the enormous body of soccer-specific analytics work in the academic domain), was a virtual side-dish to the main menu of NFL, NBA, and MLB.

Nevertheless, the panel included Alexi Lalas (former player, manager and now broadcaster), Drew Carey (comedian and owner of the Seattle Sounders), as well as current performance analysts Steve Houston (Hamburg FC), Steve Brown (Everton FC) and Scott McLachlan (Chelsea FC).

Following the theme of my previous entry on the tennis analytics, perhaps the best way to convey the threads covered in the discussion is to grab a few choice moments from the session.

Is there buy-in to analytics in soccer?

DC: Seattle is “analytics driven”. Also, they copy everything that Chelsea do.

SH: There is a core group of analytics-driven teams in the EPL and in Europe: Everton, Fullham, Manchester City, Chelsea, plus Dortmund and Hamburg.

DC: The MLS is still a long way behind the EPL, but gradually learning.

DC: The MLS is a small-budget league, and Seattle is a small budget team, so a lot of top players are only getting $100,000 per year, and teams can’t afford $5M players. So it is an even market, and analytics is valuable when you are trying find small differences in player value.

What are the impediments to better use of analytics?

SH: Visualisation of data is a real challenge, and if that was done better then it would be easier to communicate with coaches on a level they understand.

SB: It is generally used very widely in recruitment, but there are years of accumulated data now, and using that is still in its infancy.

DC: The culture of the team has everything to do with it. It’s not about telling the coach what he needs to be told, it’s about the coach and players searching for information.

SM: You can spend a lot of time “selling up” to managers to convince them that analytics works.

How does analytics help in soccer?

“Players are revisionists.” - Alexi Lalas.

…so they need to be challenged with hard evidence and video.

SH: Analytics can be used to find the video that matters. And it should distill complex information into something usable.

SH: The Germans are very analytical and structured, especially with player development. Analytics has more currency in the Bundesliga also because the difference between the teams is not as great as the EPL.

AL: Ironically, as GM at LA Galaxy, responsibility for the business side was more analytically driven than for the playing side. At the time AL was “relieved of his duties”, the business side was doing great, but the playing side had suffered.

What is the future for soccer analytics?

SH: Baseball has the Box Score, so people like Bill James go off on their own and develop new ideas based on publicly available data. But soccer data is harder to get and can be very expensive, and so it is not so accessible.

My thought: that’s a hugely important point. The clubs in baseball didn’t drive the analytics revolution in that sport, it was the mathematically-minded fans who thought of better ways to measure performance. It limits the rate of analytics progress when fewer people can access the big data.

DC: If there was a single metric that could show why A leads to B, then it would have been found. Things are much more fluid and complex in soccer than baseball.

SM: Companies like Decision Technologies are getting into soccer now and that is promising.

SH: Defensive ratings are the hardest to achieve. Stopping things is always more difficult to measure than scoring goals.

AL: Off the ball activities are incredibly valuable, but are not measured because nobody has thought of a way to do that.

SH: Data doesn’t show lazy players, so live scouting is still incredibly important.

My final thoughts: not a bad session and an interesting (if a little secretive) discussion about what teams in the EPL are actually doing with their data. Notably there was not a single mention of the player tracking technologies from ProZone or InMotio.!?

(Graphic by Martin Lames Technical University Munich)

Tennis analytics

Sat, 03 Mar 2012 14:17:19 -0500

Former top ranked players Todd Martin and Paul Annacone were joined by US-based Australian coach Craig O’Shannessy to present the Tennis Analytics panel discussion.

The conversation was wide-ranging and interesting, so for the sake of brevity I will simply relate some of the significant discussion points. On the rare occasion I have a comment to add, these are in italic.

In the latest ESPN magazine (devoted entirely to sports analytics), tennis rated second last, behind boxing, in the use of analytics by coaches and the media.

On what can we learn from player tendencies and patterns?

“All players have tendencies, but we are the best players in the world. We can change what we do” - Pete Sampras to Paul Annacone on statistics.

My thought: looking for aggregated tendencies alone is likely to be futile. What a player does can be influenced by what their opponent allows them (or forces them) to do. So the context of behaviour is where it is at. But, all players have systematic strengths and weaknesses, and player vulnerabilities can be useful if you can understand why they occur. This is how data can become actionable knowledge in the competitive domain.

On how data should be delivered.

Todd Martin: any insight into player tendencies can be useful, but it is important to be mindful of how it is delivered.

“Raw data is a waste of time. It has to be delivered in a context.” - TM

Paul Annacone: in team sports, players need to adapt to the delivery style or philosophy of the coach. In tennis, the opposite should be true, and information should be delivered by the analyst in a way that the player can absorb it.

On playing strategy, exploiting strength and hiding weakness.

O’Shannessy: you don’t have to be good at everything, but you do have to be good at something.

“Return of the 2nd serve is the beating heart of winning matches” - CM

Martin: not all points are equal. Players don’t try to win every point, so analytics needs to be aware of which points are important and contested.

All: unsuccessful points can have a longer term strategy that is hard to pick up in statistics. For instance, Sampras would rush the net and in many instance lose the point, but, he would force more double faults than other players since they knew he would be aggressive on a short 2nd serve.
My thought: this is a pivotal point for many sports. Many outcomes in sports have a long timeline.

O’Shannessy: the “Serve +1” statistic is very powerful in maximising strength.
NB: Serve + 1 refers to the serve as the most effective weapon in all of tennis, combined with the players next-most dangerous weapon in the next stroke. For example, Nadal averages 83% of serves where he successfully combines his two most devastating weapons - his serve and forehand .

O’Shannessy: not all statistics have equal meaning. For instance, a clay courter making 80% of first serves is not as dangerous as Djokavic serving 70% of first serves.
My though: again, context always matters.

A final cogent point from Todd Martin:

“Tennis is like boxing. You don’t get an invitation to hit someone on the jaw, you have to keep jabbing away to keep your opponent from hurting you. You wait for an opportunity to force your opponent into a position of weakness where you can strike them.”

The athletic brain

Sat, 03 Mar 2012 12:25:00 -0500

There is a truism that strikes a chord for me about the cultural difference between Australians and Americans - that in Australia nobody believes “hype”, but that in America nobody believes you without hype.

The second presentation in the Evolution of Sport stream certainly appealed to the latter.

Jason Sara from Axon Sports opened his discussion with a video describing his exciting and revolutionary developments in perceptual training (insert tongue into cheek).

The concept, in the smallest nutshell, is to use digital media and portable technologies like the iPad to train the perceptual skills of athletes using techniques such as temporal occlusion and structured recall.

Perceptual skill is multi-faceted. In sports fan parlance, perceptual skill is having “footy smarts”, or “being one step ahead of the play”. In the world of skilled perception research it relates to the capacity to recognise spatial and temporal patterns which manifests in apparently faster reflexes and better decisions in game play.

In a temporal occlusion task, the viewer watches a baseball pitcher or some similar event, and at a critical (but variable) moment during the pitch the vision is blocked out. Skilled players generally use other cues from the body mechanics of the pitcher to correctly read the pitch, and they don’t need to see the ball flight itself. So athletes with high levels of perceptual skill can generally anticipate the type and direction of pitch very accurately, even when the pitch is occluded at or before the release of the ball.

There is strong evidence that the techniques used by Axon Sports such as temporal occlusion reliably differentiate between levels of skill in a whole range of sports. The evidence, however, that on-field skills can be improved by practicing the task in a temporal occlusion paradigm is less convincing.

There is general view that practicing a perception task on an iPad only makes you better at doing the perception task on an iPad.

We explored these ideas back in my days at the Victorian Institute of Sport as far back as 2001, and I did my first experiments in this area at Swinburne University before that.

So I admit to feeling some umbrage at the hype about “new” ideas we worked through a decade ago…

Finally, and call me a pedant, Jason attributed the “10 years, 10,000 hours” rule proposed by K.A. Ericsson, but popularised by Malcolm Gladwell to the latter. Falsely.

(Image owned by Axon Sports)

The second innings

Sat, 03 Mar 2012 10:16:00 -0500

First on the agenda for day 2 at #SSAC was my former colleague and current countryman Peter Blanch, who provided a compelling account that there is no such thing as responders and non-responders, rather, that there are believers and non-believers.

Peter showed a powerful placebo effect with evidence that when subjects in experiments where told they had received performance enhancing supplements such as caffeine, sodium bicarbonate, or even beetroot juice, their performance in a range of timed performance tasks (eg. 1000m running time trial) improved by 1% or more, regardless of whether they actually received the supplement. Likewise, in the same experiments where the subject were told they did not receive the performance enhancing supplement, their performance did not improve, again regardless of whether they actually received the supplement.

His discussion touched on important methodological considerations for designing double blind experiments that test the efficacy of nutritional or therapeutic interventions. He proposed that the expectancy of a positive impact of an intervention participants should always be measured.

Cumulative Win Probabilities in NCAA Basketball

Fri, 02 Mar 2012 15:13:00 -0500

Mark Bashuk presented his work on cumulative win/loss probabilities using play by play analyses of NCAA basketball games. He began by describing the way that coach perceptions of the quality of their team’s performance can be biased by events that occur later in matches - the so-called “buzzer-beater” plays that can turn a result.

This presentation described a method of calculating the probability of a win using a cumulative probabilities with weighted factors for strength of schedule (SOS), home court advantage and temporal segments of the game.

The dataset included 1,091,627 plays, collected from 3500 games during the 2011 season.

The paper found here also includes some very interesting links to some more detailed quantitative analysis.

http://www.sevenovertimes.com/

The audiences seem quite diverse, and I found myself seated next to Thomas Kelly III, Vice President of Lockheed Martin. His question to the speaker was telling:

“How does quality of coaching effect the outcome of the game in the final 2 minutes?”

There was not a substantive answer.

"If you torture a number for long enough it will tell you anything"

Fri, 02 Mar 2012 15:08:13 -0500

“If you torture a number for long enough it will tell you anything”

- Attributed to “somebody once said”

"Designing a metric is only part of the problem. Making decisions is still the hard part."

Fri, 02 Mar 2012 14:38:18 -0500

“Designing a metric is only part of the problem. Making decisions is still the hard part.”

- Dean Oliver (author Basketball on Paper) - Basketball Analytics panel discussion.

Sports analytics twitterverse

Fri, 02 Mar 2012 14:30:24 -0500

Follow the tweets on #SSAC

Predicting the next pitch

Fri, 02 Mar 2012 13:51:45 -0500

The question is posed, if the bases are loaded, the game is tied, and the pitch count is 2 and 2 (two balls and two strikes), what is the next pitch?

One way to approach this is to simply calculate the frequency of pitch types thrown by each major league pitcher. This is the naive model based simply on priors.

John Guttag (MIT) presented a very interesting paper using Support Vector Machines (SVM) to predict whether a particular type of pitch will be thrown given with much greater accuracy than the naive model.

SVMs are a supervised machine learning technique that have become very popular in a range of computer science domains, which make it possible to detect patterns and classify features of a data set using input factors.

The input factors used here to build a model of pitch prediction include:

batters profile (slugging percentage, runs scored)
pitchers prior tendency (pitch types, pitch results, velocity, location)
game state (inning, number of pitches already thrown)
at bat state (number of balls, number of strikes).

The latter was the most important, and predictive.

The data included 359 pitchers and considered 6 pitch types.

Interestingly, but predictably, the accuracy of the SVM classifier was greatest for the at-bat states where the ball count was 2 or 3, meaning the next pitch is very important!

The paper can be accessed here, but the shortest take home message is that the SVM model improves on the naive model by 18%.

The presenter concluded that most pitchers have a dominant pitch, but that at-bat state, and other features can add significantly to the accurate prediction of the next pitch.

I’m curious to know how many major league hitters can do a support vector machine analysis in their head?

In the interest of the game

Fri, 02 Mar 2012 13:46:44 -0500

Proceedings have begun with the sort of polished presentation that you might expect from a conference sponsored by a global broadcaster in ESPN.

The organisers from the MIT Business School welcomed 2200 delegates, some from “as far away as Australia!”.

Included in the Aussie contingent are former AIS staff members Peter Blanch (now Cricket Australia) and David Rath (now Hawthorn Football Club).

As an indication of the growing reach of the conference, organisers registered over 1,000,000 online votes for the people’s choice paper award. The accepted research papers are available here.

In fact, the online audience is larger than the number of direct delegates, and subscribers may watch online and live-post SMS questions for the panellists.

The opening panel included representatives from the professional sporting leagues in the US: Gary Bettman (Commissioner, NHL), Rob Manfred (VP, Major League Baseball), Steve Tisch (New York Giants), Adam Silver (NBA). The panel was moderated by “everybody’s favourite sports commentator(!)”, Michael Wilbon (ESPN).

The discussion focussed on the business models for each of the major professional leagues, and the challenges they currently face (player lockouts, equity between rich and poor teams, expansion into China and the pacific rim, and player career longevity.

Predictably the conversation moved to Jeremy Lin, and how it is that a player so capable of success can remain “undiscovered”, until a confluence of circumstance and opportunity launched him into a whole new level in basketball.

Something new..

Thu, 01 Mar 2012 14:12:00 -0500

So we embark on new experiences - a new conference and a new reporting method..

I’ve arrived at the annual MIT Sloan Sports Analytics Conference in Boston, MA. This conference will be different to most others I suspect, with a focus on the “Moneyball” aspect of sports data analytics. But I come with real enthusiasm and no preconceived ideas.

http://www.sloansportsconference.com/

At Tim Kelly’s urging, and with Keith Lyon’s link to tumblr, I will endeavour to report daily on the sessions I see, the people I meet, the discussion I have, and the observations I make….

This will be interesting!