November 5, 2012
by Sal Robinson
Forget Nate Silver, Devavrat Shah has got our number: the MIT associate professor has developed an algorithm which he claims can predict with 95% accuracy what topics will be trending on Twitter within an hour and a half, and sometimes four or five hours away, which has got to be like the distance from Earth to Jupiter, in Twitter time.
Shah and his assistant Stanislav Nikolov, who, interestingly, works at Twitter, will present their findings next Friday at the MIT Museum. And I have a feeling this may be a rather heavily attended academic presentation.
Shah and Nikolov’s algorithm is based on existing Twitter data and is nonparametric, meaning (and thank you, MIT press release, otherwise I would not have known this) “it makes no assumptions about the shape of patterns.” They started off with a training set:
In Shah and Nikolov’s experiments, the training set consisted of data on 200 Twitter topics that did trend and 200 that didn’t. In real time, they set their algorithm loose on live tweets, predicting trending with 95 percent accuracy and a 4 percent false-positive rate.data on 200 Twitter topics that did trend and 200 that didn’t.
MIT points out that:
The algorithm could be of great interest to Twitter, which could charge a premium for ads linked to popular topics, but it also represents a new approach to statistical analysis that could, in theory, apply to any quantity that varies over time: the duration of a bus ride, ticket sales for films, maybe even stock prices.
My Twitter feed has always seemed like more or less the epitome of randomness, so I, as a genuine Twitter dope, will be interested to hear what Shah and Nikolov have come up with. For one thing, most of my feed is in German. I remember following “The British Library” at the start of my Twitter-life, and from there on, it’s a long, jumpy blur, ending up somehow with Friedrich Nietzsche and Live With Kelly and Michael. Even cupcakes, apparently a constant in all other realms of adult New York life, only come up at surprising times. Even cupcakes, people.
Sal Robinson is an editor at Melville House. She's also the co-founder of the Bridge Series, a reading series focused on translation.