How do you know Big Data is big these days? Well, there were three stories on it just in Sunday’s New York Times.
Whether that’s a sign that the meme has gone past mainstream – the classic sell signal for the stockmarket is when that’s all everyone is talking about – is a separate question; but it’s clear people are delving into the topic in a big way.
The three stories in the Times spanned the spectrum of views on Big Data – from a quick overview of where the field is to a more sinister look at government power to one questioning how well some data analysis worked at all. Taken together, it’s a nice way into what’s clearly going to be a big part of our lives.
The first piece covers the basic ground, noting that data-driven analysis is seeping into all spheres of life – from business to sports to health and more. It quotes research by Erik Brynjolfsson, an economist at Massachusetts Institute of Technology’s Sloan School of Management, that shows that companies that adopted
“data-driven decision making” achieved productivity gains that were 5 percent to 6 percent higher than other factors could explain.
But it also notes that the tidal wave of data we’re increasingly drowning in can lead to all sorts of erroneous correlations if we’re not careful.
With huge data sets and fine-grained measurement, statisticians and computer scientists note, there is increased risk of “false discoveries.” The trouble with seeking a meaningful needle in massive haystacks of data, says Trevor Hastie, a statistics professor at Stanford, is that “many bits of straw look like needles.”
True enough – and another piece in the Sunday paper, written by two social psychology professors, delves into matchmaking sites and comes to precisely that conclusion.
…can a mathematical formula really identify pairs of singles who are especially likely to have a successful romantic relationship?
We believe the answer is no.
In essence, what they say is that while there’s lots of data to mine about people who sign up on dating sites – likes and dislikes, looks, movie preferences and so on – there’s no evidence that any of those data points predict success in relationships.
On the other hand, yet another piece points to an all-too-successful use of data analysis to track down leakers and the journalists they leak to. It paints a much scarier picture of a world where governments can tap into a plethora of data sources – phone logs, emails, surveillance cameras, GPS devices – to uncover who you’re talking to, and when. No more, it says, will governments have to subpoena reporters to find out who they’re talking to. They’ll already know.
It quotes Lucy Dalglish, executive director of the Reporters Committee for Freedom of the Press, on how to cope with this brave new world:
“For God’s sake, get off of e-mail,” she said. “Get off of your cellphone. Watch your credit cards. Watch your plane tickets. These guys in the N.S.A. know everything.”
It’s a shame, in some ways, that the leak finders aren’t as efficient as the match makers and vice versa. But we’re clearly in a world where data is taking a more central place in our lives – whether it works well or not. And we’re certainly in a world where we’re going to read a lot more about it.