Too often when we talk about Big Data, we talk about the inputs — the billions (trillions?) of breadcrumbs collected from Facebook posts, Google searches, GPS data from roving phones, inventory radio-frequency identification (RFIDS), and whatever else.
Those are merely means to an end. The end is this: Big Data provides objective information about people's behavior. Not their beliefs or morals. Not what they would like their behavior to be. Not what they tell the world their behavior is, but rather what it really is, unedited. Scientists can tell an enormous amount about you with this data. Enormously more, actually, than the best survey research, focus group, or doctor's interview — the highly subjective and incomplete tools we rely on today to understand behavior. With Big Data, current limitations on the interpretation of human behavior mostly go away. We can know whether you are the sort of person who will pay back loans. We can see if you're a good leader. We can tell you if you're likely to get diabetes.
Scientists can do all this because Big Data is beginning to expose us to two facts. One, your behavior is largely determined by your social context. And two, behavior is much more predictable than you suspect. Together, these facts mean that all I need to see is some of your behaviors, and I can infer the rest, just by comparing you to the people in your crowd.
Consequently, analysis of Big Data is increasingly about finding connections between people's behavior and outcomes. Ultimately, it will enable us to predict events. For instance, analysis in financial systems is helping us see the behaviors and connections that cause financial bubbles.
Until now, researchers have mostly been trying to understand things like financial bubbles using what is called Complexity Science or Web Science. But these older ways of thinking about Big Data leave the humans out of the equation. What actually matters is how the people are connected together by computers and how, as a whole, they create a financial market, or a government, a company, or any other social structure. They can all be made better with Big Data.
Because it is so important to understand these connections Asu Ozdaglar and I have recently created the MIT Center for Connection Science and Engineering, which spans all of the different MIT departments and schools. It's one of the very first MIT-wide Centers, because people from all sorts of specialties are coming to understand that it is the connections between people that is actually the core problem in making logistics systems work well, in making management systems work efficiently, and in making financial systems stable. Markets are not just about rules or algorithms; they're about people and algorithms together.
Understanding these human-machine systems is what's going to make our future management systems stable and safe. That's the promise of Big Data, to really understand the systems that make our technological society. As you begin to understand them, then you can build better ones — financial systems that don't melt down, governments that don't get mired in inaction, health systems that actually improve health, and so much more.
Getting there won't be without its challenges. In my next blog post, I'll examine many of those obstacles. Still, it's important to first establish that Big Data is people plus algorithms, in that order. The barriers to better societal systems are not about the size or speed of data. They're not about most of the things that people are focusing on when they talk about Big Data. Instead, the challenge is to figure out how to analyze the connections in this deluge of data and come to a new way of building systems based on understanding these connections.
- Will Big Data Kill All But the Biggest Retailers?
- Who's Really Using Big Data
- Big Data's Human Component
- What Executives Don't Understand About Big Data