By 'analyzing patterns of human experience' researchers say they can detect emerging behavioral trends. Pictured: A 'caixer' (horse rider) rears up on his horse during the Gracia Festival in Mahon, Spain, Sept. 8, 2018.

By 'analyzing patterns of human experience' researchers say they can detect emerging behavioral trends. Pictured: A 'caixer' (horse rider) rears up on his horse during the Gracia Festival in Mahon, Spain, Sept. 8, 2018. Photo: Matthias Oesterle/Zuma Press

Social physics first emerged over 200 years ago as an attempt to understand society and human behavior using laws similar to those of the physical sciences. But, it wasn’t until the past two decades that we finally had enough data, powerful computers and sophisticated mathematical algorithms to develop quantitative theories of human social interactions. We were now able to reliably predict how large groups of people make decisions by analyzing how information and ideas flow from person to person.

“The engine that drives social physics is big data: the newly ubiquitous digital data now available about all aspects of human life,” wrote MIT professor Alex (Sandy) Pentland in his 2014 book, Social Physics: How Good Ideas Spread: The Lessons from a New Science. “Social physics functions by analyzing patterns of human experience and idea exchange within the digital bread crumbs we all leave behind as we move through the world, - call records, credit card transactions, and GPS location fixes, among others. These data tell the story of everyday life by recording what each of us has chosen to do… Who we actually are is more accurately determined by where we spend our time and which things we buy, not just by what we say we do.”

The book was the result of over a decade’s worth of research in the Human Dynamics group at MIT’s Media Lab. Prof. Pentland, along with his graduate students and research associates, partnered with a variety of companies to obtain and analyze real-world data, properly anonymized to protect the privacy of the individuals whose behavior was reflected in the data. They eventually discovered that all event-data representing human activity contain a special set of social activity patterns regardless of what the data is about. These patterns are common across all human activities and demographics, and can be used to detect emerging behavioral trends before they can be observed by any other technique.

For example, as described in a 2013 Harvard Business Review paper, Prof. Pentland's team devised a research project based on data from the social trading platform eToro. The platform allows both single trades, where investors make decisions on their own, as well as social trades, where investors can automatically copy the leading traders in the eToro community and earn money by having their trades copied by others.

After analyzing almost 10 million financial transactions conducted by 1.6 million eToro users, they found that traders fell along a continuum. At one end where the explorers, who came up with investment ideas on their own and followed few, if any, other traders. At the other end where the hyperconnected social learners who followed, and were followed by, many others. Many eToro users were somewhere in the middle, engaging in a moderate level of social learning but also making a fair number of individual trades.

Properly used, social learning made a big difference. “The traders who had the right balance and diversity of ideas in their network - meaning that their social learning was neither too sparse nor too dense - had a return on investment that was 30% higher than the returns of both the isolated traders and those in the herd. In this digital trading environment, the sweet spot resides between the two extremes. This intermediate zone is where social learning - that is, copying successful people - yields real rewards.”

These social physics patterns have been tested across a variety of applications involving people, including strategy formulation in business, economic activity in cities, and--working with an intelligence agency--the detection of potential terrorist activity based on Twitter data. As long as the data involves human activity, similar behavioral dynamics apply.

What accounts for such universal social physics principles? The answer most likely lies in evolutionary biology and natural selection. Survival is clearly a key evolutionary imperative. And surviving in a changing environment requires a combination of social learning and new ideas.

To help me better appreciate social physics, I looked at it through the lens of physics, my long-ago field of study at the University of Chicago. Think, for example, about finding and tracking potentially hazardous, fast moving, near-Earth objects by detecting very small changes in the sky, or looking for Earth-size extrasolar planets by detecting the faint changes in a star’s light caused by a potential planet quickly passing by. No matter how much data we might have access to, it would be near impossible to detect the weak signals associated with either task without the physics principles and models developed over the past few hundred years.

How about our recent advances in machine learning methods? AI is rapidly becoming one of the most important technologies of our era. Machine learning and related algorithms like deep learning, have played a major role in AI’s recent achievements, giving computers the ability to learn by ingesting and analyzing large amounts of data instead of being explicitly programmed. They’ve enabled the construction of AI algorithms that can be trained with lots and lots of sample inputs, which are subsequently applied to complex problems like language translation, natural language processing, and playing championship-level Go.

Human behavior and interactions are dynamic, complex and ever-changing, exhibiting a high degree of variance which make them hard to predict and subject to emergence. The whole might well be different from the sum of the parts. Predicting human behavior requires the ability to frequently analyze relatively small data sets collected over short periods of time.

In 2014, Prof. Pentland co-founded Endor, an Israeli-based startup focused on making fast, accurate predictions using data derived from human behavior. Its prediction algorithms integrate social physics technologies into its data analytic engine, enabling it to extract the underlying social attributes embedded in the raw data being analyzed. Endor’s clients include Coca-Cola, Mastercard, Walmart, and BCG.

Social physics is a new discipline, still requiring considerable effort and real-world experimentation to demonstrate its value. “In the end, I believe that the potential rewards of a data-driven society operating on the principles of social physics are worth the efforts and the risk,” wrote Prof. Pentland in the concluding paragraph of Social Physics. “Imagine: We could predict and mitigate financial crashes, detect and prevent infectious disease, use our natural resources more wisely, and encourage creativity to flourish. These dreams used to be the stuff of science-fiction stories, but that fantasy could become a reality - our reality, if we navigate the pitfalls carefully. That is the promise of social physics and a data-driven society.”

Irving Wladawsky-Berger worked at IBM for 37 years and has been a strategic advisor to Citigroup, HBO and Mastercard. He is affiliated with MIT and Imperial College, and is a regular contributor to CIO Journal.