Hugh Williams: Why you should care about patterns?



Posted on: April 10, 2013 by Jordi Safont Guillen





On December 15th 1664 a shipwreck occurs near the cast of Wales; 82 people are killed, only one person named Hugh Williams survives. On December 5th 1785 a similar accident takes place in roughly the same spot; 70 casualties this time, and only one survivor, whose name was coincidentally Hugh Williams. Yet, on August 5th 1860, near Scotland, another shipwreck happens. All passengers died but one. Can you figure out the name? If you said Hugh Williams, you guessed it right.

The moral of this unlikely chain of nautical disasters could be to avoid any cruise near the British Islands, or rather change your name to Hugh Williams if you want to be on the safe side.

This type of astonishing coincidences coined the concept of serendipity, first introduced by Horace Walpole, after the Persian fairy tale The Three Princes of Serendip, whose heroes were always making discoveries -by accidents and sagacity- of things they were not in quest of. As with all serendipities, apparently all three accidents in our story were unrelated, but maybe there was a hidden relation that made the outcome of them unavoidable.

Sometimes we wonder if the world and its laws create a fixed destiny. In a deterministic scenario, by knowing all the inputs and context conditions the outcome of any given situation could be accurately predicted. That could well be the case, unfortunately it would require god-like insight and unlimited time to know all the variables that can affect the final results of a certain action in a specific moment. That's how probability is introduced in our interpretation of reality. Based on a limited background knowledge, and on our previous experiences, we can predict the likelihood of certain results, which usually is good enough to make informed decisions.

As technology has evolved over the decades, our calculation capabilities and the amount of data we are able to process has increased dramatically. And this trend is bound to continue in the future with the advent of new promising innovations, like the quantum computers and new materials like graphene, that could multiply by several orders of magnitude our present computers’ performance.

Additionally, we are generating an unprecedented amount of data, coming from millions of devices, sensors and their interactions, as well as from electronic process transactions, allowing governments and companies alike to access information that was simply not available before.

So, on the one hand we are accessing more information, and on the other hand we have more processing power that we ever had before to deal with it. These two topics pose their own unique technological challenges, more or less summarized under the Big Data umbrella, but apparently they get us closer to a deterministic characterization of the reality.

But are we getting any closer to being gods, to have the superhuman ability of predicting the future? Certainly we have more information, and the means to store it and process it in several different ways and faster than any time in our recent history, but this is just a part of the tale, and not the trickiest one. The real challenge is being able to make any sense out of it.

It is often said that information is power, which is only partially true. It is what we do with it and our ability to make the right predictions and take the appropriate actions from where power stems. Going back to our introductory story, it would have been impressive to know all the ships' names in the world, their routes and the costs of all tickets, but for all the poor fellows in the last episode, the key thing they needed to know was a distinctive pattern that announced that if they were cruising those days around the British Islands, and not named Hugh Williams, they would have better remained on land.

So, do data tell stories? Are all correlations meaningful, or most of them are just misleading? As patterns are discovered, what is the value and impact of events that apparently defy such patterns? These are the tough questions that have to be answered in this era of information deluge, and going beyond Big Data, thinking about data interpretation through patterns and exceptions is the probably the right approach to this challenge.

Image source: Flickr.com