There is an old saying in idiomatic English, that ‘climate is what you expect, weather is what you get’. A salutary reminder, that even the best of plans and expectations are meaningless, in the face of empirical evidence – and that that evidence, in any given instance, might be very different to the average which the model predicts.

Knowing what to expect from the weather has been of vital interest to the human species since prehistory, because our wellbeing is so intimately tied to the ability to farm and hunt, as well as the ability to shelter appropriately from the elements.

Our ancient ancestors turned to the wisdom of their elders, who had the longest memories of previous seasonal weather cycles – and as such the largest datasets to draw upon. Due to the heavy weight of expectation upon them they doubtlessly looked to all current available environmental signals to try and get it right. Our evolution of big storytelling brains helped us develop the cognitive bias of the clustering illusion, seeking patterns and causality where it may or may not have existed: significant and unexplained phenomena such as a comet in the skies or a freak animal birth might coincidentally correlate with an unexpected storm or drought, and from herein the mental model could take shape – “next time, we will know what this sign foretells!” Perhaps with the right sacrifices, we could even invoke the same output somehow..?

By the 20th century the prediction tools had improved somewhat, and there had largely been a move away from the corollary assumptions that the weather could be influenced by human behavior and ritual. But it still depended heavily on human expert interpretation of available data – trying to extrapolate the position of weather fronts and air movements from dispersed and delayed data points, understanding arcane tools and charts… and frequently predictions were highly reliant on the instincts and experience of senior forecasters.

Not so very different from the shamans of old. Of course, they did understand the basic physics of meteorology, and the role of cause and effect. But they depended more than anyone tended to admit on gut instinct, to supplement what could be observed directly – as the latter was so limited. And of course sometimes those ‘educated guesses’ were inevitably uncannily accurate, accruing guru-like status to the one who had uttered it.

Finally things started to change in 1960, with the launch of the polar-orbiting satellite TIROS 1 (the first in the series of Television and Infrared Observation Satellites) – though the spacecraft operated for only 78 days, meteorologists worldwide were delighted with the realtime pictures of the Earth and its cloud cover that TIROS relayed back to the ground.

Nowadays, networks of satellites span the skies, and the exponential growth in computer processing power and data transmission speed means that weather forecasting in the short and mid range terms has become incredibly accurate, to a local resolution of 300m – and we can all access the outputs of this complex range of calculations for our pinpointed location via an app on our phone – which beeps to tell us that it’s going to rain in the next 10 minutes.

Weather forecasting has become truly data-driven.

And it’s exactly the same when it comes to financial predictions and investment decisions. For decades, complex models have been invoked by learned experts, behind the doors of academic institutions – depending on a fair bit of instinct gained by experience and occasionally blind coincidence, to generate returns ahead of the market average. The data involved to input into these models too was carefully gate-kept, and simply never in the hands of the average consumer.

But again, we can now have an app on our phone. Right next to the one which helps us decide whether or not to take a coat, is one which tells us what stocks are rising or falling, based on live data in real time. We can look at patterns, invoke our own instincts, and make decisions… Then invest directly via retail brokerages and exchanges, instead of having to pay fund managers and defer to their expertise.

Even weather data might be used to inform our investments, because companies can use long-range forecasts (formerly the stuff of outright guesswork) to automate investment decisions – such as in the futures market for corn, orange juice, or next year’s holiday travel trends. The data available helps us to shorten the time from realtime weather phenomena to actionable investment decisions. We don’t need to wait and see the harvest first in order to predict its value, and futures traders can respond by longing and shorting the market as appropriate.

Certainly in meteorological terms this technological revolution came at just the right time, because other human interventions were starting to wreak havoc in the climate-driven models anyway. Climate is changing, and this meant that historical data was less and less helpful, even as the years of its having been recorded grew larger.

And even when the model fit the evidence, they never forecasted the extremes, the outlier behavior – such as the famous ‘hurricane’ which hit the UK in 1987. The prediction may have allowed for that as a possible outcome, but due to the likelihood it would never have made the forecast. Hindsight is a wonderful thing, especially when you’re trying to reverse-predict stock market crashes from the length of women’s hemlines.

Financial models too, based on expectation rather than evidence, will not predict the ‘black swans’ of unexpected influence, such as the rise of cryptocurrency markets – where any human instinct about potential impact is likely to be as good, if not better, than the opinion of an expert.

And the satellites which now span the skies stream back terabytes of data on everything we can possibly measure, from traffic movements (and what kind of vehicles and fuel types are involved), human migration, ocean currents and insect populations. We can combine that with web crawler sales data, facial hair trends and solar radiation measurements, to come up with our own correlations, predicting whatever we identify and want to explore, which possibly nobody else has. The point is we have access to staggering amounts of real data, which we can combine in ways that might unexpectedly reveal hidden outcomes and potential.

A data-driven world in one in which we can all make evidence-based decisions, and benefit directly from their outcomes. Hedging against the downside risks of unexpected storms, and making hay in the sunshine.

Exciting, isn’t it?