Later this year the Met Office’s new £97 million supercomputer is due to become operational. The machine promises to greatly improve the precision, detail and accuracy of the weather forecast.

How will it do this? I research weather prediction at Oxford University’s department of Atmospheric, Oceanic and Planetary Physics and the challenge of getting better forecasts is not just about getting bigger computers – although that obviously helps – but about using them in cleverer ways.

Let’s take a step back to see how things used to be done, since weather prediction has changed hugely over recent decades.

Until the 1960s the forecast was based on making records of observations, and identifying patterns, or “analogues”, in these records.

The idea was very simple. If a long enough record of the weather is maintained, the forecaster has the (relatively) easy job of looking through the record for a day when the atmosphere looks pretty much the same as today, and then issuing the historical evolution of the atmosphere from that starting point as today’s weather forecast for the coming week.

But this didn’t work very well. The reason for this is chaos, or the butterfly effect. The evolution of the weather over days to weeks is very sensitive to small details in the state of the atmosphere, but these details may be too small to detect using the satellite and weather balloon data available.

Poor as they were, however, analogues were the best option available since the other method – using equations to create mathematical models – was not practical until the birth of the electronic computer.

The British mathematician Lewis Fry Richardson pioneered the use of mathematical models during WWI. He had a big problem though. To calculate the forecast for six hours ahead required him to solve partial differential equations by hand – which took six weeks to compute, and was wildly inaccurate to boot.

Yet Richardson had the right idea, and now with computers atmospheric simulators are indispensable.

A modern weather forecast starts with the maths – the equations which describe the evolution of the atmosphere:

Firstly, we have the Navier-Stokes equation – really three equations describing conservation of momentum in each of the three Cartesian directions. Here, we’ve accounted for the Earth’s rotation by transferring to a rotating frame of reference - the second term on the right hand side is the Coriolis force, and the third term is the centrifugal force.

This equation is particularly difficult to solve because the advective derivative D/Dt hides some nasty nonlinear terms in u (there’s a reason why understanding solutions to the Navier-Stokes equation remains one of the unsolved Clay Mathematics Institute million dollar problems…).

Next up, we have the continuity equation. What flows into a box must flow out, or the density inside the box must increase.

Thirdly we have the thermodynamic energy equation, where Q is the diabatic heating rate. And finally we have the equation of state for the atmosphere.

OK, so where do we go from here?

The first step is to discretise our equations of motion. It is impossible for us to calculate exactly how every little flurry of wind will swirl, and actually not really necessary.

So we split the atmosphere up into little cubes – in a weather forecasting simulator, these might be 10x10 km horizontally, and from a few hundred metres to a few kilometres vertically. Within one of these cubes, we represent the atmosphere as being constant, with one number representing the average temperature, one representing humidity, wind speeds, and so on.

It’s immediately apparent that we’re going to have a problem – what about processes happening on smaller scales than this?

These processes, such as clouds, are still very important for our forecast, so must be represented. Crucially, they affect how the larger scale will evolve, but also describe important weather phenomena for us down on the ground – like rain, or gusty winds.

We represent these processes using approximate equations, called parametrisation schemes. These approximations and simplifications are a large source of error in weather forecasts.

Ideally, we would make our boxes as small as possible. And we should certainly include as many small-scale processes as we can think of. And make these schemes as accurate as possible. But in the end we have to accept that our computer simulator will never be perfect. It will always be just that - a simulator.

So instead of trying to do the impossible, and predict exactly what the weather will be next Tuesday, with 100% accuracy, wouldn’t it be more useful if we accepted our limitations and instead produced a probabilistic forecast for the weather next week?

Instead of predicting rain with 100% certainty, we recognise the uncertainty in our forecast – perhaps the probability of rain is just 90%, for example.

To do this, we need to look critically at our simulator, and identify exactly where the errors in our forecast come from.

This is what my research focuses on. I work with a new technique, called stochastic parametrisation schemes. These use random numbers (that’s all “stochastic” means) to represent uncertainty introduced into our forecast by the unresolved small-scale processes. Instead of calculating only the most likely clouds over Oxford, for example, we calculate the effect of many different possible clouds on the large-scale weather patterns to see how this affects the forecast.

In other words, our parametrisation schemes are now probabilistic.

Now, instead of making a single, best-guess, forecast, a set of forecasts are made for the weather next week. The set of forecasts start from different, but equally likely, starting conditions estimated from our measurements of the atmosphere. Each forecast also uses different random numbers in the stochastic parametrisation schemes indicating different possible effects of the small-scale processes.

The use of stochastic processes to represent uncertainty is not new at all – they are ubiquitous in financial modelling for example – but their use in weather forecasting is only just taking off, despite the fact that meteorologists were among the first to describe chaotic systems.

Interestingly, it has been found that certain weather patterns are very predictable – the errors due to the starting conditions and model simplifications stay small as we look to the future, and our set of forecasts track each other closely.

A good example of this is a blocking anticyclone – a high pressure weather system that squats over Scandinavia for days or even weeks at a time, sweeping down cold air from the North, and deflecting storms south of the UK. Bitterly cold but beautifully clear winter days? Sounds like a block.

On other occasions, including these representations of uncertainty leads to a large divergence in the forecast for the coming week, indicating the atmosphere is in a very unpredictable state. This information is very useful! The perfect example of this is the infamous Great Storm of 1987. It’s not Michael Fish’s fault he got the forecast wrong – the atmosphere was just in a very unpredictable state that evening.

Facebook Twitter Pinterest The Great Storm of October 1987, as forecast by a modern probabilistic forecasting system from 66 hours previously. Top left shows the observations – a deep low-pressure system with very strong winds. Top right shows the best guess forecast – this is what Michael Fish would have seen. The other fifty panels show equally likely outcomes from a modern probabilistic weather forecasting system, indicating substantial uncertainty in the forecast. Source: Royal Society

Over time, as we get bigger and better computers (and more and better observations), our forecasts do improve.

The graph below shows the skill of the “best-guess” forecast made by the European Centre for Medium Range Weather Forecasts (ECMWF), based in Reading (whose computer simulator I work with and which, incidentally, currently has a much bigger supercomputer than the Met Office). You can see how the accuracy of their forecasts increases over time. A seven day forecast made today is as good as a 5 day forecast made twenty years ago.

Facebook Twitter Pinterest Graph showing the percentage accuracy of the “best-guess” weather forecast over the last three decades in the northern and southern hemispheres. Source: ECMWF

We can also measure how good our probabilistic forecasts are – it’s not all just a cunning ruse to avoid committing to a prediction (“well we only said it might be sunny…”). It is possible to measure statistically how reliable our probability distributions are, and in fact we see a rapid improvement in probabilistic forecast skill over the last decade – a 7 day forecast today is as skilful as a three day forecast was 20 years ago.



Facebook Twitter Pinterest Graph showing the performance of probabilistic weather forecasts for the northern hemisphere (as measured by the “ranked probability skill score”) over the last two decades. Green: 7 days. Red: 5 days. Blue: 3 days. Source: ECMWF

At the end of the day, the problem of limited computer power never goes away. It is fantastic news that the Met Office is getting a new supercomputer, but it just introduces the question of what to do with the extra resources. I do hope that some of this extensive new computing power will be used for improvements in probabilistic weather prediction for the UK.

It is impossible to be certain of what the future will hold, including the weather next week. However by acknowledging that this is the case, and instead striving to accurately indicate the uncertainty in our prediction, we can provide honest weather forecasts to the public who can then choose how to use the extra information.

To keep in touch with this blog please follow/like/add Alex on Twitter, Facebook, or Google+.