Predicting the weather accurately is a hard enough computing problem. Predicting the weather for a specific location down to a square kilometer—and how it will affect the people and infrastructure there—is a problem of a much different sort. And it's that sort of "hyper-local" forecasting that IBM’s Deep Thunder aims to provide. Unlike the long-term strategic weather forecasts that many companies rely on to plan business, Deep Thunder is focused on much more short-term forecasts, predicting everything from where flooding and downed power lines will likely occur to where winds will be too high for parade balloons up to 84 hours into the future.

IBM executives hope Deep Thunder, which has been in development since 1996, will become a must-have tool for local governments, utility companies, and other organizations with weather-sensitive needs. By changing how weather data is displayed from the traditional meteorological map to a more manager-friendly approach tailored to the needs of a government or utility company, IBM researchers are hoping to get customers to buy into weather modeling as part of their operations management.

So far, major utilities and other commercial customers aren't biting. But the company has signed on at least two public-sector customers so far who are piloting their own Deep Thunder systems. One of them is the City of Rio de Janeiro, where officials are looking to beef up their emergency services preparedness in advance of the 2014 FIFA World Cup and 2016 Olympics.

In a recent demonstration for government officials using a test deployment of the system in the New York City metropolitan area, IBM showed how data from Deep Thunder could be deployed through a mobile application to give precise predictions about weather at a specific location, as well as alerts to specific risks based on the needs of the user. The "apps" IBM showed members of Congress and press last week, aren’t anything you’ll be able to download from Apple’s or Google’s online stores anytime soon, but the visualization technology used in them is being integrated into IBM's Intelligent Operations Center software and other existing operational environments.

While traditional weather forecasting can predict general weather conditions with some accuracy, it doesn’t always give government agencies and utilities the kind of information they can take action on. For example, the record snowfalls that fell in New York in January of 2011 shut down the city’s transit systems and airports, but had nearly no impact on utilities—because there were relatively little winds, and the snow didn’t stick to power lines. Still, Deep Thunder’s forecasting can sort through weather conditions and alert users to specific conditions that affect their operations.

Lloyd Treinish, the leader of the environmental science team at IBM’s Thomas J. Watson Research Lab, told Ars in an interview that the mobile app prototype—which runs on Apple iOS, Google’s Android OS, and in Web browsers—was indented as an exploration of how to disseminate the complex data that comes out of Deep Thunder’s forecasting. "Right now it’s set up to use our New York metro area forecast," he said. "If you use the default settings, based on your current location, it will give you the forecast at that location for 84 hours, updated every 12 hours. Alternatively, you can enter an address and get a forecast for where you’re going to be."

That data can be used in the field to make decisions, for example, about whether winds will be too high to do "overhead work" without having to rely on a dispatcher to make the call. Using the time and location data of forecast, maintenance crews could plan their work based on conditions at different sites, avoiding stoppages because of localized weather conditions. “You’d see the differences for JFK and La Guardia (airports) and from Central Park to lower Manhattan,” Treinish explained.

The iPad and Android interfaces, which IBM showed members of Congress and press last week, aren’t anything you’ll be able to buy from an app store. But the technology that creates the data behind them is already deployed as a result of two partnerships formed by IBM with government customers. The Rio de Janeiro effort is one of the partnerships that IBM launched last year, in the wake of a major weather-related disaster. In April 5, 2010 coastal storms in the Brazilian city caused massive mudslides, killing 200 people and leaving 15,000 homeless. And IBM's other partnership is with the University of Brunei Darussalam, which is using the technology at a national level for flood forecasting, and as part of a program to predict the impact of climate change on the country’s rainforests.

Models upon models

Predicting weather in such a highly localized fashion, whether it be in New York or Rio, is a much more complex problem than most weather forecasting models can handle. “Weather isn’t just driven by the atmosphere,” Treinish explained. “It’s also driven by how the atmosphere interacts with surface features. In urban areas, you have to look at how the urban environment impacts weather, and the sorts of feedback the atmosphere gets from highly urban environments, such as urban heat islands.”

Accurately modeling that sort of interaction requires much greater precision than weather models are capable of. While Deep Thunder’s weather models are relatively reliable at a square kilometer scale, “when you go to flooding forecasting, some of the simulation issues go to even finer scale,” Treinish said—using geographical and hydrological data that gets to resolutions of one square meter per data point. And for some predictions, such as high winds, the weather requires greater “vertical” resolution as well, to take the affect of features such as hills and buildings into account.

But collecting data to drive models of cubic-meter precision for weather is impossible, because there’s no way to either collect or process the sensor data required to feed such a beast. While it might be conceivable to cover a city with weather sensors, few cities have the IT infrastructure to handle that sort of an effort—and even those sensors wouldn’t be able to provide information about the conditions in outlying areas that will drive local weather. Even if the data was magically available for a one-meter resolution forecasting model, each step up in the “resolution” of forecasts in time and space can require an exponential increase in the computing power required to create the three-dimensional models. “If you want to double resolution, that requires eight times the compute power,” Treinish said. “If you need to increase the vertical resolution, it could be 10 or 20 times the compute power.”

To overcome that problem, IBM is using an approach for Deep Thunder that relies on “coupled models”—where one model’s data is used as an input for another’s—and historic statistical data and other data sources that are used to validate the results.

“For Rio, we’re running the weather models at four different resolutions at the same time, using telescoping grids” said Treinish. “We start off with a global model from NOAA, and from that zoom in on Rio—the horizontal resolution goes from 27 kilometers to 9, then to 3, then to 1. That's a way of balancing geographic area with the physics required to solve the business problem.” The precipitation data extracted from the 1-kilometer model is in turn fed into a 1-meter-resolution geographic and hydrologic model to forecast flooding, he explained.

By taking this layered model approach, the IBM Research team has been able to take what qualifies as a fairly big high-performance computing problem and shrink it down to a relatively small IT footprint of a few parallel processing systems—“a rackful of equipment,” Treinish said, for the level of targeted weather forecasting currently being done in Rio and other places. “The architecture is similar to the large systems that IBM sells to NOAA and Air Force’s weather agency, but just much smaller.”

The Rio configuration includes a few POWER7 systems interconnected with an Infiniband switch. For larger applications of the technology, the compute power can scale up quickly, however—a nationwide meteorological monitoring system based on Deep Thunder being run at the University of Brunei Darussalam runs on an IBM Blue Gene supercomputer.

Building and tweaking Deep Thunder’s mass of models requires a lot of data sources. “We use whatever data we can get our hands on,” Treinish said. That includes public sources such as satellite and other sensor data from NOAA, NASA, the US Geological Survey, and the European Space Agency. “In the work we’ve done at Rio and Brunei, we’re utilizing remote sensing of surface temperatures.

IBM is also using private data sources to feed Deep Thunder—or at least validate and tune its forecasts, depending on the type of data available. In addition to connecting to any weather sensors a municipality or company may have in place, IBM has also connected to partners for validation data; in New York, for example, the company has teamed with Earth Networks to pull data from its WeatherBug network of sensors. “Their sensors are clustered in the US around urban and suburban areas,” Treinish said, “but so are a lot of our applications. We get the data every five minutes—it’s very valuable to validating and improving the model.”