2:18 p.m. update, Thursday: It appears that some satellite imagery has been restored, but that the feed of satellite data used for models remains somewhat incomplete. “The last update I received was that we are not yet fully restored, but there has been some improvement in datasets for the models,” said Chris Vaccaro, National Weather Service spokesperson. A technical notice from the National Weather Service stated the “majority” of satellite data has become available for the models.

Original post, from 3:40 p.m. Wednesday



Latest satellite image posted to NOAA’s Geostationary Satellite Server Web site was posted at 5:45p on October 20. (NOAA)

Since at least Tuesday, some satellite data – an important input to weather prediction models – has stopped flowing into the National Weather Service due to an apparent network outage.

At 1 p.m. today, the National Weather Service’s National Center for Environmental Prediction (NCEP) issued the following statement cautioning the outage could impact forecast quality:

NCEP HAS NOT RECEIVED A FULL FEED OF SATELLITE DATA FOR INPUT INTO THE NUMERICAL MODELS SINCE 22/0000Z…POTENTIALLY IMPACTING THE MODEL FORECASTS.NESDIS AND NCEP ARE INVESTIGATING THE ROOT CAUSE OF THE ISSUE. ONCE THE SITUATION IS RESOLVED ANOTHER MESSAGE WILL FOLLOW.

It is unclear if the data outage is only impacting the National Weather Service or whether it extends to other international modeling centers such as the European Centre for Medium-range Weather Forecasting, home of the top-performing European model, and Environment Canada.

Within the National Weather Service, the outage could affect the performance of National Weather Service’s Global Forecast System (GFS), North American (NAM), and HRRR (High-Resolution Rapid Refresh) models.

Chris Vaccaro, a spokesperson for the National Weather Service, said officials at NCEP told him these model forecasts can still be considered credible, despite the outage. “[T]here’s a lot of redundancy in the observing system that can help to offset the data loss and that the model guidance will have integrity and will be sound,” he said in an email.

A technical discussion from NCEP posted at 2:50 p.m. said the data loss was “significant”, but not so much to degrade forecasts.

Capital Weather Gang’s Steve Tracton, who worked with computer models over several decades at the National Weather Service, agreed the effects of the outage would be minimal in the short term, but could increase with time. “If satellite data are not returned within 24-36 hours, there is a chance of some forecast degradation though still likely small,” he said. “It will be interesting to assess after the fact.”

Another repercussion of the outage is that all or most of satellite imagery has stopped publishing to NOAA and National Weather Service Web sites; the “current” imagery is at least a day old. This imagery is used to track and analyze storms such as the Nor’easter affecting the East Coast and the tropical depression in the Gulf of Mexico. At the time of publication of this blog post, NOAA and the National Weather Service had not posted messages on relevant Web sites alerting users that the imagery is now obsolete.

Although the National Weather Service is not receiving the latest imagery, NASA satellite imagery Web sites appear up-to-date, indicating this is a National Weather Service-specific problem.

This outage is the latest in a string of network and information technology problems to afflict the National Weather Service in the last two years. In late August, the National Weather Service Web site crashed due to a barrage of data requests from an external Android application. And in May, a firewall upgrade crippled the Weather Service’s warning dissemination system.

There were also previous Web site failures before that, as Slate’s Eric Holthaus summarized:

Previous technicalities weren’t quite as serious, such as an unbelievably large flood warning in mid-April (coincidentally, the day before the movie Noah was released), and a website crash in early April (though warning products continued to be issued as normal through more traditional channels).

Note that in 2013, I penned the piece: Weather Service systems crumbling as extreme weather escalates

(Clarification, 5:30 p.m.: It is possible the data outage is affecting modeling centers outside the National Weather Service; a statement in the original post published at 3:40 p.m. stating the outage likely only applied to the National Weather Service was amended.)