Hurricane Sandy reminded us that Manhattan hosts a surprising number of data centers. The expense of real estate in Manhattan and the potential for flooding would, you might think, drive data center operators to less expensive and safer climes.

But the data centers face demands from customers who want low latency and close proximity to their servers, and many of those customers are in New York. Even the people who run the data centers say they're surprised the demand is so high.

“Just the sheer number of data centers on Manhattan is quite surprising,” said Steven Orchard, senior VP of development and operations at the data center hosting company Internap. “There's a real need and want—whether it’s a need we could all likely debate, but there’s a real want—by a certain customer base to have their infrastructure in Manhattan. It’s one of our largest infrastructures because customers request and require that the majority of their gear be there.”

And so Internap has since the late 1990s operated a data center at 75 Broad Street in New York City, a building hit hard by Hurricane Sandy on the night of October 29. “The entire building was engulfed,” Orchard said. Water was rushing down the elevator shafts, filling the lobby with anywhere from one to three feet of water.

“There were evacuation orders. They were not mandatory."

In the basement, salt water hit Internap’s fuel pumps, taking them out of commission, and a 10,000-gallon fuel tank suffered damage when water penetrated from the outside of the building through a breather pipe. With utility power out and that fuel supply unavailable, Internap turned to its 1,200-gallon header tank on the mezzanine level (where it also stores its generators), providing enough power to keep servers running for about 12 hours.

Internap has 47,000 square feet of office and data center space at 75 Broad, with thousands of servers on the 14th floor. Throughout Sandy and its aftermath, the company had seven or eight people on site. But in a flooded building with fuel floating around, “people were a little bit worried and on the verge of perhaps having to evacuate because of fear of fire,” Orchard said.

Orchard spent a few days in New York after the storm hit but was in his home base of Atlanta in the early parts of Sandy and its aftermath. “There were evacuation orders. They were not mandatory, at least we were not told that they were mandatory. We knew if it got worse we were going to get people out of there because life safety is more important than anything else,” he said.

Internap’s data center staff didn’t want to abandon ship, even though the building was partially in the city’s emergency evacuation zone. “We took their word for it; we were all on the conference bridge together keeping in constant contact. They saw the water come into the lobby, and then they started to see it recede an hour or an hour and a half later and got more comfortable with where they were," Orchard said. "We had preparations for them to get out, but they made the judgment call and were able to struggle through.”

The servers went offline once the header tank on the mezzanine level exhausted its fuel, and they stayed that way for 12 hours until an emergency supply of fuel arrived.

Hospitals and critical care facilities were at the “top of the list for local fuel distributors,” Orchard said. “They took precedence over everyone else, obviously, and rightfully so.” Internap ended up having one of its vendors drive two trucks and pumps up from the Baltimore area. It took a few hours to drive up the Eastern Seaboard and “probably a couple more driving around in circles” to find a way onto Manhattan, he said.

The trucks arrived around 8pm on October 30. But getting fuel to the mezzanine level wasn’t so simple. A new fitting had to be fabricated to connect the hose to the fuel tank. “It’s just finding the right parts and pieces and perhaps having to either weld them together or jerry-rig something out of six pieces and make it into one piece so it's actually functional,” Orchard said.

Ultimately, the generator was fired up again and power flowed back to the customer’s servers by 11:30pm that night. They stayed on generator power for 10 days, until late afternoon on Saturday, November 10, when the site finally transferred to utility power from Con Edison.

In the meantime, Internap burned 15,000 to 20,000 gallons worth of fuel, keeping usage as low as possible by turning off anything non-critical (lights, coffee pots, etc.). The story was very similar for Peer1 Hosting, which also has data center space at 75 Broad Street. The two companies helped each other out where they could, but they were each responsible for their own generators and tanks. Peer1 staff, volunteers, and customers had it rougher in at least one way, as they were having to carry buckets of fuel up 18 flights of stairs to keep a tank there filled.

“They had their brigade of people carrying five-gallon buckets up 18 flights of stairs or whatever it was to get to their auxiliary generator. Quite a task by those guys,” Orchard said.

There’s still much work to be done to clean up from Sandy and harden the data center’s power supply against future storms. Although utility power is back, Internap is keeping a fuel truck outside the building just in case it’s needed, and the company expects it to stay there for a couple of weeks while the large fuel tank in the basement is repaired. “We are having to pay for that tanker to sit on the street. It does run into the four figures a day,” Orchard said.

Given its 10,000-gallon capacity, Internap’s primary fuel tank isn’t likely to be moved. But Orchard said Internap plans to weather-proof it as much as possible.

“Longer-term we'd like to see if there is a different type of tank that can have submersible fuel pumps or the like,” he said. If that doesn’t work, “we will move electrical infrastructure and pumps up to the mezzanine level or higher ground if at all possible.” In the short term, Internap is replacing the pumps and moving them to a less vulnerable part of the basement.

At 75 Broad, companies like Internap and Peer1 all have their own infrastructure. But at another data center site in the city, 111 8th Avenue (a Google-owned building), the building management owns and operates the fuel system for everyone.

The situation was less dire there. Internap struggled through three or so hours of downtime, but the building was getting 25,000 gallons of fuel delivered per day to keep generators running. Utility power was restored to that site by November 3.

Customers inconvenienced, but understanding

Internap manages servers on behalf of many of its customers and also has customers that manage their own servers within the Internap data centers. Many customers come to the New York data centers each day to visit and work on their equipment.

During the power outage, elevators weren’t working and some floors were completely dark. “Getting customers to walk up and down 14 flights was impossible—not allowed,” Orchard said. So Internap had part of its staff on hand to handle requests from customers who needed to make a change to their servers.

In addition to wanting physical access to equipment, some businesses in New York desire extremely low latency. In general, financial services and media customers are the ones who use the New York data centers because of latency concerns. But it's not just them; a business in New York might have a file server hosted with Internap because it wants “almost real-time response” to nearby office space, Orchard said.

During the outage, Internap helped customers move workloads to its other data centers, such as Dallas or Santa Clara.

“Quite a few customers had disaster recovery plans in place,” Orchard said. “We helped folks implement those in other sites. Other customers who may not be as formalized on their DR plans, we were able to get them set up in a temporary fashion. Perhaps not a full-blown infrastructure like they had in New York, but we were able to get them up and operational in other sites.”

Power outages always lead to some peeved customers. But for the most part, Internap customers were patient.

“I would say for the big majority that’s the case,” Orchard said. “Everybody was very understanding, especially the local folks who saw the actual devastation.”

Listing image by Eric Konon.