“In 2002, we had 14,191 servers with single-core, two-socket CPUs, totaling 28,000 cores,” Krishnapura says. “Now we have 260,000 servers, we have 2 million cores, more than 315 petabytes of storage and more than half a billion network ports within the data center.”

While Krishnapura talks, I become aware of something relatively unusual about the facility: “You're already sweating, because it's very hot,” Krishnapura observes.

When retrofitting D2, Krishnapura read a paper from Google that revealed the search giant operates its facilities at 78°F (25.5°C), in the cold aisle. “We said 'why limit it at that? What's the maximum we can go to?'” All available IT equipment supported inlet temperatures of up to 95°F (35°C), so the company settled on a cold aisle target of 91°F (32.7°C).

“It ranges between around 78-91°F in the cold aisle, depending on the outside temperature. The hot aisle is usually 20-30°F hotter.”

Looking up, Krishnapura says another difference is the height. “Industry-standard full IT racks are 42U, roughly 6ft. We are much taller, our racks are 60U, it's 9ft.” They are also slimmer: instead of the standard 24-inch format, they are trimmed to 20 inches, allowing for a few more racks to be crammed in.

“In 50 linear feet, where you can put 25 standard racks, we can put 30 of them. And as they're taller, we can put a lot more servers: each rack supports all the way up to 280 servers, and each rack can support up to 43kW peak power load.”

These are used internally for many of the things one would expect from a large enterprise, from running SAP workloads, to hosting webservers, to running Intel's video conferencing tools.

They are also used to design the company’s chips. “We use it for all the pathfinding to 7nm, to 5nm, some of the quantum physics algorithms - how the electrons scatter - all of that,” Krishnapura says.

By using Intel products in the company's facilities at scale, Krishnapura is additionally able to “give feedback back to the data center chip design business - we want to eat our own dogfood and learn from it.”

This creates a “feedback loop into the data center business, about how do we innovate and what kind of chips we want to make,” Krishnapura says. “It could be FPGAs from our Altera acquisition, or it could be the discrete graphics which we are working on, or it could be the other accelerators like Nervana.”