At the entrance to a nondescript building on the sprawling Symantec campus in Silicon Valley, the company's Senior Director of Operations, Identity and Authentication, Paul Meijer, is presenting his badge and entering his personal identification number to get inside. A second door not far away requires him to repeat the process all over again. A dozen or so feet further is a third door, and this one requires him to press his index finger against a sensor to prove he's one of fewer than 100 Symantec employees permitted to enter.

As he negotiates a series of additional mazes inside, he comes upon still more security checkpoints. One room at the center of the building—inside two concentric squares protected by a double layer of metallic mesh that isn't easily drilled, cut, or welded, requires two authorized Symantec employees to enter. To enter, Meijer and a colleague must key in a PIN and show a fingerprint. Inside are cabinets housing special-purpose computer servers that neither of the two employees can open because the combination is held by a different class of employees. A separate room where digital certificates are generated under rigorous "key-signing ceremonies," also requires dual occupancy. To further ensure the security of the operation, the second employee who must accompany Meijer is one of fewer than two dozen people with the required access codes.

Welcome to Symantec's SSL certificate vault, the company's repository that's built to military-grade specifications. The assets protected here aren't made of gold, silver or any other tangible material. Rather, they're the secret mathematical keys in the public key infrastructure that forms the basis of virtually all encrypted communications between websites and end users. Ars Technica recently took a rare tour of one facility where Symantec mints, stores, and sometimes revokes keys on behalf of companies such as Amazon, PayPal, and British Telecommunications.

The security model of this digital Fort Knox in some ways resembles an onion, with each inner layer harder to access than the outer one preceding it. At the center are special computers known as hardware security modules (HSMs), which contain their own cryptography-dedicated processor and storage system. Unlike regular computers with keyboards, mice, and other peripherals, HSMs accept input only from small devices that look like something like Spock's tricorder. The key to unlock an HSM is so long that it's broken up into chunks and distributed to 16 separate employees and kept a closely guarded secret. To generate, store, retrieve, or revoke an SSL key, three of those employees must plug their unique physical key into the tricorder device and, sometimes, enter their chunk of a corresponding PIN into a device keypad, all while in the presence of the others.

"You should think of it as the holiest of holies within the public key infrastructure," Ralph Claar, Symantec Senior Manager of PKI Operations, said of the HSMs. "It would be similar to saying we wrote our ultra-secure computer system and no one person knows the password. We made three different people come up and each type four characters in a specific row. So now every time we need to log in as root those three people all need to come in and each one of them needs to type the four characters."

To extend Claar's metaphor, if the HSM is the highly privileged root account on a Unix computer, the tricorder device is the keyboard used to enter the multipart password, and the three Symantec employees together comprise the admin with the required password.

The room where keys are generated and revoked can be accessed only when two of a handful of Symantec employees both present their separate credentials. With the escort of these two employees, the three additional employees with the HSM keys are joined by Symantec customers for what's known as a "key ceremony." The process, which can last as long as 12 hours, painstakingly records every keystroke, mouse click and utterance, so the legitimacy and authenticity of the key can be documented throughout its lifetime. A video camera records the event so it can be played back should there ever be a question or challenge about the legitimacy of a key or key revocation.

Remember DigiNotar?

To be sure, there's nothing in this facility that's unique to Symantec's certificate authority business. Military grade data centers, HSMs, and three-factor authentication are fairly standard security measures observers will find at most or all of the world's biggest 10 or so CAs, said Ryan Hurst, the chief technology officer at GlobalSign, one of Symantec's top competitors.

"The focus on these shiny, glitzy rooms is super important," Hurst told Ars. "These are super-valuable keys and anybody who's in this business or visits that room, if they get compromised, you're out of business and there's no getting back into business."

In August 2011, the world got a graphic demonstration of just how crucial security is for such key-generating systems. That's the month that security experts spotted a fraudulent SSL certificate for Google.com that gave attackers the ability to impersonate the website's mail service and other offerings. The counterfeit certificate was minted after attackers pierced the security of Netherlands-based DigiNotar and gained control of its certificate-issuing systems. Within a few days of the discovery, most of the major Web browsers issued updates to block the bogus certificate, but not before some 300,000 people, many located in Iran, had been exposed to the certificate as they accessed Gmail servers.

In a recently issued report, security firm Fox-IT said the attackers hijacked all eight servers that managed DigiNotar certificates even though they weren't directly connected to the Internet. The Web servers were breached as early as June 17, 2011, after the attackers targeted known vulnerabilities in the DotNetNuke content management system the servers ran. The attackers then spent the next two weeks working their way to more sensitive systems, in part by using the compromised systems to exchange malicious files with other servers on DigiNotar's network.

The certificate-generating servers, which were located in a "highly secured room" still couldn't be manipulated to issue counterfeit certificates because they required the use of a private cryptographic key stored on a smartcard that remained locked in a vault. It took another nine days for the intruders to work around this restriction, most likely by taking advantage of an automatically recurring process that used the private key to update certificate revocation lists. In all, the attackers generated 531 rogue certificates, most of which were trusted by all major browsers and worked for 140 unique domain names, including Twitter, Microsoft, Skype, and Mozilla. DigiNotar officials revoked most the forgeries after discovering the breach, but kept it a closely guarded secret. It wasn't until August that the breach became public knowledge after someone in Iran discovered the rogue Google certificate thanks to a feature known as certificate pinning that's included in the company's Chrome browser.

The perpetrators behind the attack were never publicly identified, since they took great care in covering their tracks. On one occasion, an intruder used an IP address originating from Iran, presumably in error. The same address was used to test whether a fraudulent Yahoo certificate would be accepted by major browsers. Based on the evidence, Fox-IT investigators speculate the attackers were located in Iran.

DigiNotar went out of business after its root certificate was removed from Internet Explorer, Mozilla Firefox, Google Chrome and other browsers. The episode underscores the fragility of the SSL system. Such threats are only growing.

"From day one, the CA industry has been under attack," said Hurst. "The one thing I'd say has changed is the attacker is more advanced now. Today, we have to worry about nation states. That wasn't part of out threat model in 1995."