Havelt's team exploited the backdoors to remotely take control of the cameras. With the ability to view their output, change their direction, and zoom in and out, the Trustwave employees trained them on computer keyboards as employees in the unidentified company entered passwords. With the help of the cameras' 10x zoom, the pen testers were able to grab a "ton" of credentials and use them to log in to the company's network. From there, the employees escalated privileges to gain administrative control of the network. (The employees later reported the vulnerability to the camera manufacturer, resulting in the eventual release of this security advisory.)

We "ended up with domain admin on the internal network just because [the client] left these cameras on the Internet," Havelt said during a talk at last year's RSA conference.

Havelt recalled a separate engagement in the last 12 months that involved a different client. After his team gained access to a system that was on the company's internal network, the hired hackers injected malicious code into webpages regularly accessed by the company's developers. The malicious Java applet exploited a recently discovered vulnerability in the Java software framework that Oracle had yet to patch. With full access to one of the developer's machines, the payload installed a new set of cryptographic keys that was authorized to access the company's servers using the SSH, or secure shell protocol. With that significant toehold established, the pen testers were able to escalate their control over the client's network.

Adriel Desautels, CEO of pen testing firm Netragard, is also no stranger to the use of zero-day exploits, although he said he's often able to infect his clients using less sophisticated methods. During a recent engagement for a sensitive governmental agency located in the US, for instance, his team used social engineering to trick an agency employee into clicking on a link. The link, unbeknownst to the employee, installed "Radon," which is the name of pseudo-malware designed by Netragard to allow employees the same kind of sophisticated access many state-sponsored hackers behind espionage campaigns have.

With the employee's desktop computer infected, Radon rummaged through the agency's network and added malicious commands to the "batch file" every computer ran when it logged in. The modified file caused each computer to also become infected with Radon. Seizing control of hundreds of independent machines gave the Netragard hackers a higher likelihood of maintaining persistence over the network, even in the event that the initial infection was discovered and cleaned up.

"Eventually, it was game over," Desautels told Ars. "We had more control over their network than they did. That's how you do it. You don't just infect one system and stick it in their network and then try to infect the company. That doesn't guarantee you're going to be successful."

Desautels praised the architects of Operation Loopback because Facebook "did more than most other companies in this industry will do." But he went on to say that the engagement was significantly more limited than most attacks waged by well-funded and experienced hackers who are intent on penetrating a Fortune 500 company.

"If this were a real attack, they probably would have gone after multiple employees, especially with a zero day," he explained. "Why target one user when you have potentially hundreds of users you can target and get hundreds of points of entry?"

Facebook, he continued, "probably got some good insight. But [the engagement] is not nearly as realistic as it would be if it was a nation-state attack just because [Operation Loopback] was very singular."

Stress testing Facebook's incident response

To be fair, the drill Facebook executives devised wasn't intended to replicate every characteristic of a real-world attack. Instead, the executives wanted to develop employees' ability to work together to respond to an attack that could have a catastrophic effect on the site's security. Sullivan, Facebook's CSO, calls it a "stress test" of his incident response team.

"The team had grown substantially in the prior year, and we wanted to see if everyone is going to start screaming at each other or blaming each other because 'your logging system broke,' or 'your automated alerting should have triggered over here.' That was the human side of the test."

Operation Loopback also wasn't the first drill to test employees' ability to respond effectively in times of crisis. Six months earlier, McGeehan, the company's security director, installed a host of powerful hacking tools on a laptop computer, connected it to the Facebook internal wireless network, and stashed it behind a supply cabinet in a public hallway. A few days later, employees with the company's physical security team reported the discovery of the mysterious laptop to the security team, touching off another tense response. Over the following day, employees scouring server logs found the computer's MAC, or media access control, address had accessed key parts of Facebook's network.

"The first thing is: 'Oh my God. Panic,'" McGeehan said as he recalled his team's response to the incident. For almost 24 hours, the situation gave most employees every indication of being real. "As we're dealing with this, we realize that our network has been intruded on by some bad guy. Everyone in this room [is] thinking about 'how are we going to tear down our entire network? How are we going to basically deal with the worse-case scenario as a security incident?"

To ratchet up the stress even further, the drill organizers sent an e-mail to members of Facebook's security team a few hours after the laptop was disconnected from the Facebook network. The e-mail purported to come from members of what's known as the Koobface Gang, whose members last year were identified as the perpetrators of virulent malware that spread over the social networking site. It made a series of demands of Facebook and promised serious reprisals if they weren't met.

With Project Vampire, as the drill was dubbed, the employees worked a full 24 hours before they learned it wasn't a real hack.

"We felt it was a necessary thing to have a great security team to put them through this kind of stuff," Sullivan explained. The organizers made an exception, however, when early in the drill, an employee said the magnitude of the intrusion he was investigating would require him to cancel a vacation that was scheduled to begin the following week. McGeehan pulled the employee aside and explained it was only a drill and then instructed him to keep that information private.

Drills that use real zero-day vulnerabilities, require outside penetration testing firms, and suck up hundreds or thousands of man hours on non-production activities are expensive to carry out. But in a post-Operation Aurora world, where companies as security-savvy as Google and RSA are hacked and ransacked of valuable data, it is becoming increasingly necessary.

"These things used to be unheard of when back when, except for governmental type organizations," Trustwave's Havelt said. "Now, you're seeing this more in the private sector. It's good to see. If it were any other industry and it was any other critical function of a product not doing this you'd have people screaming that [the companies] were negligent and wanting to sue them left and right."