Government officials have been vague in their testimony about the data breaches—there was apparently more than one—at the Office of Personnel Management. But on Thursday, officials from OPM, the Department of Homeland Security, and the Department of the Interior revealed new information that indicates at least two separate systems were compromised by attackers within OPM's and Interior's networks.

OPM has not yet revealed the full extent of the data exposed by the attack, but initial actions by the agency in response to the breaches indicate information of as many as 3.2 million federal employees (both current federal employees and retirees) was exposed. However, new estimates in light of this week's revelations have soared, estimating as many as 14 million people in and outside government will be affected by the breach—including uniformed military and intelligence personnel. It is, essentially, the biggest potential "doxing" in history. And if true, personal details from nearly everyone who works for the government in some capacity may now be in the hands of a foreign government. This fallout is the culmination of years of issues such as reliance on outdated software and contracting large swaths of security work elsewhere (including China).

The OPM breaches themselves are cause for major concerns, but there are signs that these are not isolated incidents. "We see supporting evidence that these attacks are related to the group that launched the attack on Anthem [the large health insurer breached earlier this year]," said Tom Parker, chief technology officer of the information security company FusionX. "And there was a breach at United Airlines that's potentially correlated as well." When pulled together into an analytical database, the information could essentially become a LinkedIn for spies, providing a foreign intelligence organization with a way to find individuals with the right job titles, the right connections, and traits that might make them more susceptible to recruitment or compromise.

Preliminary evidence points to a group dubbed by Crowdstrike as "Deep Panda," a Chinese cyber-espionage group. In the past, the group has used Windows PowerShell attacks to implant remote access tools (RATs) on Windows desktops and servers. It is this malware that investigators are believed to have discovered on OPM's network and in the Department of the Interior's data center.

Handing out bandages

The two systems breached were the Electronic Official Personnel Folder (eOPF) system, an entity hosted for OPM at the Department of the Interior's shared service data center, and the central database behind "EPIC," the suite of software used by OPM's Federal Investigative Service in order to collect data for government employee and contractor background investigations.

Ars contacted both OPM and DHS while researching this story, but officials at both agencies refused to confirm or deny that these systems were part of the breach due to the ongoing investigation. However, sources familiar with OPM projects identified these systems as the ones most likely to be at the heart of the breaches.

In the weeks following the breach discovery, OPM officials scrambled to find a contractor to handle the "Privacy Act event." The organization issued a call in late May and awarded a contract five days later (on June 2) to Winvale Group, a Washington, DC-based technology services company that also helps businesses sell services to the government. OPM classified the transaction as a blanket purchase agreement to allow for multiple additional purchases. The $20.8 million "first call" was for 3.2 million "units" of credit monitoring and identity theft recovery services, indicating the agency's early assessment of how many individuals might have been affected by the breach.

The Winvale Group may get a lot more business based on OPM Director Katherine Archuleta's statement to the House Government Oversight Committee this week. "In early May, the interagency incident response team shared with relevant agencies that the exposure of personnel records had occurred," Archuleta said. "During the course of the ongoing investigation, the interagency incident response team concluded—later in May—that additional systems were likely compromised, also at an earlier date. This separate incident—which also predated deployment of our new security tools and capabilities—remains under investigation by OPM and our interagency partners. In early June, the interagency response team shared with relevant agencies that there was a high degree of confidence that OPM systems related to background investigations of current, former, and prospective Federal government employees, and those for whom a federal background investigation was conducted, may have been compromised."

To date, OPM has no idea how many individuals' background investigations were exposed. All Archuleta said was that the agency was "committed to notifying those individuals whose information may have been compromised as soon as practicable."

In the meantime, the Obama administration has ordered a “30-day Cybersecurity Sprint." Agencies must perform vulnerability testing and patch existing holes in security. They must prune the number of privileged user accounts and expand adoption of multifactor authentication for all systems. The Department of Defense and intelligence community have led the way on that last requirement, but many civilian agencies (such as OPM) have been slow to put it in place.

Just how much this "sprint" will improve government security remains to be seen, especially since agencies such as OPM have been repeatedly warned in the past about minimum "security hygiene." Thirty days is not likely enough time to correct a decade-plus of neglect of antiquated systems, poor leadership, and spotty attempts at modernization.

Employees must wash hands

OPM is not alone in neglecting basic security guidelines spelled out for them by both federal regulations and executive orders for much of the past decade. Even those agencies that have implemented systems to comply with the letter of FISMA (Federal Information and Security Management Act) and other regulations have had problems keeping on point because of the constantly changing nature of information security threats. And the complex plaque of information systems that agencies have built up often defies any sort of security management because the vendors who built many of the systems have long since disappeared.

By and large, government agencies in the last 20 years have become increasingly dependent on outside contractors to provide the most basic of information technology services—especially smaller agencies like OPM. The result has been a patchwork IT systems and security, and the Office of the CIO at OPM has a direct hand in fewer and fewer projects. Of the 47 major IT systems at OPM, 22 of them are currently run by contractors. OPM's security team has limited visibility into these outside projects, but even the internally operated systems were found to be lacking in terms of basic security measures.

While OPM instituted continuous monitoring of some systems using security information and event management (SIEM) tools, those tools covered only 80 percent of OPM's systems according to a fiscal year 2014 audit by OPM's Internal Office of the Inspector General (OIG) audit team. And as of October 2014, monitoring didn't yet include contractor-operated systems, according to the same organizational oversight.

"The OCIO (Office of Chief Information Officer) achieved the FY 2014 milestones outlined in the roadmap which included quarterly reporting for high impact systems," the OPM OIG reported in its audit. "The next stage in the OCIO’s plan involves requiring continuous monitoring by contractor-operated systems and implementation of the DHS Continuous Diagnostic and Mitigation program." In other words, OPM had no idea what was going on inside contractor-provided networks and only a limited grasp on what was going on within its own network.

There were significant gaps in OPM's security testing as well. Seven major systems out of 25 had inadequate documentation of security testing—four of which were systems directly maintained by the OPM's internal IT department. Three out of the 22 contractor-operated systems had not been tested in the last year; the remainder had only been tested once a year.

The greatest lapse within OPM's security, perhaps, is the way that it has handled user authentication. The OPM IG report has found progress on access controls, including the use of multi-factor authentication to access OPM's virtual private networks and even to log into workstations using Personal Identity Verification (PIV) card readers—essentially guarding the entry points into the OPM network. But "none of the agency's 47 major applications require PIV authentication," the Office of the Inspector General reported, a violation of an Office of Management and Budget mandate for federal systems.

OPM's Office of the CIO responded that "in [fiscal year] 15 we will continue to implement PIV authentication for major systems."

But OPM's systems, including central user authentication services used by most of the agency's applications, and the entirety of EPIC, were also operating without authorization—meaning, the systems had not been fully vetted for security, and were not even technically supposed to be in use. The OPM's Inspector General report recommended that EPIC and other systems that were operating without "Authority to Operate" (ATO) be shut down until they were judged secure, calling the systems' poor auditing a national security concern.

Ironically, federal officials have been blaming the messenger to some degree through anonymous statements to the press. NPR reported that investigators were looking into whether the IG report "tipped off hackers to some of the agency's vulnerabilities," and reporter Dina Temple-Raston found that investigators believed the attack came "about a month" after the IG report was published. "Among the things the inspector general found that could have helped hackers was that nearly a quarter of the agency's systems did not have valid authorization procedures," she said. "The reason that's important is because one of the departments that didn't have the correct procedures was the Federal Investigative Services. That's the group responsible for background investigations of federal employees. So that data's very sensitive, and as we know now, this is one of the databases that was hacked."

But those problems had been well-documented prior to the 2014 IG report. Attacks on two OPM investigative contractors—USIS and KeyPoint—could have provided plenty of intelligence on just how bad OPM's systems were. Even a quick Web search would have given attackers plenty of ideas about how to get into OPM's sensitive systems. For example, the "secure" Web gateway to OPM's background investigation systems is a contractor-hosted website at an application service provider. That Web gateway is reached through a Windows Web server running JRun 4.0, Adobe's Java application server, as well as ColdFusion, a platform that has been used for a number of breached government servers in the past few years.

In 2013, someone hacked into Adobe and stole the ColdFusion source code. And Adobe dropped the JRun product line entirely in 2013—with extended "core" support ending in December of 2014. There is no evidence that OPM or its application provider had purchased expensive extended, dedicated support, but JRun would hardly be the only unsupported platform still used by OPM. The agency still has systems based on Windows XP (supported under a custom support agreement with Microsoft), and many of the core systems run by the agency are based on mainframe applications that haven't been updated since their COBOL code was fixed for the Y2K bug in the late 1990s.

It would be incorrect to say that these older systems (especially the COBOL code) couldn't be updated to support encryption, however. There are numerous software libraries that can be used to integrate encryption schemes into older applications, including libraries from PKWare. Other government agencies and financial institutions already utilize such software, according to Matt Little, VP of Product Development at PKWare. The problem is that, as DHS Assistant Secretary for Cybersecurity Andy Ozment noted during his testimony, OPM didn't have the kind of authentication infrastructure in place for its major applications to take advantage of encryption in the first place. Encryption, he said, would "not have helped in this case."

Since multi-factor authentication and encryption were not integrated into any of OPM's 47 major applications, all an attacker had to do was to gain access to a system on the network—nearly any system. Based on the testimony before Congress and other publicly available data, we know that hackers found at least two systems and were able to easily expand their access laterally within OPM and then contractor and service provider networks afterward.

"There's a process failure in every spot there," said PKWare's Little. "It's just bad security controls. It looks ridiculous—they didn't even have basic IP (network) access controls. This is not something we typically see in a serious security customer."

As Ars has reported, those problems were not just found at OPM itself. Contractors working for the agency may have introduced some unique security issues of their own, including employing Chinese nationals—some working from overseas—as part of subcontracting teams. Allegedly, that project was an implementation of SAP's SuccessFactors software, undertaken by a systems integrator for OPM and affiliated agencies, and included access to employee personnel data for the Department of Energy, the Transportation Security Agency, and others. SuccessFactors is used as part of a human resources system called the Talent Management System (TMS), "an integrated learning management and performance management system based on the industry leading SAP/Plateau/Success Factors software" hosted for multiple agencies by a data center at the Department of the Interior. SAP could not provide information about the program, the integrator, or even confirm that Interior or OPM were a customer without OPM authorization.