In August of 2011, while in the middle of upgrading its network security monitoring, the Federal Communications Commission discovered it had already been hacked. Over the next month, the commission's IT staff and outside contractors worked to identify the source of the breach, finding an unspecified number of PCs infected with backdoor malware.

After pulling the infected systems from the network, the FCC determined it needed to do something dramatic to fix the significant security holes in its internal networks that allowed the malware in. The organization began pulling together a $10 million "Enhanced Secured Network" project to accomplish that.

But things did not go well with ESN. In January, a little less than a year after the FCC presented its plan of action to the House and Senate's respective Appropriations Committees, a Government Accountability Office audit of the project, released publicly last week, found that the FCC essentially dumped that $10 million in a hole. The ESN effort failed to properly implement the fixes, and it left software and systems put in place misconfigured—even failing to take advantage of all the features of the malware protection the commission had selected, leaving its workstations still vulnerable to attack. In fact, the full extent of the problems is so bad the GAO's entire findings have been restricted to limited distribution.

"As a result of these and other deficiencies, FCC faces an unnecessary risk that individuals could gain unauthorized access to its sensitive systems and information," the report concluded. And much of the work done to deploy the security system must be redone before the FCC's systems approach anything resembling the security goals set for the project.

The FCC's leadership acknowledges there's a lot left to be done. "The GAO's review of this project covers a period of time during which the Commission faced an unusual level of urgency, and we look forward to sharing our further progress with Congress and GAO at a later time, when these security initiatives are more fully deployed and developed," FCC Managing Director David Robbins wrote in response to the GAO's findings. But the commission also has some personnel issues to address—all of this is transpiring as the FCC looks for a new chief information officer. Ironically, the FCC's CIO Robert Naylor stepped down in January to take a new job; he is now the CIO of a cyber security firm that caters to the intelligence community.

Measure once, cut twice

The FCC is a small organization as government agencies go, with about 2,000 employees and a budget request for 2013 of $340 million. It relies heavily on outside help for its IT operations—and on more outside help to figure out how to buy that help. The aquisition of the ESN project was managed by Octo Consulting Group, a company led by three former Gartner executives and the former CIO of the Department of Agriculture's Forest Service. The company claims on its website to have "designed the FCC Cyber Security Strategy, and managed and executed three defining Cyber Security contracts." The consulting firm also provided contracting support for the FCC's CIO as all of its major IT support contracts were preparing to expire mid-2012.

Update: "Octo was responsible for providing 'acquisition support to the FCC' for the ESN contract (i.e. Assisting FCC Acquisition & Contracts personnel with developing the Statement of Work used to acquire the hardware and services for the $10M ESN contract you referenced)," Octo Consulting Group president Mehul Sanghani said in an email to Ars. ""Once the contract was awarded, Octo was also tasked with providing project management support to supplement the FCC IT staff that was tasked with overseeing the work." The actual work on ESN was done by MicroTech and subcontractor Booz Allen Hamilton.

At the time of the discovery of the network intrusion in 2011, the FCC's network security was dated at best. The ESN project, which was originally projected to be completed this month, is intended to "enhance and augment FCC’s existing security controls through changes to the network architecture and by implementing, among other things, additional intrusion detection tools, network firewalls, and audit and monitoring tools," according to the GAO. The program was also supposed to provide the FCC with an ongoing "cyber threat analysis and mitigation program" that would do continuous risk assessment and reduction and control the damage from attacks that managed to breach the commission's security measures.

Contracts to do the work on ESN were awarded in April of 2012, just two months after plans for the project were submitted to Congress. By June, all of the security hardware and software licenses had been purchased. Implementation was in full swing.

But apparently the work was done so quickly that no one bothered to check it. While new security hardware and software was deployed, the GAO found that "FCC did not effectively implement or securely configure key security tools and devices to protect these users and its information against cyber attacks… Certain boundary protection controls were configured in a manner that limited the effectiveness of network monitoring controls."

The rush to get things in place also led to some other sloppy work. The GAO's auditors found that passwords to gain access to some of the network monitoring systems "were not always strongly encrypted." And while tools had been put in place to detect malware and block malicious network traffic, the tools had been left only partially configured.

The mishandling of security is being raised as an issue by some who do business with the FCC, especially because news of the original breach was never disclosed to the public—even as the FCC was formulating a proposed a rule that would require people with commercial interests in broadcast stations to submit their social security numbers to an FCC database. As Harry Cole, a communications lawyer with the firm Fletcher, Heald, and Hildreth put it in a post to the firm's blog," it seems extraordinarily inappropriate for the Commission, knowing of those vulnerabilities, to then propose that a huge number of folks must provide to the FCC the crown jewels of their identity, their social security numbers."

Listing image by Open Gate Farm