Your Tax Dollars At Work: How Commerce Dept. Spent $2.7 Million Cleaning Out Two Malware-Infected Computers

from the burning-a-hole-in-taxpayers'-pockets dept

The cyber-Pearl Harbor is upon us and the only way to defeat it is to sink our own ships at the first sign of invasion. This is the sort of thing that happens when the legislators and advisors with the loudest voices value paranoia over rational strategy. The Department of Commerce, aided by a tragicomic string of errors, managed to almost stamp out its malware problem.

The Commerce Department's Economic Development Administration spent almost half of its IT budget last year to remediate a cyber attack that barely happened.



EDA's drastic steps to limit the damage by shutting down much of the access to the main Herbert Hoover Building network ended up costing the agency more than $2.7 million to clean up and reconfigure its network and computers. The IG said the bureau destroyed more than $170,000 in IT equipment, including desktop computers, printers, keyboards and mice.

In an effort to identify infected components, DOC CIRT’s (Dept. of Commerce Computer Incident Response Team) incident handler requested network logging information. However, the incident handler unknowingly requested the wrong network logging information... Instead of providing EDA a list of potentially infected components, the incident handler mistakenly provided EDA a list of 146 components within its network boundary. Accordingly, EDA believed it faced a substantial malware infection.

The HCHB network staff member then performed the appropriate analysis identifying only two components exhibiting the malicious behavior in US-CERT’s alert.

DOC CIRT’s second incident notification did not clearly explain that the first incident notification was inaccurate. As a result, EDA continued to believe a widespread malware infection was affecting its systems.



Specifically, the second incident notification began by stating the information previously provided about the incident was correct. EDA interpreted the statement as confirmation of the first incident notification, when DOC CIRT’s incident handler simply meant to confirm EDA was the agency identified in US-CERT’s alert. Nowhere in the notification or attachment does the DOC CIRT incident handler identify that there was a mistake or change to the previously provided information.



Although the incident notification’s attachment correctly identified only 2 components exhibiting suspicious behavior—not the 146 components that DOC CIRT initially identified—the name of the second incident notification’s attachment exactly matched the first incident notification’s attachment, obscuring the clarification.

Thank you for reading this Techdirt post. With so many things competing for everyone’s attention these days, we really appreciate you giving us your time. We work hard every day to put quality content out there for our community. Techdirt is one of the few remaining truly independent media outlets. We do not have a giant corporation behind us, and we rely heavily on our community to support us, in an age when advertisers are increasingly uninterested in sponsoring small, independent sites — especially a site like ours that is unwilling to pull punches in its reporting and analysis. While other websites have resorted to paywalls, registration requirements, and increasingly annoying/intrusive advertising, we have always kept Techdirt open and available to anyone. But in order to continue doing so, we need your support. We offer a variety of ways for our readers to support us, from direct donations to special subscriptions and cool merchandise — and every little bit helps. Thank you.

–The Techdirt Team

Also included in the mass destruction were cameras and TVs. It wasn't just cyber-paranoia that led to this hardware cull. There was plenty of miscommunication too, along with the usual doses of bureaucratic clumsiness. The Inspector General's report breaks down the chain of missteps, which all began with a response team member grabbing the wrong network info.Yes. Much like "Reply" and "Reply All" willget the job done, onlyis the correct choice when firing off a devastating critique of your soon-to-be-former coworkers. The same goes for network logs. One shows you the correct info. The other "indicates" that more than half the EDA's computers are suffering from a malware infection.DOC CIRT did try to get this fixed, pointing out the error to the handling team and re-running the analysis using thenetwork log. Turns out, the original estimate wasoff.This new data in hand, a notification was sent out ostensibly to clear things up, but this too was mishandled so badly someone unfamiliar with bureaucratic ineptitude might be inclined to suspect sabotage.For five weeks, things went from bad to worse to comically tragic to tragically comic to full-scale computercide. Looking atlist (2 components), DOC CIRT asked the EDA to attempt containment by reimaging the infected items. Looking atlist (146 components), the EDA responded that reimaging half its devices would be "unfeasible." Taking a look at the EDA's list (from the first, mistaken network log analysis), DOC CIRT assumed the EDA had received additional analysis indicating the malware had spread, and changed its recommendations accordingly.Finally, both departments were on the same (but entirely wrong) page and scaled up the response accordingly. A copy went to the DHS, stating that "over 50%" of the EDA's devices were infected. The DHS then accepted this without seeking independent confirmation. The NSA cranked out its own concerned report, quoting heavily from the DHS report (which was still in draft form),were based on DOC CIRT's first erroneous report. This went undetected for over a year, until the OIG informed the involved agencies of its findings in December 2012.The end result? The EDA and DOC CIRT worked together, attempting to head off a "severe" malware threat before it spread to other connected government computers. Despite gathering more information from outside consultants that indicated the malware was neither "persistent" nor a threat to migrate, the two agencies began destroying devices in May of 2012, finally stopping three months later when the "break stuff" budget had been exhausted.Fortunately for the agencies, taxpayers and the surviving equipment (valued at over $3 million), the OIG's findings were brought to the agencies' attention before the fiscal year began and a new "break stuff" budget approved. All in all, the EDA spent over $2.7 million fighting a malware "infection" confined to two computers.There's nothing in this report that makes the EDA look good. A chart on page 8 shows the EDA has persistently ignored the OIG's recommendations on agency computer security, with some assessments going back as far as 2006. It's no surprise it managed to (along with the Dept. of Commerce's response team) transform a 2-computer infection into a nearly $3 million catastrophe.

Filed Under: commerce department, malware, miscommunication