How a Routine Malware Outbreak Cost One Government Agency Millions

Some days you just have to stop and pay some attention to gross incompetence when you see it. That was my reaction to a story first reported on The Verge that is fascinating in the way that watching a train wreck can be fascinating.

The report below by the U.S. Department of Commerce’s Office of Inspector General details the case of a malware outbreak on computers belonging to the Economic Development Administration in late 2011. In what could only politely be described as a, er, cluster-frig, when only two piece of the agency’s IT infrastructure were infected, it thought that 146 had been.

You won’t believe what ultimately happened. Over the course of five weeks, miscommunication between the Commerce Department’s Computer Incident Response Center and the EDA led its CIO to ultimately order the physical destruction of $170,000 worth of IT components, including PCs, printers, TV sets (what?), digital cameras and mice. On top of that, it paid a security contractor more than $823,000, spent more than $1 million on temporary infrastructure and shelled out $688,000 for contractors to help with a “long term recovery solution.” All told, the agency spent about $2.7 million, or more than half its annual IT budget, fighting a virus that should have taken at most an afternoon to correct. At one point the agency was borrowing surplus computers from the Census Bureau so that employees could get their work done.

The report cites one key factor: Staff members at DOC CIRT were “inexperienced” and suffered from “inadequate knowledge,” and lacked the ability to respond properly to a malware outbreak, which hindered the application of an appropriate response. The person who handled the call from the EDA “had minimal incident response experience, no incident response training, and did not have adequate skills to provide incident response services.”

It sounded so bad I couldn’t believe it until I read the report myself. All 33 pages documenting American tax dollars at work in agonizing, bureaucratic detail are below, though one critical detail is missing: I want to know, was anyone fired?

OIG-13-027-A