Reports show that hacking and cybercrime swamp federal agencies as US struggles to keep pace with international groups of hackers

A $10bn-a-year effort to protect sensitive government data, from military secrets to social security numbers, is struggling to keep pace with an increasing number of cyberattacks and is unwittingly being undermined by federal employees and contractors.

Workers scattered across more than a dozen agencies, from the defense and education departments to the National Weather Service, are responsible for at least half of the federal cyberincidents reported each year since 2010, according to an Associated Press analysis of records.

They have clicked links in bogus phishing emails, opened malware-laden websites and been tricked by scammers into sharing information.

One was redirected to a hostile site after connecting to a video of tennis star Serena Williams. A few act intentionally, most famously former National Security Agency contractor Edward Snowden, who downloaded and leaked documents revealing the government’s collection of phone and email records.

Then there was the contract worker who lost equipment containing the confidential information of millions of Americans, including Robert Curtis, of Monument, Colorado.

“I was angry, because we as citizens trust the government to act on our behalf,” he said. Curtis, according to court records, was besieged by identity thieves after someone stole data tapes that the contractor left in a car, exposing the health records of about 5m current and former Pentagon employees and their families.

At a time when intelligence officials say cybersecurity trumps terrorism as the No1 threat to the US – and when breaches at businesses such as Home Depot and Target focus attention on data security – the federal government isn’t required to publicize its own data losses.

Last month, a breach of unclassified White House computers by hackers thought to be working for Russia was reported not by officials but the Washington Post. Congressional Republicans complained even they weren’t alerted to the hack.

To determine the extent of federal cyberincidents, the AP filed dozens of Freedom of Information Act requests, interviewed hackers, cybersecurity experts and government officials and obtained documents describing digital cracks in the system.

Hackers breached the White House’s unclassified network in October. Photograph: Mark Wilson/Getty Images

That review shows that 40 years and more than $100bn after the first federal data protection law was enacted, the government is struggling to close holes without the knowledge, staff or systems to outwit an ever-evolving foe.

Fears about breaches have been around since the late 1960s, when the federal government began shifting its operations onto computers. Officials responded with software designed to sniff out malicious programs and raise alarms about intruders. And yet, attackers have always found a way in, exposing tens of millions of sensitive and private records that include employee usernames and passwords and veterans’ medical files.

From 2009 to 2013, the number of reported breaches just on federal computer networks – the .gov and .mils – rose from 26,942 to 46,605, according to the US computer emergency readiness team. Last year, US-CERT responded to a total of 228,700 cyberincidents involving federal agencies, companies that run critical infrastructure and contract partners. That’s more than double the incidents in 2009.

And employees are to blame for at least half of the problems.

Last year, for example, about 21% of all federal breaches were traced to government workers who violated policies; 16% who lost devices or had them stolen; 12% who improperly handled sensitive information printed from computers; at least 8% who ran or installed malicious software; and 6% who were enticed to share private information, according to an annual White House review.

Documents released to the AP show how workers were lured in.

In one incident around Christmas 2011, Education Department employees received an email purportedly from Amazon.com that asked them to click on a link. Officials quickly warned staff that it could be malicious. The department did not release information to the AP about any resulting damage.

Reports from the Defense Department’s defense security service, tasked with protecting classified information and technologies in the hands of federal contractors, show how easy it is for hackers to get into DOD networks. One military user received messages that his computer was infected when he visited a website about schools. Officials tracked the attacker to what appeared to be a Germany-based server.

“We’ll always be vulnerable to … human-factor attacks unless we educate the overall workforce,” said assistant secretary of defense and cybersecurity adviser Eric Rosenbach.

Facebook Twitter Pinterest The US government is projected to spend $65bn on cybersecurity bettween 2015 and 2020. Photograph: Rick Wilking / Reuters/Reuters

Although the government is projected to spend $65bn on cybersecurity contracts between 2015 and 2020, many experts believe the effort is not enough to counter a growing pool of hackers whose motives vary. Russia, Iran and China have been named as suspects in some attacks, while thieves seek out other valuable data. Only a small fraction of attackers are caught.

For every thief or hostile state, there are tens of thousands of victims like Robert Curtis.

He declined to talk about specifics of his case. According to court records, a thief in September 2011 broke into a car in a San Antonio garage and stole unencrypted computer tapes containing the Pentagon workers’ information. The car belonged to an employee of a federal contractor tasked with securing those records.

Criminals have tried to get cash, loans, credit – even establish businesses – in Curtis’ name, according to court records. He and his wife have frozen bank and credit accounts. A lawsuit brought by victims was dismissed.

“It is very ironic,” said Curtis, himself a cybersecurity expert who worked to provide secure networks at the Pentagon. “I was the person who had paper shredders in my house. I was a consummate data protection guy.”