US director of National Intelligence James Clapper. Credit:AFP This shrouded, multimillion-dollar hunt for insider threats has suffered from critical delays in recent years and uneven implementation across agencies, the budget records show. And the spy agencies' detection systems never noticed that Mr Snowden was copying highly classified documents from different parts of the NSA's networks. He subsequently fled to Hong Kong and then Moscow, where he remains after being granted temporary asylum. Contractors like Mr Snowden, an NSA spokeswoman said, were not included in the plans to reinvestigate 4000 security clearances. CIA officials said the number of applicants ultimately tied to terrorist networks or hostile foreign governments was "small" but declined to provide an exact number or the reasons the broader group of applicants initially raised concerns.

"Over the last several years, a small subset of CIA's total job applicants were flagged due to various problems or issues," one official said in response to questions. "During this period, one in five of that small subset were found to have significant connections to hostile intelligence services and or terrorist groups." The official, like others interviewed for this article, spoke on the condition of anonymity. The intelligence community's dramatic emphasis on insider threats came in the wake of disclosures by WikiLeaks in 2010. The anti-secrecy group received hundreds of thousands of military and diplomatic documents from Army private Bradley Manning, now known as Chelsea Manning. Congress made security a top priority and in 2011 ordered Director of National Intelligence James Clapper to set up "an effective automated insider threat detection program" to guard against similar security failures. The program was supposed to flag possible abuses, identify double agents and prevent leaks. The project was delayed several times because the intelligence community was preoccupied with handling the fallout from Manning's leaks, the budget documents show. Among a subset of job seekers whose backgrounds raised questions, roughly one out of every five had significant terrorist and/or hostile intelligence connections

Congress gave Mr Clapper an October 2012 deadline to install the automation system and until October 2013 to have it fully operating. At the Obama administration's request, the deadlines were each pushed back a year. Steven Aftergood, a government secrecy expert at the Federation of American Scientists, which analyses national security policy, said he suspects the agency may respond to a lot of "false positives" - alerts for activity that is actually innocent and work-related. "If the 4000 cases turn up only two or three actual threats, they need to adjust their detection threshold or they'll be using a lot of resources for no purpose," he said. An intelligence official knowledgeable about insider-threat programs said government agencies may need to better calibrate their software and reassess their criteria for what constitutes a realistic or likely threat. But he added, "Insider-threat detection is an important security tool that needs to be put in place."

NSA spokeswoman Vanee Vines said the investigations are part of an effort to try to reduce risks, not investigate known threats. "In FY 2013, NSA planned to initiate 4000 re-investigations on civilian employees to reduce the potential of an insider compromise of sensitive information and missions," Ms Vines said in a statement. "Periodic re-investigations are conducted as one due-diligence component of our multifaceted insider threat program." President Barack Obama issued a national insider-threat policy in November 2012 that defines the problem as any risk that insiders will use their access to government secrets, knowingly or unknowingly, in a way that hurts US security. The damage can come through "espionage, terrorism, [or] unauthorised disclosure of national security information, or through the loss or degradation of departmental resources or capabilities." The policy puts leakers of classified information on par with terrorists and double agents, an equivalency that critics of government secrecy find worrisome. "It's disturbing, because they are not the same," Mr Aftergood said. "There are such things as a good leak. Some classified things should be public.The official policy does not admit that distinction."

The newly obtained budget records also reveal what some consider a new security risk in the making. The NSA is creating a massive new database, code-named WILDSAGE, to facilitate the use of sensitive intelligence. The system "provides a mechanism for cybersecurity centres to share signatures at the SECRET classification level," the budget document said. The NSA had not fully implemented WILDSAGE, according to the budget document. But the intelligence community also reported that it "made significant progress in FY 2011 in increasing collaboration" across cybersecurity centres and in creating architecture to share important information on risks." Such shared databases give government employees information they may need to track and disrupt terrorist or cyber plots, officials said. The investigation of a 2009 plot by al-Qaeda's affiliate in Yemen to bring down a civilian aircraft over Detroit found that government agencies failed to share information that could have flagged the would-be bomber and put him on a terrorist watch list. A report by the Senate Intelligence Committee identified 14 failure points, including that the CIA did not disseminate some its reporting, which allowed bomber Umar Farouk Abdulmutallab to board a US-bound flight from Amsterdam.

But the new NSA system has stoked fears about insiders' access after Manning downloaded the entire contents of a similarly shared State Department database. "It's a valid concern," said one intelligence official. "But the other side of this . . . is the need to share information." In the case of Manning's leak, the State Department maintained a shared Net Centric diplomacy database where all secret-level cables were deposited and available to all intelligence agencies to access and search. Manning reviewed the database while creating analytic intelligence products in Iraq. Manning was convicted last month of leaking classified information and subsequently sentenced to 35 years in prison. The prosecutor in Manning's case said the US government ultimately has to entrust its secrets to its employees - even low-level ones.

"Military intelligence professionals go to work every day and use the information available to them to do their jobs," said prosecutor Captain Joe Morrow. "We can take all the mitigation steps in the world, but the bottom line is that there is no step we can take as a nation, as a military, that's going to stop the determined insider." The Washington Post