The Pentagon’s taskforce charged with stopping insurgent bombs has for years inappropriately acted as an intelligence agency, according to a declassified internal report obtained by the Guardian.

The Joint Improvised Explosive Device Defeat Organization (JIEDDO) collected information on American companies and their executives, people inside the United States, US military personnel and Afghan farmers. Despite internal Pentagon criticism, it continues to carry out intelligence functions.

JIEDDO used aliases and impersonated US college students to gather information. It pursued US firms doing business with a Pakistani company with no real ties to terrorism. It collected and “improperly retained” US telephone numbers, as well as those from among the US’ “Five Eyes” intelligence partners: the UK, Australia, New Zealand and Canada. On at least one occasion, JIEDDO mishandled information it accessed from a National Security Agency database.

JIEDDO disputes that it actually “collected” intelligence, preferring to say it “aggregated” already-existing intelligence and public data.

Some of JIEDDO’s intelligence work, particularly the collection of data on US companies, occurred “at the behest of [JIEDDO] leadership”. All of it violated a raft of Defense Department and executive-branch regulations, up to and including Executive Order 12333, a foundational intelligence guideline. Those violations led to the first-ever establishment of an inspector general within JIEDDO, William Rigby.

JIEDDO’s previously unknown intelligence activities are detailed in an 80-page investigation conducted by the Pentagon inspector general, issued secretly in April and acquired by the Guardian and other news organizations through the Freedom of Information Act. It is the latest example of a post-9/11 proliferation of intelligence operations by unapproved US government organizations.

JIEDDO continues collecting and retaining intelligence, including intelligence on Americans. One analyst told the inspector general that when the organization acquires Americans’ data, its officials “tuck it to the side”. As of April 2014, the inspector general wrote: “We remain concerned about JIEDDO conducting activities that fall out of the scope of its original charter.”

That mission creep first led JIEDDO to view expansively its minimal intelligence authorities related to foreign IED threats. Then, confusion amongst the workforce about what intelligence activities were permissible “allowed contract analysts and government supervisors to collect information about US persons that fell outside of [a JIEDDO component’s] foreign intelligence function”. As far back as February 2007 – barely a year into JIEDDO’s existence – the director of the Defense Intelligence Agency warned that JIEDDO should be properly reclassified as an intelligence enterprise to remove dangerous ambiguity about the organization’s true activities.

David Small, a JIEDDO spokesman, said that while JIEDDO is a “non-intelligence community” entity, it retains the ability to gather ostensibly public information, including on Americans, so long as it is not collecting the intelligence itself in the first instance.

“What JIEDDO analysts failed to do in the cited incidents was properly label and compartmentalize the information on US Persons while conducting work that involved US Persons data. So the violation wasn’t that JIEDDO ‘collected info’ on US Persons. The issue is in how JIEDDO analysts stored that information. Storing that information can also be referred to as ‘collection’ in the temporal sense,” Small said.

JIEDDO also retains access to “hundreds” of databases maintained by US intelligence agencies, Small said, but cannot task those agencies to collect intelligence for JIEDDO.

Under US law and executive order, intelligence operations can only be conducted by authorized agencies. Those agencies follow a defined oversight chain, detailing who in the executive branch can approve intelligence operations and who in the legislature must be informed about them.

JIEDDO is not one of those agencies. It was established in 2006 to overcome the lethal battlefield threat of insurgents’ homemade bombs in Iraq and Afghanistan. It was never an intelligence entity – though it has received tens of billions of dollars from lawmakers eager to mitigate the signature weapon of post-9/11 insurgent wars.

“It is not illegal to collect information on US Persons, and JIEDDO has the authority to aggregate information collected by others within the intelligence community, including info that may have data on US Persons,” Small said.

From its inception, JIEDDO’s charter permitted one narrow band of intelligence activities. It was to establish a “joint common intelligence picture” of the global threat of improvised explosive devices, or IEDs. JIEDDO’s “sole intelligence component” was called the COIC, which ultimately stood for Counter-IED Operations/Intelligence Integration Center. The COIC would comprise 1,359 people, all but 58 of whom were contractors.

According to the Pentagon inspector general, the COIC would open a floodgate of inappropriate data collection, despite assurances to the contrary by JIEDDO leadership. COIC leadership was described as “comfortable using a liberal understanding of an IED connection for accepting outside tasks”. In 2012, the Government Oversight Agency cited the agency for “redundancy” in supporting some 70 “electronic data collection and analysis tools” at a cost of at least $184m.

COIC documents reviewed by the inspector general state that it has no intelligence collection role. JIEDDO’s vice director told the inspector general in August 2012 that neither “JIEDDO nor the COIC collects information on US persons”. Yet COIC would indeed collect information on, among others, American companies that did business with a Pakistani fertilizer firm.

The vast majority of Taliban IEDs rely for their explosives on ammonium nitrate, an ingredient in certain fertilizers, prompting a US effort to halt the influx of those fertilizers and related material into Afghanistan. One Pakistani fertilizer company that attracted JIEDDO’s attention was a subsidiary of the Lahore-based Fatima Group. Yet according to the inspector general’s report, not even JIEDDO possessed sufficient suspicion to link the Fatima Group to IEDs, nor did the DIA find any evidence of illicit activity by Fatima after a “deep-dive study”.

Still, in 2012, a JIEDDO senior official, whose name is redacted in the report seen by the Guardian, instructed the COIC to “collect information on US companies that did business with Fatima”. The goal was to find firms that could be enlisted to “pressure” Fatima.

The data sought was on the order of a “mergers and acquisitions analysis”, the sort of due diligence that another firm interested in buying Fatima might conduct, including “an abundant amount of financial and investor data that would illuminate with whom Fatima interacted”. That data could, if necessary, give JIEDDO “options” for taking action against Fatima, including leaning on the Pakistani firm’s US business partners.

But under surveillance law and practice, “US persons”, including companies, are supposed to be a protected class. The CIA, according to the inspector general, refused a JIEDDO request for aid, saying JIEDDO had failed to establish Fatima’s links to terrorism. An analyst tasked with researching Fatima blew the whistle to the Defense Department inspector general in 2012, after saying that an intelligence oversight officer at the COIC verbally communicated that US companies had to be off-limits “due to a lack of derogatory information”.

But data collection on US companies proceeded. The specific names of the companies JIEDDO investigated are blacked out in the report. JIEDDO used “open-source” methods of gathering and analyzing publicly available information. But some of the easiest-to-access data were insufficient to JIEDDO’s purposes. It began using webcrawling software called Halogen to trawl the internet and retrieve the names of leaders of companies doing business with Fatima, the services provided and the depths of their business connections to the Pakistani firm.

An August 2012 internal JIEDDO document explained: “By approaching these partner companies – and their shareholders – and making them aware that they are associated with a company whose product is being misappropriated, causing 10,840 casualties in 2011 alone, Fatima may be encouraged to be more cooperative …” JIEDDO senior leadership “personally contacted and met with US company CEOs” and told them Fatima was involved in IEDs that killed US troops.

A subsequent briefing JIEDDO prepared on US companies tied to Fatima inappropriately disseminated information on at least several US persons. A different briefing, this one about Bosnian Islamist extremist networks, disseminated information on at least eight US citizens and a permanent resident.

The Fatima incident was not the only illicit intelligence effort on Americans that JIEDDO conducted. The inspector general found that JIEDDO ordered analysts to investigate Yonathan Melaku, a Marine Corps reservist who was arrested in 2011 in Arlington National Cemetery on bomb-making suspicions. (Melaku later pleaded guilty to unrelated crimes.) The organization also traded information with special-operations personnel about Sergeant Bowe Bergdahl, the soldier formerly held hostage by the Taliban, and apparently aided a law-enforcement investigation into Iraqi refugees in the US suspected of being bombmakers. None of these functions are related to JIEDDO’s mandate, nor permissible by Defense Department regulation.

A JIEDDO contractor even posed in 2009 as a US college student to call Afghan farmers, using the unauthorized cover story of working on a scholastic research project, to “ask about their crops and opinions on the Coalition war effort in that country”. The contractor said a co-worker suggested using the fake cover, and the contractor, discomfited by the practice, stopped calling after two days.

JIEDDO also evidently used aliases in a broader manner, mined Facebook and Twitter to collect information and inappropriately accessed an NSA database, but sections on those practices are blacked out of the declassified report. So is nearly an entire page on what “US persons information” JIEDDO retained beyond a 90-day regulatory limit to establish relevance or purge. One contractor told the investigation that “no record existed to indicate that any US persons information had ever been deleted”.

Small said JIEDDO now conducts a “weekly” review over what material to retain or purge. He said that American data considered relevant to JIEDDO’s expanding mission can be retained and disseminated beyond the 90-day limit, once examined by the organization’s lawyers and approved by a senior official.

The Pentagon inspector general recommended that JIEDDO stop all intelligence collection activities unless specifically authorized by the Department of Defense. Yet the JIEDDO director, Army Lt Gen John D Johnson, responded that doing so would jeopardize the anti-IED fight in Afghanistan – a position Small said Johnson still holds.

Johnson, in an addendum to the report, partially concurred with many of the inspector general’s findings, though he disputed several. “JIEDDO COIC’s legal authority to collect intelligence is very clear. However, the nature and scope of its intelligence mission under which it accomplishes its collection activities should be clarified,” he wrote in December 2013.

Johnson did not dispute that JIEDDO collected information on US persons, and said that the COIC’s internal procedures on retaining and disseminating US data had “improved”.

JIEDDO’s recent, surreptitious move into the intelligence world is not the first by a US defense agency that does not have an intelligence mandate. The Defense Advanced Research Projects Agency (Darpa) ran a “population-centric” data analysis program in Afghanistan, called Nexus 7, that sought to divine trends in the war through examining fluctuations in the prices of daily goods – a sprawl in Darpa’s mission that discomfited intelligence veterans.

Unlike DARPA, JIEDDO has a host of Capitol Hill critics who consider its success at stopping IEDs meager compared to the $22bn Congress has provided it over its lifespan. As recently as last year, lawmakers asked the Pentagon to phase JIEDDO out of existence entirely.

In a statement, Small said JIEDDO “appreciate[d]” the inspector general review.

“The IG substantiated the allegation that JIEDDO’s COIC illegally or inappropriately collected info about US persons. The incidents involved regarded technicalities of policy and process and were corrected as soon as it was recognized. Many of the IG’s recommendations to assure such actions do not occur in the future have already been acted upon,” he said in the statement.