Persistent foreign spying threats prompted Canada's electronic-eavesdropping agency to embark on a counterespionage campaign so aggressive that its former chief says he "shut the place down" before it could be exposed to allegations of wrongful domestic surveillance.

The Communications Security Establishment Canada had launched the program in the years after 9/11 in the hopes that collecting and analyzing Canadian communications trails – or "metadata" – could point to foreign hackers targeting Canadian government computer systems.

But the program was reined in by former CSEC chief John Adams during his 2005-11 tenure. "I'm saying 'Whoa. Whoa,' " Mr. Adams recalled in an interview. He said he ordered the pause to iron out potential privacy problems – even though officials in other federal departments in Ottawa were upset with CSEC because they saw cyber espionage as a problem.

Story continues below advertisement

Mr. Adams did not say when the program started or when he ordered its end. Revelations of CSEC's program to root out foreign hackers on domestic soil come a day after U.S. prosecutors laid corporate espionage charges against five Chinese nationals suspected of being part of People's Liberation Army "Unit 61398," a group of state-sponsored hackers allegedly tasked with stealing North American secrets for Beijing's benefit.

The same spying squad is suspected of targeting a Canadian federal department in 2011, according to a U.S. cybersecurity expert interviewed by The Globe.

Records show that the same year, the Ottawa bureaucracy was grappling with fallout from significant hacks of unknown origin against at least five federal departments.

CSEC's handling of the campaign is the latest example of modern communications blurring the lines between foreign and domestic surveillance.

Controversy has dogged CSEC after leaks in recent months about its efforts to spy on Brazil's energy sector and also track digital devices that had passed through a Canadian airport.

Collecting intelligence about foreigners accounts for roughly two-thirds of CSEC's almost $500-million operational budget. The other third goes to its role in safeguarding federal communications systems.

After doing both jobs for nearly 70 years, CSEC is grappling with what constitutes lawful and unlawful surveillance in the Internet age. Without active review by legislators and judges, who are rarely apprised of CSEC operations, the agency's executives and lawyers are left to chart their own legal course as surveillance capabilities grow more powerful.

Story continues below advertisement

Mr. Adams, the retired former chief, stressed he always tried to instill a "culture of compliance" to his staff.

CSEC remains broadly banned from eavesdropping on Canadians' conversations. "I said to them, 'If you cross the line, we will be out of business.'"

But over the past 10 years, the agency has given some legal latitude to look at the data around citizens' communications – phone logs, Internet Protocol addresses and other "metadata" material. The original hope was that such material can be used to find foreign terrorists, but the sensibility is morphing into the realm of cybercounterespionage.

Foreign hackers often hide their aims and identities by first compromising Canadian or U.S. computer servers. The devices then become staging grounds for attacks.

To combat this threat, spy agencies are being drawn into grey-area practices – such as taking a closer look at pools of domestic communications, in hopes of pinpointing the hostile hackers.

In an interview last November, Mr. Adams spoke about how CSEC had started adapting its mandate to spy on foreign communications – "signals intelligence" or "SigInt" – to its job within Canada of searching for threats to "information technology security (ITS)."

Story continues below advertisement

"Protecting Canada means you're going to be hitting Canadians," Mr. Adams said. While stressing that this is not illegal for CSEC, "the trouble with it was they were applying 'SigInt' practices to internal [communications] and it was just a little too loose" – given that the technique would reap mostly Canadian information.

He said he told his ITS staff that "we've got to sit down and change our procedures."

Mr. Adams said, "I shut the place down when I was worried about us trying to run before we walked."