A federal spy agency inadvertently shared logs of Canadians' phone calls and Internet exchanges with intelligence allies such as the United States for years, a newly disclosed report says.

The revelation that the Communications Security Establishment compromised Canadians' privacy while sharing clandestinely captured data appears in a confidential watchdog's report obtained by The Globe and Mail from court filings related to a lawsuit against the Canadian government.

The report said software that was supposed to remove identifying information on Canadians from material CSE captured during international surveillance operations had failed. This meant that Canada's intelligence allies received data that Canadian laws say they should not see.

Story continues below advertisement

The 2015 report puts in sharper focus the spy agency's struggles to protect Canadians from foreign threats while also safeguarding individual citizens' privacy. The problem was first revealed publicly in January by Defence Minister Harjit Sajjan and CSE officials.

The confidential report was written by Jean-Pierre Plouffe, a retired Quebec judge who heads the Office of the CSE Commissioner, the spy agency's watchdog agency. In it, he suggests the unlawful seepage of Canadians' phone and Internet records to foreign intelligence agencies could date back to the mid-2000s, and that the overall amount of compromised material is unclear.

Given this, Mr. Plouffe is urging Parliament to pass laws spelling out how it wants the spy agency to function. "As CSE's collection posture has strengthened, … the volume of metadata collected has increased considerably," Mr. Plouffe writes in his 2015 report. He urged federal politicians to give clearer direction on surveillance.

"Metadata" are logs of communications without the content of the conversation. The watchdog's report reveals that, during its international spying, CSE has been capturing phone logs and sharing them with allies since 2005. Internet logs have been shared since 2009.

In 2014, CSE suspended sharing both sorts of records when it realized its automated systems had failed to scrub out what it calls the "Canadian identifying information" that turned up in the wider mix. Mr. Plouffe, who has the last word on such matters, eventually ruled that although CSE's system failures were inadvertent, they violated the Privacy Act and National Defence Act.

CSE is part of the world's most powerful spying alliance. Since the 1940s, the "Five Eyes" – electronic-espionage agencies in the United States, the United Kingdom, Canada, Australia and New Zealand – have been working closely together. The collective's members cannot eavesdrop on their own citizens, but their governments have relaxed their rules covering telecommunications trails – metadata – in the hopes it could help the Five Eyes track al-Qaeda terrorists.

Metadata collected and shared on a massive, global scale can show intelligence analysts who is talking to who, even when the contents of the underlying conversations are unknown.

Story continues below advertisement

Parliament passed a law in 2001 giving CSE increased latitude to collect data, subject to orders from defence ministers that spell out what it can and cannot do. The Globe reported in 2013 that both Liberal and Conservative governments have since signed such metadata ministerial directives.

The 2015 watchdog's report reproduces one of these directives. "CSE may search any metadata acquired" to help track "a foreign individual, state organization, terrorist group, or other such entities," the directive says. It adds: "CSE will share metadata … with international allies to maximize" surveillance capabilities, but "Canada's allies shall not be granted access to metadata known to be associated with Canadians located anywhere."

But CSE cannot guarantee it can avoid capturing Canadian telecommunications trails. On Wednesday, a director-general with the agency, Scott Millar, attested to this fact in a Federal Court proceeding related to the lawsuit. He said that while CSE does sometimes collect metadata on Canadians, this is a "very rare occurrence."

Just how CSE collects metadata is a state secret, but it is known that it is gathered in huge volumes indiscriminately. In his 2015 report, Mr. Plouffe says CSE "metadata is acquired without having gone through a targeting selection process."

Only after the initial collection do CSE analysts seek to "minimize" privacy violations by scrubbing out Canadian identifying information, the report says. The agency refers to this as "minimization."

The report reveals that CSE refers to the phone logs it collects as "Dialled Number Recognition" (DNR) metadata. The agency started sharing such material with Five Eyes allies in 2005, thinking it had devised ways to automatically strike out telling portions of any Canadian phone numbers that turned up.

Story continues below advertisement

Then, starting two years ago, CSE discovered that "DNR metadata was not being minimized properly," according to the watchdog report. Mr. Plouffe added: "CSE is unable to determine how many systems were impacted and for how long."

CSE calls the Internet logs it collects "Digital Network Intelligence" (DNI) metadata, and this material can consist of e-mail addresses and Internet protocol addresses that indicate who is communicating to who.

A scrubbing system was developed for that material as well – but this, too, failed. "DNI metadata was being shared with [Five Eyes] Second Parties … with minimization applied to Canadian e-mail address fields, but no minimization applied to Canadian IP address fields," Mr. Plouffe writes.

He adds that "CSE was under the impression that minimization was taking place, when in fact it was not."

The spy agency suspended sharing when the problems were discovered in 2014, and apparently have not resumed it.

If CSE is to return to exchanging such information, the report said, the Liberal government will likely have to enshrine in law how it wants CSE to reconcile individual privacy and security imperatives.

Story continues below advertisement

"I am recommending to the Minister of National Defence that the National Defence Act be amended in order to clarify CSE's authority to collect, use, retain, share and disclose metadata," Mr. Plouffe wrote in a letter to the Minister of National Defence last fall.

Records show Mr. Sajjan has since replied to say he accepted recommendations – but he did not commit to introduce new laws or directives.

With reports from Michelle Zilio and Chris Hannay